From 83a0f16a2187d2d924a793cf7ae8fb673cb2d490 Mon Sep 17 00:00:00 2001 From: "mudler's LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Mon, 20 Jul 2026 18:43:02 +0200 Subject: [PATCH] feat(gallery): let one gallery entry offer several builds of the same model (#10943) * feat(system): expose raw detected capability for model meta resolution Model meta gallery entries express hardware fallback through candidate ordering rather than a capability map, so they need the undecorated detected capability string without Capability's default/cpu fallback chain. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * refactor(system): drop duplicate capability accessor, cover DetectedCapability ReportedCapability was added with a body identical to the existing DetectedCapability. Keep one accessor and move the specs onto it, since DetectedCapability had no direct coverage of its no-fallback behavior. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(vram): parse IEC binary size suffixes (KiB..PiB) ParseSizeString accepted only SI suffixes, so a "20GiB" floor was rejected outright. Model and VRAM sizes are conventionally quoted in IEC units, and silently reading GiB as GB would understate a floor by about 7%. Purely additive: these inputs previously returned an unknown-suffix error. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): add Candidate type for meta model entries Candidate is one option in a meta entry's ordered variant list. It names a concrete gallery entry and declares when that entry suits the host. EffectiveMinVRAM resolves the VRAM floor, letting an authored min_vram win over a nightly-inferred one. An unparseable floor errors instead of being treated as absent: swallowing a typo would turn a constrained candidate into an unconstrained one and select a too-large variant rather than fail loudly. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): add hardware-aware model variant resolver Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): allow gallery model entries to declare variant candidates A gallery entry with a non-empty candidates list is a meta entry: it names an ordered list of concrete entries and resolves to the first one the host can satisfy, instead of describing model files directly. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): resolve meta model entries to hardware-appropriate variants at install Meta gallery entries carry an ordered candidate list; at install time the first candidate the host satisfies is resolved and its payload installed under the meta's name, so the model keeps a stable name regardless of which variant backs it. The resolution is recorded in the installed gallery config so a reinstall honors a prior pin and operators can see the backing variant. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(gallery): key meta pin recall on the installed name and detach resolved entries Six review findings on the meta-entry install path. Pin recall was keyed on the gallery entry name while applyModel writes the record under the install name (req.Name when supplied), so a meta installed under a custom name with a pin lost that pin on reinstall and was silently re-resolved onto a different variant, possibly swapping its backend. Compute the install name with applyModel's own precedence before the recall. ResolveMetaModel returned a shallow struct copy, so the resolved entry's Overrides aliased the gallery entry's map and the install path's in-place mergo merge wrote the caller's request into the shared catalog. Detach Overrides, ConfigFile, AdditionalFiles, URLs and Tags. Not exploitable today only because this path re-unmarshals the gallery per call, which is a property nobody should have to rely on. Also: overlay the meta's name onto the persisted config for meta installs so the gallery file no longer records the variant's name; move the pinned-VRAM warning below the variant validation so a pin naming a nonexistent entry does not warn about VRAM before failing for an unrelated reason; and stop seeding config.URLs in the config_file branch, which duplicated every declared URL. Add seven network-free specs driving InstallModelFromGallery with a meta entry: variant payload wins over the meta's legacy url fallback, the resolution record round-trips to disk, a pin is recorded and honored on reinstall including under a custom install name, and the resolved entry does not alias the gallery's maps. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(gallery): deep-copy meta overrides and make two specs functional ResolveMetaModel detached the resolved entry's Overrides and ConfigFile with maps.Clone, which only copies the top level. Gallery overrides are nested in practice (parameters.model is near-universal) and the install path merges the caller's request with mergo.WithOverride, which recurses into nested maps and overwrites them in place, so the gallery entry's own inner maps were still reachable and still got rewritten by the last caller to install. Copy both maps all the way down instead, recursing through the container shapes a YAML decoder produces. ConfigFile is not mutated on the install path today, but it carries the same kind of nested payload and leaving it shallowly cloned would invite the bug back. Also fix two specs that passed whether or not their target fix was present: - "does not write the caller's overrides back into the gallery entry" re-read the catalog from disk, which re-unmarshals fresh structs and so cannot observe in-memory aliasing. It now asserts against the in-memory gallery entry and drives the real mergo merge. - "round-trips the resolution record to disk under the meta's name" asserted a name that is already correct in the config_file branch. It now drives the url branch via a file:// fixture, where the meta-name overlay actually applies. Both were verified red by reverting their fix. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * test(gallery): lint meta model entry invariants in index.yaml Adds Ginkgo specs that parse the shipped gallery/index.yaml and enforce the invariants that keep meta entries safe: a legacy url fallback equal to the final candidate's url, references only to existing non-meta entries, a min_vram floor on every candidate but the last-resort one, a capability drawn only from the vocabulary the system can report, and descending VRAM floors within a capability group. The capability check is the only compensating control for a typo there. Candidate matching is a case-sensitive exact comparison against SystemState.DetectedCapability(), so an unknown value never matches and falls through silently instead of erroring. The vocabulary therefore mirrors the raw return set of getSystemCapabilities(), which notably excludes "cpu": that is a fallback key inside Capability(capMap) on the meta backend path, never a reported capability. A CPU-only host reports "default". These pass vacuously until the pilot meta entry lands; the guard is intentionally in place before the thing it guards. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * test(gallery): close coverage gaps in the meta entry lint The ordering invariant grouped candidates by capability and asserted floors descend within a group. A candidate with an EMPTY capability matches every host, so it does not belong in its own group: it dominates every later candidate whose floor is at or above its own, across capability groups. Track a running minimum floor over the unconditional candidates instead, which subsumes the old same-group check for the empty capability. Every spec skipped non-meta entries, so with zero meta entries in the index all five bodies were no-ops. Aligning GalleryModel.IsMeta() with GalleryBackend.IsMeta(), whose semantics are deliberately opposite, would have made all of them pass while checking nothing. Extract each invariant into a helper over a slice of entries returning the violations it finds, and cover those helpers with synthetic fixtures so the logic stays tested at zero meta entries. The index-driven specs are now a thin application of already proven logic. Also assert the index parses non-empty, report every violation in one run rather than aborting on the first, and parse the index once for the suite. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * ci(gallery): add nightly denormalization of meta model candidates Fills the read-only backend, quantization and inferred_min_vram fields on meta gallery candidates and opens a PR, modeled on the existing checksum_checker job. Computing these needs network access, so it happens nightly rather than at install time. An authored min_vram is never modified: a human who measured a real load knows more than a pre-download estimate does. The index is rewritten via yaml.Node rather than a document round-trip. A full round-trip reflows all ~26k lines of gallery/index.yaml, which would bury the computed values and make the nightly PR unreviewable. The rewrite touches only the three derived keys, so authored styling survives and a run that computes nothing leaves the file untouched. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(ci): keep the gallery denormalize diff reviewable and self-healing The nightly denormalization job edits YAML nodes instead of round-tripping structs so its PR stays small enough for a human to review, but the write path undid that: yaml.Marshal re-encoded the node tree at yaml.v3's default 4-space indent and dropped the leading document marker, reflowing roughly 6000 lines around the handful of real changes. Encode through yaml.NewEncoder at the index's authored 2-space indent and restore the header. A write that changes three fields now changes three lines. Stale inferred_min_vram values were also never cleared. Both skip paths (an authored min_vram is present, or the candidate is the last resort) returned before touching the field, so a candidate that gained a floor or became the last resort after a reorder kept an inferred value that EffectiveMinVRAM reported as a real constraint, failing the meta lint with no way for the job to self-heal. Clear the field before both skips. The workflow discarded a whole night's work on any single failure: the program exits 1 when a candidate cannot be estimated, which aborted the job before the PR step, so one unreachable candidate blocked every other refresh indefinitely. Capture the status, open the PR with what was computed, mark the PR body as partial, and fail the run afterwards so the problem still surfaces. Also preserve the index's existing file mode instead of forcing 0644, and drop the redundant //go:build ignore tag, since Go already skips dot directories and the sibling modelslist.go carries no tag. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): add nanbeige4.1-3b meta entry with hardware-resolved variants Adds the first real meta entry to the gallery index. It resolves to the Q8_0 build on hosts with at least 6GiB of VRAM and to the Q4_K_M build everywhere else, installing either payload under the stable name nanbeige4.1-3b. The entry carries a url equal to its final candidate's url. LocalAI releases that predate candidates support parse the index non-strictly and drop the key silently, so without that url they would list the entry and install nothing. A regression spec parses the index the way those releases do and asserts every meta entry stays installable for them. Also teaches core/schema/gallery-model.schema.json about candidates. The schema sets additionalProperties: false at the top level, so an author following CONTRIBUTING.md and adding the yaml-language-server comment would otherwise get a validation error on this entry. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): make candidate entries complete, installable entries Reworks hardware-resolved gallery variants after a design pivot. There is no longer a separate "meta" entry kind. A gallery entry is a normal, complete entry that may additionally carry candidates:, a list of hardware-gated upgrades over itself, and the entry is itself the last-resort candidate. The previous design relied on a bare url: as the fallback for LocalAI releases that predate candidates support. That fallback is empty in practice: none of the 80 gallery/*.yaml files carry a top-level files:, and 1216 of 1281 index entries carry their payload in the index entry itself, so a url alone yields a config template with nothing to download. Since every released LocalAI reads gallery/index.yaml live from master, merging a payload-less entry would have shown every existing user a model that installs to a broken state. Making the entry its own base candidate removes the problem at the root: old clients drop the candidates key and install the entry exactly as they do today. Resolution order is now explicit pin, then capability plus VRAM over the declared upgrades, then the entry itself. The entry ALWAYS installs: when its own min_vram or capability is unmet the installer warns and installs it anyway, because there is nothing below it and refusing would make the gallery behave worse the newer the client is. A pin naming the entry's own name is valid and is how an operator declines an upgrade. IsMeta() becomes HasCandidates(), ResolveMetaModel becomes ResolveVariant, and the persisted meta_name record key becomes entry_name. GalleryBackend.IsMeta() is a separate concept and is untouched. The lint drops the three rules the pivot makes wrong (url equality with the final candidate, no inline payload, unconstrained final candidate) and gains one: the entry's own floor must sit strictly below every candidate's, since a base that outranks a candidate makes that candidate unreachable. The pilot entry is now the existing nanbeige4.1-3b-q4, which gains a 2GiB floor of its own and a single 6GiB upgrade to nanbeige4.1-3b-q8, replacing the separate nanbeige4.1-3b entry added in d0d441bb4. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): select model variants by hardware fit, not authored order Gallery entries could already carry a list of alternatives, but selection was an authored, ordered, first-match policy: every candidate declared a `capability` string and the VRAM floors had to descend in a hand-tuned order. That pushed hardware knowledge onto whoever edits the gallery and made ordering load-bearing, so a reordered list silently changed what users installed. None of it was necessary. SystemState.IsBackendCompatible already derives hardware support from a backend name alone: it knows MLX and metal are Darwin-only, CUDA is NVIDIA-only, ROCm AMD-only, SYCL Intel-only. Selection can read that instead of asking authors to restate it. Authoring is now just a list of names: - name: qwen3.6-27b min_memory: 4GiB variants: - model: qwen3.6-27b-mlx-8bit - model: qwen3.6-27b-gguf-q8 min_memory: 28GiB and all the intelligence moved into the selector. Given a host it drops the variants whose backend cannot run here, drops those whose known memory requirement exceeds what the host has, and takes the LARGEST of what is left, because a bigger footprint is a higher quality quantization of the same model. A variant of unknown size is kept, since nothing proves it does not fit, but it ranks last so a proven fit always beats a guess. An explicit pin still wins outright, and if nothing survives the entry installs its own payload: the base always installs, this never refuses. Available memory is VRAM when a GPU was detected and system RAM otherwise, read through xsysinfo so a cgroup limit is honored and a container gets its own limit rather than the node's RAM. Capability disappears entirely, from the types, the schema and the lint. VRAM and RAM collapse into one `min_memory`, because a model's footprint is roughly the same wherever it lives and one figure is compared against whichever applies. The lint rules about ordering, the capability vocabulary and floor relationships are deleted with the hazards they described; what remains is that every variant names an entry that exists and does not itself declare variants, plus that any memory figure actually parses. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * gallery: size model variants with a live probe, drop the nightly denormalizer Selection needs each variant's size to decide whether it fits and to rank largest-first. That figure was written into the index by a nightly job, which made the gallery carry a derived value that could drift from the entry it was derived from. Derive it at install time instead. pkg/vram already sizes a model without downloading it, and the gallery UI already uses it: a remote GGUF header range-fetch, then an HTTP HEAD for the content length, then any declared size:. It caches its results, so reuse it rather than writing a second probing path. A probe failure must never fail an install, so an unprobeable variant is treated as unknown: it survives the memory filter, because nothing proves it does not fit, and it ranks last, so a known-good fit always beats a guess. If every probe fails, selection still terminates on the base entry. The probe is injected through ResolveEnv rather than called directly, for the same reason the backend compatibility check is: specs pin an exact size, or an exact failure, without reaching the network. With that in place three things are dead weight and go: - The nightly job and the fields it populated. Variant.Backend was redundant because the backend is resolved live from the referenced entry during selection, and Quantization was display-only that nothing read. - min_memory on the base entry. The base always installs and its floor could only warn, so it could not change any outcome. - The lint rules and schema entries for both. min_memory on individual variants stays, as the override for when the probed size is wrong. An authored figure now suppresses the probe entirely rather than merely outranking it, so it costs no round trip. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): expose model variants for selection over API, CLI and MCP A gallery entry may carry `variants:`, alternative builds of the same model. Selection already worked at install time, but nothing could see what an entry offered or ask for a specific build, so the feature was undrivable. Listing: `GET /api/models` now reports `variants` and `auto_variant` for the entries that declare variants. Each variant carries its resolved backend, its measured size and whether it fits this host. `auto_variant` is what installing without a choice would pick right now. The new gallery.DescribeVariants runs the same variantOptions + SelectVariant pass the installer runs, so the reported default cannot drift from what installing actually does, and HostResolveEnv is extracted so both derive the host and share pkg/vram's probe cache from one place. Performance: an entry that declares no variants returns early without touching the probe, so the ~1280 ordinary entries cost exactly what they cost before. Selection: `variant` is accepted on POST /models/apply, as a query param on POST /api/models/install/:id, on the gallery apply file/string request, as `local-ai models install --variant`, and as a parameter on the install_model MCP tool (both the httpapi and inproc clients). Empty means auto-select. An unknown variant name now fails the install naming what was requested. This closes a real hole: an entry declaring no variants short-circuits before selection runs, so a requested variant was previously dropped silently and the install reported success. startup.InstallModels ends in a variadic model list, so install options could not be appended to it; InstallModelsWithOptions is added alongside and InstallModels delegates to it. No caller signature changed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): drop the redundant variant min_memory field Variant.MinMemory was an authored override for when the live probe misreads a variant's footprint. It duplicated an existing field: probeEntryMemory already passes the entry's declared size: into EstimateModelMultiContext, whose cascade prefers that declared size over its own guesswork. Correcting size: on the referenced entry fixes the figure for every consumer rather than only for variant selection, so min_memory shadowed the right answer. A variant is now nothing but a name. Its effective size is exactly the probe result, and an unknown stays unknown: it survives the filter and ranks last. EffectiveMemory loses its error return along with the field. The authored string was the only thing that could fail to parse, so the error had no remaining source and was propagating dead nil-checks through SelectVariant, DescribeVariants and the pin warning. Selection behaviour is unchanged. The specs covering probe-derived sizing, ranking, filtering, the unknown-size path, pin recall, entry/variant metadata split and deep-copy isolation all survive; the three install specs that needed a definite size now declare it through the referenced entry's own size:, which exercises the documented escape hatch directly. gallery/index.yaml is untouched: no entry ever carried the key. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(gallery): rank the entry's own build against its variants Variant selection pulled the declaring entry's own payload, the base, out of the candidate set and consulted it only once every declared variant had been rejected. Two real failures followed. A variant whose size the probe cannot determine deliberately survives the memory filter, because nothing proves it does not fit. As the only survivor it then won outright on any host, however small: a 2GiB machine installed an unmeasured variant in preference to the 4GiB build the entry itself ships, with no warning. 241 of the 1280 current index entries carry no files and no size, which is exactly that shape. "Largest wins" also broke whenever the base was the largest. An author writing a Q8 entry that offers a Q4 downgrade for small hosts, a natural shape that nothing in the lint, schema or docs discourages, had the Q4 installed on every large host instead. Make the base an ordinary participant. It is still exempt from both filters, so selection always terminates on something installable, but it is now ranked against the variants: a proven fit first and largest, then the base, then any variant whose size nothing could measure. Both failures disappear together. The base is probed for its size accordingly, which it was not before, because an unsized base would lose every contest to an unmeasurable variant. FellBackToBase is kept but narrowed to "no declared variant survived", rather than "the base was chosen", since the base now also wins on merit and that is not worth warning about. A recalled variant pin also became a permanent install failure. A pin the caller supplies on this request must stay fatal, but one recalled from ._gallery_.yaml can be invalidated by any later gallery edit, and failing on it turned one rename into a model that could never be reinstalled or upgraded again short of deleting a dotfile the user has never heard of. A stale recalled pin is now dropped with a warning naming it, and selection runs as if it had never been recorded. Also drop the last textual reference to two abandoned designs from the DetectedCapability comment, correct the documented variants JSON example, which showed a memory_bytes of 0 that omitempty makes impossible, and remove an em dash from the install skill. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(gallery): budget variant memory from RAM when a GPU reports no VRAM Variant selection read its memory budget from VRAM whenever a GPU capability was detected, and from system RAM only when none was. Apple Silicon satisfies the first branch and fails the premise: arm64 macs report the metal capability unconditionally, without probing anything, while TotalAvailableVRAM has no discrete VRAM pool to find and returns zero. The budget therefore came out as zero on every Mac. Zero drops every variant carrying a known size, so the base build was installed on all of them however much memory the machine had. The feature was inert on the platform, and silently: falling back to the base is a legitimate outcome, so nothing looked wrong. Take VRAM only when it is actually a number, and fall back to RAM otherwise. On a unified-memory host RAM is not an approximation of the budget, it is the budget, since the GPU shares it. A discrete GPU whose VRAM could not be read also lands on RAM, which overstates what the card holds but understates nothing the host has; the previous zero understated both. An unreadable RAM figure still yields zero and still installs the base, so a genuinely unknown host is not talked into a larger download. This is what turned tests-apple red: "installs a fitting variant's payload under the entry's own name" asserts on selection, and the runner resolved to the base because its budget was zero. The added specs pin the branch directly rather than relying on a macOS runner to notice again. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(ui): add a model variant picker to the models gallery PR #10943 shipped the server side: a gallery entry may declare `variants:`, `GET /api/models` attaches `variants` and `auto_variant` to declaring entries, and `POST /api/models/install/:id` accepts a `variant` query parameter. Nothing in the UI consumed any of it, so the feature was not reachable from the browser. This wires it up. modelsApi.install takes an optional second argument and appends an encoded `?variant=` only when one is given, so every existing call site keeps sending exactly the request it sent before. On the models table, an entry that declares variants gets a split button. The primary Install still installs the auto-selected build, because auto is the default and the point of the feature; the chevron opens a menu for a deliberate override. It follows the Backends.jsx precedent: one shared Popover re-anchored per row, rendering .action-menu items, which brings Escape, outside-click and focus return along with it. An entry that declares no variants renders exactly as it did before. A variant that does not fit is dimmed but stays selectable, since the server honors an explicit choice with a warning rather than refusing it. memory_bytes is omitempty on the wire, so an absent key means the size is unknown and never zero. A single helper guards both the menu and the detail row, because formatBytes would otherwise render a falsy value as "0 B", which reads as "needs nothing". The expanded detail row gains a Variants section listing each build's backend, size, whether it fits, which is the entry's own build, and which one auto-selection would pick, built from the existing DetailRow helper and .badge classes. Eight Playwright specs cover the picker, including that plain Install sends no variant parameter and that choosing one sends it. One pre-existing assertion was scoped with .first(): the Variants section legitimately adds more llama-cpp badges to the detail row, which tripped strict mode. UI line coverage 49.42% -> 49.36% against a 40.0 baseline and 0.8pp tolerance; branch coverage rose 72.04% -> 72.66%. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(gallery): describe model variants from a companion endpoint Variant description probes each referenced entry's weight files over the network: an HTTP HEAD plus a ranged GET, serial, five seconds per probe with no aggregate deadline. Running it inline in GET /api/models made one listing cost (entries x variants) round trips. The Manage page fetches with items=9999, so at 200 declaring entries that is ~1000 serial probes, minutes of a blocked handler and gigabytes of range traffic for a single page load. Only one entry declares variants today, but the feature exists so that many will. Follow the precedent already set for VRAM estimates. The listing now reports only has_variants, a length check on loaded metadata that touches nothing, and GET /api/models/variants/:id returns the description for one entry, mirroring estimate/:id in route shape, auth and error handling. DescribeVariants itself is unchanged; only its caller moved. The picker fetches lazily at the two points where a user asks to see variants, opening the split-button menu and expanding the detail row, and caches per entry for the page session. An entry declaring no variants issues no request at all. A spec counts real HTTP hits on the weight files, so it goes red if description becomes reachable from the listing path again through any caller. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(ui): filter the model gallery to entries that declare variants The gallery is heading towards showing parent entries and hiding the individual builds they reference, so a user sees one row per model rather than six quantizations of it. Adoption is a single entry today, so defaulting to that would leave a one-row gallery. This ships the migration-phase inverse instead: the default is untouched, and a toggle narrows the list to only the entries that declare variants. It previews the end state and changes nothing until someone asks for it. The filter is server-side, next to term/tag/backend/capability and above the pagination arithmetic. The listing paginates at 9 items, so narrowing on the client would leave totalPages and availableModels describing the unfiltered set and hand the user empty pages. It selects on HasVariants(), which reads already-loaded metadata, so it issues no variant probes. The parameter is named has_variants after the listing field it selects on, and is compared against "true" like the other boolean query params (all_users, save_checkpoint), so has_variants=false reads as absent. With it omitted the response is byte-for-byte what it was before. The control is the shared Toggle component, matching the fitsFilter toggle already on this page: same wrapper class, same icon and label shape, same localStorage persistence. Unlike fitsFilter it resets to page 1 on change, which a server-side filter has to do. Stacking the toggle with a tag or backend filter easily yields nothing while one entry declares variants, so the empty state now names the variants filter as the cause rather than leaving a user to conclude the gallery is broken. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(ui): render gallery model descriptions as Markdown Gallery descriptions are Markdown, but the React UI dumped them raw, so a model whose description opens with an ATX heading showed a literal "# Qwen3.6-27B [](https://chat.qwen.ai)" in the list. Full-description areas now render through renderMarkdown (marked + DOMPurify), matching how Backends.jsx and the Manage detail panels already handle the same content: - Models.jsx expanded detail row - VoiceLibrary.jsx voice detail header The truncated one-line previews must not render block Markdown: a leading "#" would become an

and wreck the row height and rhythm. They get a new stripMarkdown() helper instead, which reduces Markdown to a single line of readable plain text. It is used for the cell text and for the title tooltip, since a tooltip full of "[](url)" is no better than a cell full of it: - Models.jsx gallery table description cell - Manage.jsx model and backend resource-row descriptions stripMarkdown walks marked's lexer output rather than running regexes over the source, so what it strips is by construction what renderMarkdown would have rendered, and it needs no new dependency. Output lands in JSX text nodes, so React escapes it; no new dangerouslySetInnerHTML beyond the two full-description sites, both of which run DOMPurify. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(ui): strip Markdown from the backends table description cell Commit b35d630cf fixed this for gallery models but left the Backends admin page with the same asymmetry: its detail panel renders the description through renderMarkdown, while the collapsed table row dumped the raw gallery string into both the cell body and the title tooltip. That is user-visible. 40 of the 949 entries in backend/index.yaml carry Markdown - insightface uses inline code backticks, others use lists and links - and backend descriptions also contain embedded newlines, so the one-line cell showed literal syntax. The cell now runs stripMarkdown over the description once and uses the result for the text and the title, matching Models.jsx and the ResourceRowDesc component in Manage.jsx. The '-' placeholder is preserved, and now also fires when a description reduces to nothing after stripping. The detail panel is untouched and no new dangerouslySetInnerHTML is introduced: stripMarkdown output lands in a JSX text node, so React escapes it. Three Playwright specs cover it: a description with a heading, inline code and a link renders as clean text with no literal syntax and no block element in the cell, the title tooltip carries the same stripped text, and a backend without a description still shows the placeholder. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * ui(models): polish the variant detail view and scope rendered Markdown The gallery detail pane rendered every field through the same two-column label/value row, including the description. Multi-paragraph prose in a value cell ran eight rows tall at the top of the pane on a ~1200px measure, breaking the grid's rhythm exactly where the eye enters. Move it into its own full-width block above the table, capped at a 68ch measure, keeping the label. Rendered Markdown had no scoped typography anywhere in the app, so a description opening with `#` inherited the browser default 2em inside a 13px surface while a `##` further down was indistinguishable from body text. Add a reusable .markdown-body block mapping h1-h6, paragraphs, lists, links, code, blockquotes, images and tables onto the existing type scale, and apply it to every renderMarkdown() consumer: the models detail, the backends detail, both Manage details and the voice library detail. Rebalance the variants list so the name leads. Backend and size drop from badge/secondary weight to muted metadata; the FITS badge goes entirely, since it was true of nearly every row and so said nothing, while the variant that does not fit keeps a warning badge and a dimmed name. AUTO-SELECTED stays marked because it answers what a plain Install produces. Rows share the parent's grid tracks via subgrid so name, backend, size and status line up down the list instead of raggedly following name length. Finally, make each variant row actionable. It looked like a list of choices but was inert text, with per-variant install hidden behind the split-button chevron elsewhere; each row is now a button onto the existing handleInstall(modelId, variant) path, with hover, keyboard focus and a disabled state while an install is in flight. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): collapse the listing to one row per model The listing supported has_variants=true, which narrowed to entries that DECLARE variants. With adoption at three entries that showed three rows, which is useless; it was always a placeholder. Replace it with the view that is actually useful: the deduplicated gallery. Show every entry installable in its own right and nothing twice, which means the parents plus every entry nobody references, and hide only the builds another entry already offers as a variant, since those are reachable through their parent. The parameter is renamed to collapse_variants accordingly: the filter is no longer a predicate on a row's own metadata but a view over the whole gallery. Default stays off, so the response with the parameter absent is unchanged. VariantReferencedIDs never reports an entry that declares variants of its own, so parents are always visible. That guarantees every hidden entry has a visible entry offering it, and no chain can strand a row. Variant resolution already refuses to install such a reference, but the listing has to stay coherent in the presence of a gallery that has one rather than silently swallowing entries. Self-references and dangling references hide nothing. The referenced set is computed over the whole gallery rather than over what the other filters left, so an entry is hidden because a parent offers it and never because of what the user searched for. The pass is over metadata already in memory: it resolves nothing over the network and triggers no variant description or size probe, so the listing's zero-probe contract still holds. The UI toggle keeps its behaviour (persistence, page reset, clear filters) and becomes "One row per model", which says what the user gets. Its localStorage key moves too, since the stored value meant a different filter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(ui): show the collapsed model listing by default The gallery listing is what a user reaches for to answer "what can I install". Answering that with several rows for the same model, one per build, makes the reader do the deduplication the collapsed view already does, so the collapsed view is the one to land on. The UI now asks for collapse_variants=true unless the toggle says otherwise. The server default is deliberately untouched: a request with the parameter absent still returns the full listing, because other API clients depend on that response and collapsing it under them would be a breaking change. Opting out omits the parameter rather than sending false, so it asks for exactly the listing everyone else gets. The stored preference changes vocabulary from '1'/'0' to 'on'/'off'. The previous build wrote it from an effect that runs on mount, so a stored '0' recorded that the page had been opened rather than that anyone chose the expanded view, and honouring it would pin every earlier visitor to a default they never picked. Only the new vocabulary counts as a choice; a legacy '1' meant the collapsed view and is what the new default gives anyway, so no earlier deliberate choice is lost. Collapsing being the default also changes what the empty state may say about it. An opted-into filter can be named as the cause of an empty result; a default cannot, so the filters keep the top line and the collapsed view drops to a hint below it, shown only once filters are narrowing the set. For the same reason "Clear filters" now restores the collapsed default instead of switching it off, and the toggle alone no longer counts as a filter worth offering to clear. The label stays "One row per model": it describes the view the user is looking at rather than an action, so it reads the same whether it is opted into or out of. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * gallery: group alternative builds of the same weights under variants Sweep the gallery for entries that are alternative builds of the same weights (different quantization, precision, or runtime format) and declare them as variants of a single parent row, so the listing offers one row per model instead of one row per quantization and the installer picks the largest build that this host can actually run. 41 families over 95 entries, turning 54 entries into variants. The parent is the bare-named entry wherever one exists, so nothing changes about what any existing entry installs. Ranking already selects the largest fitting build regardless of which entry is nominally the parent, so the parent only decides the pathological case where nothing fits. For the ten families that have no bare-named entry, the smallest build is the parent, since that is the one that has to install when nothing fits. Grouping was verified against the actual model filenames rather than the entry names alone. Different parameter sizes, languages, finetunes, and products that merely share a name prefix are left as separate rows: the qwen3.6 APEX and pi-tune finetunes, the DFlash and MTP speculative-decoding pairings, English-only versus multilingual Whisper, the QAT versus non-QAT Gemma 4 weights, and the abliterated FLUX build are all distinct models. Six parents define YAML anchors that other entries pull in with a merge key, which would have handed their variants to every merging child. For the two depth-anything anchors that would have made fourteen unrelated entries advertise the base model's builds as their own. All 26 merging children therefore carry an explicit empty variants list, which overrides the merged key and is equivalent to the key being absent. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): rank model variants by host backend preference Variant auto-selection filtered candidates by whether their backend can run on the host, then ranked the survivors by size alone. The backend never influenced the choice beyond that gate, so a Mac offered both an MLX build and a llama.cpp build kept neither filtered and installed whichever was larger, leaving the native accelerated runtime unused. The same held for CUDA against CPU on NVIDIA and ROCm against Vulkan on AMD. Rank by the host's backend preference between the fit tier and size: fit stays a filter, preference decides among the builds the host can equally hold, and size still separates builds on equally preferred runtimes. The preference data stays in one declarative table in pkg/system, now read by a prefix lookup instead of a switch, so adding a capability or reordering one host's runtimes is a one-line edit and the gallery's ranking code carries no per-backend branching. MLX joins the metal rule ahead of metal itself, which is inert for the existing alias-resolution consumer because no alias group holds a candidate named for mlx. An unrecognised backend, an unrecognised capability and an absent preference list all collapse to the previous size-only ordering rather than erroring or dropping candidates. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(gallery): rank variants by engine name, not backend build tag Variant auto-selection ranked candidates with SystemState.BackendPreferenceTokens, but that function and the variant ranker speak different vocabularies. BackendPreferenceTokens returns BUILD TAGS ("cuda", "rocm", "sycl", "vulkan", "metal", "cpu"). It exists to match installed backend build directory names like "llama-cpp-cuda-12" during alias resolution in ListSystemBackends. Variant ranking instead matches a gallery entry's `backend:` value, which is an ENGINE NAME: "llama-cpp", "vllm", "vllm-omni", "sglang", "mlx" and the rest. No engine name in gallery/index.yaml contains "cuda", "rocm", "sycl" or "vulkan". preferenceRank matches by substring, so on an NVIDIA host the tokens [cuda, vulkan, cpu] matched neither "llama-cpp" nor "vllm", every candidate scored identically and size alone decided. The NVIDIA, AMD, Intel, darwin-x86 and vulkan rules were all inert. Only metal appeared to work, and only because the token "mlx" happens to equal an engine name. The mismatch does not error, it silently deletes the feature. Separate the two vocabularies. backendBuildTagPreferenceRules keeps the build tags and its original output for every capability, including metal, whose "mlx" token is removed again; its alias-resolution consumer is byte-identical to before. engineNamePreferenceRules is new, holds engine names, and is read by the new EnginePreferenceTokens, which HostResolveEnv wires into the renamed ResolveEnv.EnginePreference. Both tables sit adjacent under one block comment naming each vocabulary and each consumer, and share one lookup helper so their semantics cannot drift. On NVIDIA the order is vLLM, then SGLang, then llama-cpp: vLLM is the throughput engine and a model published with a vLLM build is published that way because that build is the one worth running. AMD and Intel get the same order, since rocm and intel builds of both serving engines ship. Metal prefers mlx over llama-cpp. Vulkan prefers llama-cpp, the only LLM engine with a Vulkan build. darwin-x86 and unknown capabilities are deliberately absent rather than guessed at, degrading to the size-only ordering that predates preference. preferenceRank stays generic and names no engine and no capability, so adding a runtime remains a one-line table edit. Specs pin the NVIDIA and metal rules through the live table and the real HostResolveEnv wiring, so emptying the engine table or wiring the build tag source back in both go red. A regression table asserts BackendPreferenceTokens' original output per capability, and mirrored locks assert neither table carries the other's vocabulary. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * docs: record that variant selection ranks by engine before size A gallery entry can now declare variants, and selection ranks the builds a host can run by engine preference before size. Nothing told a contributor adding a backend that engineNamePreferenceRules exists, so a new engine would silently rank below every known one and lose to whatever build happened to be larger on hosts where it should have won. Document the step where a backend is added, warn against the sibling backendBuildTagPreferenceRules table (build tags, not engine names: the wrong table matches nothing, scores every candidate equally and disables the preference without erroring), and index it from AGENTS.md. Fix the authoring and user docs, which still claimed the largest surviving build wins. An author grouping builds under one entry has to be able to predict what a user gets, and size alone no longer decides it. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(cli,mcp): describe variant auto-selection as preference before size The CLI flag help and the install_model tool schema both still said auto-selection takes the largest build that runs. Ranking now puts engine preference ahead of size, so on NVIDIA a vLLM build wins over a larger llama.cpp one. An assistant reading the old schema would tell users the wrong thing. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(gallery): prefer llama.cpp over GPU serving engines on hosts with no GPU engineNamePreferenceRules had no row for the "default" capability, which getSystemCapabilities() returns both when no GPU is detected and when a GPU is present but under the 4 GiB VRAM floor. A missing row yields an empty preference list, which preferenceRank reads as "score everything equally", collapsing variant selection to size alone. That would be harmless if the hardware filter dropped GPU serving engines on such a host, but it does not. IsBackendCompatible derives support from the engine NAME, and "vllm" and "sglang" contain none of the darwin, cuda, rocm or sycl tokens it keys on, so they fall through to its closing "return true". A vLLM variant therefore survives on a CPU-only box and wins whenever its build is the larger of the two on offer: the machine installs vLLM in preference to llama.cpp. darwin-x86 had the identical hole. It was documented as a deliberate omission because nothing accelerates on an Intel Mac, which is true about acceleration and wrong about consequence: with every engine tied, download size decides. Add rows for both putting llama-cpp first. The GPU engines are enumerated behind it rather than left unmatched: an unmatched engine already ranks below every listed one, so llama.cpp would win either way, but unmatched engines also tie with each other and let size decide among them. Naming them fixes that order. MLX is left off the darwin-x86 row on purpose so it ranks last, since IsBackendCompatible admits darwin-tokened engines on that capability even though MLX needs Apple silicon. Preference orders survivors and never filters, so a model published only as a vLLM build is still installed on a host with no GPU; there is a spec for it. Surveyed every other value getSystemCapabilities() can return. nvidia, amd, intel and vulkan have rows; the l4t and cuda-refined values reach the nvidia row by prefix; "apple" and "" cannot reach the vendor fallthrough because the darwin and no-GPU branches return earlier. default and darwin-x86 were the only live holes. BackendPreferenceTokens and its build-tag table are untouched, and preferenceRank stays generic, naming no engine and no capability. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * gallery: prefer speculative-decoding builds when they fit Rank serving features between engine preference and size, so a host that can hold a DFlash or MTP build of a model's weights installs it instead of the plain build. Both answer faster for the same output, so whenever one survives the filters there is no reason to take the plain build. Precedence is now fit, then engine, then serving feature, then size. Engine outranks the feature deliberately: a serving feature makes the right engine faster, it does not make a wrong engine right, so a plain vLLM build still beats a DFlash llama.cpp build on NVIDIA. Fit outranks both, and a drafter pairing is strictly larger than the plain build, so the existing size filter drops it on a host too small for it before this axis is consulted. The order lives in a third preference table in pkg/system, alongside the build tag and engine name tables. It is the odd one of the three: not keyed by capability, because no hardware prefers a plain build over an equivalent faster one, and matched against whole segments of a gallery ENTRY NAME rather than as a substring of a backend value. Nothing on a gallery entry declares a serving feature, and tags are not a usable substitute: gemma-4-e2b-it:sglang-mtp carries an mtp tag while ornith-1.0-9b-mtp and qwen3.6-27b-nvfp4-mtp carry none. Entry names are author-supplied free text, unlike the closed engine vocabulary, so a short marker can turn up inside an unrelated word and whole segment matching is what keeps smtp-assistant from ranking as an MTP build. The block comment over the tables now documents all three together and states what each is matched against; the ranking code names no feature, so adding one stays a one-line edit to the table. 29c49203b rejected these entries as serving configurations rather than alternative builds of the same weights. The definition is now "alternative ways to serve the same model", which includes them, so regroup 14 entries under 12 parents. Judged by the files each entry points at: the qwen3.6, qwen3.5, qwen3 and deepseek pairings are the base GGUF plus a drafter, the gemma-4 QAT MTP entries are the same QAT weights at a different quantization plus an MTP drafter, and the two sglang MTP entries describe themselves as the same model served with speculative decoding. Left separate: qwen3.6-27b-mtp-pi-tune, a finetune with its own weights, and every entry whose base model LocalAI does not ship as its own row, which is the whole Qwopus line plus gemmable-4-12b-mtp, mimo-7b-mtp:sglang and qwen3.5-4b-dflash. None of the twelve parents defines a YAML anchor, so no variants key can leak through a merge key and no empty override was needed this time. The index was edited by line insertion only. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * test: check env restore errors in capability and variant specs errcheck flagged ten unchecked os.Setenv and os.Unsetenv returns in the specs added while the pre-commit hook was being skipped. Restoring an env var is exactly the place a silent failure leaks state into the next spec, so assert on it rather than suppressing the linter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): make the mtp tag authoritative for serving-feature ranking Variant auto-selection ranks survivors by fit, then engine, then serving feature, then size. The serving-feature lookup read only whole alphanumeric segments of a variant's entry name, because tags were inconsistent: every dflash entry carried a dflash tag, but only 7 of 20 MTP entries carried an mtp tag. Tag the 13 untagged MTP entries, then teach the lookup to read tags as well as names. A tag is now the authoritative signal and is compared whole and case-insensitively, which is safe precisely because a tag is a deliberate declaration rather than free text: there is no word-inside-a-word failure mode, so the segment splitting the name half needs is unnecessary there. The name check stays as a fallback rather than being replaced. Switching to tags only would have regressed the six already-grouped entries on the day it shipped, and would depend on tagging discipline that does not exist yet. The lookup still names no feature, so adding one remains a one-line edit to servingFeaturePreferenceTokens. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): make a declared tag the sole serving-feature signal Variant auto-selection ranks survivors by fit, then engine, then serving feature, then size. The serving-feature lookup recognised a speculative build by either a declared tag or a whole segment of its entry name. Drop the name half: a tag is now the only signal. A name is author-supplied free text and a naming convention is not a contract, so reading a marker out of one infers a capability nobody declared. The gallery already had the failure in it: the four NVFP4 entries name MTP-bearing weights while setting no option that enables speculative decoding, and being live variants they were winning the feature axis without answering any faster. overrides.options was considered as the replacement and rejected. It carries spec_type:draft-mtp / spec_type:draft-dflash, which is what actually turns the feature on, but that spelling is llama.cpp's config vocabulary: ds4 spells the same feature mtp_path and sglang spells it speculative_algorithm in a referenced config. Keying a cross-backend ranking decision on one backend's option syntax would rank the other backends' builds as plain. Options are the curation-time check instead, and never reach the selection logic. With no fallback left, tag correctness is load bearing, so audit every entry against the rule "tagged when the entry configures that feature, in whatever vocabulary its backend uses". Three entries configure MTP untagged and gain the tag (hy3, glm-5.2, qwythos-9b-claude-mythos-5-1m, all spec_type:draft-mtp with no marker in their names). Four carry the tag while configuring nothing and lose it: qwen3.6-27b-nvfp4-mtp, qwen3.6-35b-a3b-nvfp4-mtp, qwopus3.6-27b-coder-mtp-nvfp4 and qwopus3.6-27b-v2-mtp-nvfp4, whose only option is use_jinja:true. The dflash side was checked independently rather than assumed consistent: all five dflash entries declare spec_type:draft-dflash and all five are tagged, so it needed no edits. Four entries keep a tag that a literal spec_type-only reading would strip, because they configure MTP through a different backend: deepseek-v4-flash-q2-mtp via ds4's mtp_path/mtp_draft, and the three sglang entries via speculative_algorithm in their referenced configs. Stripping those would contradict the reason spec_type was rejected as the signal and would demote four genuinely faster builds to plain. The index was edited by line insertion and deletion only, never round-tripped through a serializer. A resolved-tag diff across all 1272 named entries, taken after merge keys are applied, shows exactly these 7 changing and no entry gaining or losing a tag through an anchor. The two specs that pinned the name fallback are inverted rather than deleted, since a name silently promoting a build is the regression worth guarding. The whole-token guard survives on the tag path, where smtp must still not match mtp. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(gallery): make deepseek-v4-flash variant targets installable Clicking install on deepseek-v4-flash failed with "invalid gallery model". The parent entry is fine, but all four entries it was grouped with declared neither url: nor config_file:, and applyModel needs one of the two to have anything to build a config from. They carry urls: (plural), the informational HuggingFace link list, which is a different field. None of the four was ever independently installable, so grouping them routed a previously-working install into a broken entry. Give each the url: the parent already resolves through. virtual.yaml is a no-op base, and applyModel passes overrides to InstallModel separately from the fetched config, so backend: ds4, the parameters and the ssd/mtp options all still land exactly as authored. This is the same pattern the parent and many other GGUF entries in the index already use. Add the lint rule that should have caught this. checkVariantReferences only proved a target exists and is not itself a parent, which is structural validity: an entry can exist, declare no variants, and still be uninstallable. checkVariantTargetsInstallable mirrors applyModel's precondition instead, and names the parent, the target and the missing fields, because whoever hits it is reading a gallery entry and has no reason to know applyModel exists. The two index-driven resolution specs live in their own Ordered container: an Ordered container stops at its first failure, so sharing one with the lint rules let a lint breach skip them silently. Nine further entries gallery-wide have the same defect and are unrelated to variants, so they are broken installs that predate this branch. They are left alone here rather than buried in a regression fix, and widening the rule to cover every entry is deferred with them so the gate can ratchet up in one step instead of needing a skip list. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(gallery): install entries with no url or config_file on an empty base applyModel had three branches: fetch a base config from url:, build one from an inline config_file:, or fail with "invalid gallery model". An entry declaring neither is now installed on an empty base config, with overrides: and files: supplying everything. This is what the ~345 entries pointing at gallery/virtual.yaml were already getting. That stub is five lines carrying name, description and license. description and license are overwritten from the gallery entry immediately after the fetch, and the name never reaches disk because InstallModel prefers the install name. Crucially applyModel passes model.Overrides to InstallModel as a separate argument rather than merging it into the fetched config, so nothing an author writes depends on that base existing. The fetch bought a round trip to GitHub and nothing else. That makes f4ef80173 the wrong fix, so it is unwound. The four url: lines it added to the deepseek-v4-flash variants are reverted: they are a pointless network fetch now, and the family installs without them. Relaxing the branch would hide a real authoring mistake, so a payload rule replaces the base-config rule. An entry with no url, no config_file, no overrides and no files installs nothing and would leave an empty model directory while reporting success, so it is refused by name. The caller's request counts toward the payload, because its overrides and files are merged into the install exactly as the entry's own are. urls: (plural) is the informational link list and does not count, which is what the four entries that shipped broken had and why they were still uninstallable. checkVariantTargetsInstallable asserted every variant target declares a url: or a config_file:, which is no longer true and would now reject correct authoring. checkEntriesInstallSomething pins what survives instead, and covers every entry rather than only variant targets: the hazard is a half-written stanza and a parent can be one as easily as a target. The old rule was scoped to targets precisely because nine unrelated entries would have failed a gallery-wide version; those nine are valid now, so the deferred ratchet happens here in one step. 1280 entries, zero violations. Those nine (aurore-reveil_koto-small-7b-it, lfm2-1.2b, the six liquidai_lfm2 entries and deepseek-v4-pro-q2-ssd) become installable for free. Each carries overrides: and files:, and one of them is driven through the real install path in a spec. The no-fetch spec is paired rather than bare: an assertion that nothing was fetched proves nothing unless something could have been, so a control runs the same fixture with a url: pointing at a base config that is not there and asserts the install fails. Only then does the identical fixture without the url passing mean the read was skipped. Follow-up, deliberately not here: the ~345 entries still naming virtual.yaml can drop their url:. That is 345 index edits with their own risk, and mixing them in would bury this change. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * ui(models): let search bypass the variant collapse, drop the toggle The models page collapsed the gallery to one row per model by default and offered a toggle to see every individual build. Because the collapse composed with the search term, a build another entry offers as a variant could not be found by typing its name, so the toggle was the only way to reach those builds in the UI. A user who typed a name they knew existed got "no models found", which reads as "that model does not exist". Collapse is for browsing; search is for finding. An explicit search term now bypasses the collapse in the listing handler, so a name lookup returns matching entries whether or not a parent offers them. The term is trimmed once at the top of the handler, so whitespace is neither a search nor a bypass; previously an untrimmed blank term also narrowed the listing to whatever contained a space. Tag and backend deliberately do not bypass: they refine a listing the user is still reading rather than name an entry already known to exist. That makes the toggle redundant, so it goes, along with its i18n strings in all six locales, its localStorage persistence, its participation in "Clear filters" and the empty-state hint telling users to turn it off. The hint was doubly stale: it pointed at a control that no longer exists, and it was untrue exactly when a user has a search term, since searching now sees every build. The page always requests the collapsed listing. The stored preference key is left inert rather than cleaned up: nothing reads it, so a user who had the toggle off simply gets the collapsed view. collapse_variants stays on the API, off by default, because other clients want either view and the UI dropping its control is no reason to remove a working parameter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(ui): give the models gallery filter form a deliberate structure The filter area had accreted controls into one undifferentiated flow. The "Fits in GPU" toggle and the backend select were direct children of .filter-bar, the same wrapping container as the 18 taxonomy chips, so their position was decided by how many chips happened to wrap at the current width rather than by any layout intent. At narrow widths they were pushed past the right edge of that container's horizontal scroll and became unreachable entirely. Restructure into three bands inside the house .filter-bar-group wrapper that components/FilterBar.jsx already uses on Backends and the System tabs: 1. query scope: search plus the backend select 2. taxonomy: the chip row, alone, free to wrap 3. refinements: fits-in-GPU and context size, under a hairline rule The backend select leads the chips rather than trailing them because picking a backend disables the use cases that backend cannot serve, so it gates the row below it. Fits-in-GPU and context size share a band because they are one control group: the context size is the length the VRAM estimate is computed at, and that estimate is what the fits filter tests against. Chips had no visible keyboard focus indicator. The global focus ring is wrapped in :where(), so it carries the specificity of a bare :focus-visible, ties with .filter-btn and loses on source order, leaving focused chips showing their resting drop shadow. Restate the ring where it outranks both resting and hover. Also: aria-pressed on the chips, a real label association and aria-valuetext on the context slider (it steps over an index, so it announced "2"), disabled chip styling moved off inline styles, a prefers-reduced-motion block for the chip transition, and the hard-coded English "Context:" moved into all seven locales. No behaviour change: same filters, same state, same requests. Page reset on change, localStorage persistence and "Clear filters" verified unchanged. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(ui): let the models recommendations panel fade into the background The "Recommended for your hardware" strip rendered at full height on every visit regardless of how many models were already installed, costing 186px at 1600px wide (287px at 1100px, where its cards wrapped to two rows) and pushing the first gallery row to y=554 / y=703. Make its prominence track how much the user still needs it. The panel now defaults to a one-line summary once anything is installed, and both the collapse choice and the existing dismissal persist: collapsed = explicit user choice, if one exists : installedCount > 0 The preference is three-valued on purpose. A boolean cannot tell "the user expanded it" apart from "the user has never chosen", and those need opposite handling when the installed count later crosses zero: someone who deliberately opened the panel on an empty instance should not have it collapse out from under them when their first model finishes installing. Collapsed keeps the card, icon, title and a suggestion count, so the panel is recovered by clicking what you are already looking at rather than by hunting. Expanded is unchanged, because for a user with nothing installed it was never the problem. Collapsed reclaims 145px at 1600 and 420, and 246px at 1100. Models.jsx gains a statsLoaded flag: stats initializes to installed:0, so reading it before the fetch resolves would render expanded and collapse a frame later, which is exactly the layout shove this removes. The dismissal key moves to the page's localai-models-* convention; the old localai_rec_models_dismissed is still read, never written, so an existing dismissal is honoured rather than resurrected by the rename. Accessibility: the disclosure is a real button whose accessible name is the visible title alone, with state on aria-expanded and aria-controls resolving in both states, because the grid is hidden via the hidden attribute rather than unmounted. That also keeps the four install buttons out of the tab order while collapsed. The app's global focus ring applies; no per-component outline is added, per the warning in App.css. Reveal animates opacity and transform only, never height, and both it and the chevron rotation are disabled under prefers-reduced-motion. Only en had a recommended block, so the other six locales were falling back to English for the whole panel. Translated the complete block rather than adding one orphaned key to files that would still render the title in English. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * fix(downloader): recover from a leftover .partial on non-HTTP URIs An interrupted download leaves a `.partial` behind. The partial handling in DownloadFileWithContext gated resume on `err == nil && uri.LooksLikeHTTPURL()`, so for any URI that is not literally http(s) the branch fell through to `else if !errors.Is(err, os.ErrNotExist)`, which with a nil err is true. The download then failed with an error wrapping nil: failed to check file ".../Ternary-Bonsai-27B-Q2_g64.gguf" existence: Every gallery file URI uses `huggingface://`, so a single interrupted download made that model permanently uninstallable until someone deleted the partial by hand. The `` in the message compounded it by pointing debugging at a filesystem failure that never happened. Restructure the handling as an explicit switch over the four real states: partial exists and is resumable, partial exists and is not resumable (discard and restart, as already done for an HTTP server without range support), no partial, and a genuine stat failure. The error branch is now only reachable with a non-nil error, names the path that was actually stat'd, and wraps with %w. Discarding is required for correctness and not merely convenience: the writer opens the partial with O_APPEND, so an un-resumed download would concatenate a fresh body onto stale bytes. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(ui): tell the models gallery's variant rows apart, and let browsing see every build Both variant surfaces rendered name, backend and size. For two builds of one model that is close to no information: a variant exists precisely because the same weights are offered another way, so the backend usually matches and the sizes usually land within a few hundred megabytes. Comparing ternary-bonsai-27b-pq2 against ternary-bonsai-27b-q2-g64 meant reading two names that differ by a suffix nobody has defined anywhere in the UI. Report the quantization and the serving features on VariantView, and derive both server-side from the referenced entry rather than parsing names in the browser, so every client reads the same format out of the same file the installer will hand the backend. Quantization comes from overrides.parameters.model first, falling back to the file list. That order is load bearing: entries routinely ship a vision tower alongside the language model at a different quantization, so reading the file list first reports the mmproj's format. Matching walks `-` and `.` delimited segments right to left; `_` deliberately does not split, because it separates the parts INSIDE a quant token and splitting on it reports Q4 for a Q4_K_M build. A second, looser pass takes a segment's `_`-delimited tail, which catches the gemma-4-E2B_q4_0-it.gguf style; it runs second so a precise match can never lose to a fuzzy one further right in the name. An entry naming no format reports nothing, which is the honest answer for a backend served from a directory of weights. Features are the same tag-against-vocabulary match servingFeatureRank already ranks on, over the same host preference list. A build can therefore never be shown as faster than one selection did not actually reward, nor rewarded without being shown; a spec pins that agreement rather than trusting it. The compact dropdown gets the quantization on its meta line and the bare feature token. The detail row, which has the room, gets the quantization as its own monospaced column so precision lines up down the list, and the feature spelled out, because DFLASH names nothing to a user who has not met it. The referenced entry's description stays out of both: the detail row already renders the parent's prose above the table, and a second block per variant would push a three-variant list past a screen to restate what the columns now say precisely. The collapse toggle comes back. 462583f38 dropped it once search bypassed the collapse, on the reasoning that nothing was unreachable any more. That holds for finding a build whose name you know and does not hold for browsing: no sequence of actions enumerated the 68 builds the default view hides. Collapse is for browsing and search is for finding, and the toggle was the browsing half. It goes in the refinements band 0d4823362 established, not back among the taxonomy chips where its position depended on how many chips happened to wrap. It leads that band because it decides how many rows the other two refine over, and because unlike fits-in-GPU it is unconditional: a host with no GPU still browses. The search bypass is untouched and re-checked by a spec in the toggle's default state, since restoring the control must not restore the dead end it replaced. The empty-state hint returns but only without a search term, because a term bypasses the collapse and the hint would otherwise point at a control that cannot change the result. The stored preference reads 'on'/'off' only: an older build wrote '1'/'0' from an effect that ran on mount, so those record that the page was opened, not that anyone chose a view. Also fixes a latent flake it exposed. The collapse_variants spec compared whole response bodies byte for byte, and the listing envelope carries live host telemetry that drifts between two calls milliseconds apart, so it was asserting on the machine's memory pressure. It now compares everything the parameter governs -- the entries, their serialization and the paging -- and is green 25/25 where it was failing about one run in three. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(ui): let the models gallery show a variant's full details The variant list in an entry's expanded detail row says how the builds differ: name, backend, quantization, size, and the auto-selected, base and serving-feature markers. It cannot say what any one of them is. A variant's own description, tags, license, source links and file list are unreachable anywhere in the UI, because while the collapse is on a variant has no gallery row of its own. Give each variant row an info control that reveals its entry, rendered by the same ModelDetail a top-level row gets, so a field added to the detail view appears here too. variantData is withheld from the nested render: a variant may declare variants of its own, and recursing would nest a picker inside a picker two levels deep already. An inline disclosure rather than a modal. The control sits inside a table row that is already expanded, inside a variant list within that; a dialog opened from there stacks a dismissal on a dismissal for a handful of extra fields about the entry the user is already reading, and breaks the page's own expand idiom. The third level is carried by an inset and a left rule instead of another card. The entry is fetched by exact name from the listing, once, on first use. The listing already returns every field the detail view renders, and a search term bypasses the variant collapse server-side, so no new endpoint is needed and neither the listing nor DescribeVariants gains any work. Expanding a row costs nothing; a variant nobody opens costs nothing. A name the listing no longer returns is stated, not blanked: an empty panel reads as a rendering fault rather than as a lookup that came back empty. The control is a sibling of the install button, not a descendant, so asking about a build can never install it. The variant list keeps its content-sized columns via a trailing filler track instead of max-content sizing, so the rows are unchanged while the panel spanning them gets the pane width its file table needs. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto * feat(gallery): let search respect the collapse instead of switching it off The models listing collapsed to one row per model, and an explicit search term turned that off wholesale. Searching while collapsed therefore answered with the individual builds a parent already offers, which are exactly the rows the view the user asked for has no place for: typing "mtp" returned qwen3.6-27b-nvfp4-mtp, a row that is invisible the moment the box is cleared. The bypass was the right shape of fix for the wrong half of the problem. What a search must not do is answer "no models found" for a build the gallery does hold; that does not require abandoning the grouping the user asked for. So the term is now matched against every entry either way, hidden builds included, and the collapse decides how a match is reported rather than which matches exist. Collapsing stops being a filter that drops rows and becomes a substitution: a match on a build another entry offers is reported as that entry, the one installable in its own right. Nothing becomes unfindable and nothing comes back that the requested view cannot show. Substitution happens after search, tag and backend, so every filter is judged against the build that really carries the name, tag or backend rather than against a parent that merely offers it; the other order would let backend=vllm match a parent whose own backend is something else. The price is that the surfaced row shows the parent's own metadata while the match was on a variant, which is what grouping means, and the alternative is claiming the gallery holds no such build. It happens before the count and the page math, so both describe the rows actually handed out rather than the matches that produced them. A parent already in the result keeps its own position and absorbs its matching variants there, which is what leaves the browsing listing ordered exactly as it was; a parent surfaced only by a variant takes the position of the first variant that surfaced it. Either way it appears once, however many of its builds matched and whether or not it matched itself. Search preserves gallery order rather than scoring, so a surfaced parent has a real position rather than an invented one. VariantParents never reports an entry that declares variants of its own, so a parent is never itself hidden and one hop always lands on a visible row. The handler follows exactly one anyway: refusing the second is what makes a gallery the linter would have rejected terminate rather than loop. The empty-state hint pointing at the toggle goes with it for every server-side filter. Substitution means a match is always reported as some row, so the collapse can no longer be why a term, a chip or a backend came back empty, and naming it there sends the user to a control that cannot change the result. It survives for the fits filter alone, which runs in the browser after the substitution and judges the surfaced entry's own size: there the build that fits really can be filtered out along with a parent that does not. Searching a build's exact name while collapsed now answers with its parent, so the result no longer contains the string the user typed. That is intended, and the row is the one they can act on, but it is a real rough edge: nothing on the row explains the connection. Closing it properly means reporting which variant matched so the UI can say so, which the listing does not do today. ResetGalleryModelCache is added for tests. The model cache is a package global keyed by nothing, so a background refresh one spec triggers can land in the middle of the next and answer it with the previous spec's gallery; the extra specs here made that fail about one run in five. It waits for the in-flight refresh to publish before clearing, since clearing alone only narrows the window. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto --------- Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- .agents/adding-backends.md | 64 + .agents/adding-gallery-models.md | 102 ++ AGENTS.md | 1 + core/cli/models.go | 7 +- .../gallery/available_memory_internal_test.go | 73 + core/gallery/collapse_variants.go | 109 ++ core/gallery/collapse_variants_test.go | 157 ++ core/gallery/describe_variants.go | 120 ++ core/gallery/describe_variants_test.go | 369 +++++ core/gallery/empty_base_install_test.go | 243 ++++ core/gallery/gallery.go | 21 + core/gallery/model_artifacts.go | 11 + core/gallery/models.go | 409 +++++- core/gallery/models_types.go | 38 + core/gallery/resolve_variant.go | 435 ++++++ core/gallery/resolve_variant_test.go | 1089 ++++++++++++++ core/gallery/variant.go | 18 + core/gallery/variant_quantization.go | 147 ++ .../variant_quantization_internal_test.go | 126 ++ core/gallery/variants_install_test.go | 757 ++++++++++ core/gallery/variants_lint_test.go | 400 +++++ core/http/endpoints/localai/gallery.go | 6 + .../react-ui/e2e/backends-management.spec.js | 51 + core/http/react-ui/e2e/models-gallery.spec.js | 1290 ++++++++++++++++- .../e2e/models-recommended-panel.spec.js | 148 ++ .../react-ui/public/locales/de/models.json | 40 +- .../react-ui/public/locales/en/models.json | 32 +- .../react-ui/public/locales/es/models.json | 40 +- .../react-ui/public/locales/id/models.json | 40 +- .../react-ui/public/locales/it/models.json | 40 +- .../react-ui/public/locales/ko/models.json | 18 +- .../react-ui/public/locales/zh-CN/models.json | 40 +- core/http/react-ui/src/App.css | 499 ++++++- .../src/components/RecommendedModels.jsx | 79 +- core/http/react-ui/src/pages/Backends.jsx | 26 +- core/http/react-ui/src/pages/Manage.jsx | 23 +- core/http/react-ui/src/pages/Models.jsx | 666 +++++++-- core/http/react-ui/src/pages/VoiceLibrary.jsx | 5 +- core/http/react-ui/src/utils/api.js | 12 +- core/http/react-ui/src/utils/config.js | 1 + core/http/react-ui/src/utils/markdown.js | 47 + core/http/routes/ui_api.go | 154 +- core/http/routes/ui_api_variants_test.go | 528 +++++++ core/schema/gallery-model.schema.json | 16 + core/services/galleryop/managers_local.go | 6 +- core/services/galleryop/model_variant_test.go | 107 ++ core/services/galleryop/models.go | 13 +- core/services/galleryop/operation.go | 9 + core/startup/model_preload.go | 11 +- docs/content/features/model-gallery.md | 144 ++ gallery/index.yaml | 163 +++ pkg/downloader/partial_resume_test.go | 173 +++ pkg/downloader/uri.go | 37 +- pkg/mcp/localaitools/dto.go | 1 + pkg/mcp/localaitools/httpapi/client.go | 3 + pkg/mcp/localaitools/httpapi/client_test.go | 33 + pkg/mcp/localaitools/inproc/client.go | 1 + .../prompts/skills/install_chat_model.md | 2 +- pkg/mcp/localaitools/tools_models.go | 2 +- pkg/system/capabilities.go | 271 +++- pkg/system/capabilities_test.go | 225 ++- pkg/vram/estimate.go | 17 +- pkg/vram/hf_estimate_test.go | 5 + 63 files changed, 9529 insertions(+), 191 deletions(-) create mode 100644 core/gallery/available_memory_internal_test.go create mode 100644 core/gallery/collapse_variants.go create mode 100644 core/gallery/collapse_variants_test.go create mode 100644 core/gallery/describe_variants.go create mode 100644 core/gallery/describe_variants_test.go create mode 100644 core/gallery/empty_base_install_test.go create mode 100644 core/gallery/resolve_variant.go create mode 100644 core/gallery/resolve_variant_test.go create mode 100644 core/gallery/variant.go create mode 100644 core/gallery/variant_quantization.go create mode 100644 core/gallery/variant_quantization_internal_test.go create mode 100644 core/gallery/variants_install_test.go create mode 100644 core/gallery/variants_lint_test.go create mode 100644 core/http/react-ui/e2e/models-recommended-panel.spec.js create mode 100644 core/http/routes/ui_api_variants_test.go create mode 100644 core/services/galleryop/model_variant_test.go create mode 100644 pkg/downloader/partial_resume_test.go diff --git a/.agents/adding-backends.md b/.agents/adding-backends.md index 969684a7d..a7880a17b 100644 --- a/.agents/adding-backends.md +++ b/.agents/adding-backends.md @@ -218,6 +218,69 @@ docker-build-backends: ... docker-build- - If the backend is in `backend/python//` but uses `.` as context in the workflow file, use `.` context - Check similar backends to determine the correct context +## Engine preference for gallery model variants + +A gallery entry can declare `variants`, alternative builds of the same weights, +and LocalAI picks one per host: it drops builds whose backend cannot run here or +that do not fit memory, then ranks the survivors by **engine preference +first, serving feature second, size third** (`SelectVariant` in +`core/gallery/resolve_variant.go`). + +Ask whether your backend should outrank another one on some hardware. If it +should, add it to `engineNamePreferenceRules` in `pkg/system/capabilities.go`, +best engine first for that capability: + +```go + {Nvidia, []string{engineVLLM, engineSGLang, engineLlamaCpp}}, ++ {Nvidia, []string{engineVLLM, engineSGLang, engineMyEngine, engineLlamaCpp}}, +``` + +That is the ENGINE NAME table, matched as a substring of a gallery entry's +`backend:` value. Two sibling tables in the same file speak different +vocabularies and are matched against different things: + +| Table | Vocabulary | Matched against | Consumer | +|-------|-----------|-----------------|----------| +| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` | +| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking | +| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole and case-insensitively, and nothing else | gallery variant ranking, one rank below the engine | + +**Putting a token in the wrong table matches nothing and does not error**: every +candidate scores equal and the next sort key decides, so the preference silently +stops existing. The block comment above all three tables spells the contract out. + +The serving feature table is the odd one: it is not keyed by capability, because +no hardware prefers a plain build over an equivalent faster build of the same +weights. It reads a declared tag and nothing else. The entry name was the +original signal and is gone: a naming convention is not a contract, and names +are author-supplied free text where a short marker like `mtp` turns up inside +unrelated words or on weights whose entry enables nothing. +`overrides.options` was rejected for the mirror-image reason: `spec_type:` is +llama.cpp's config vocabulary, whereas a cross-backend ranking decision must +work the same for `ds4`'s `mtp_path:` and `sglang`'s `speculative_algorithm:`. + +**If your backend can serve the same weights faster** (speculative decoding, +multi-token prediction), say so in the docs for its gallery entries so curators +tag them: the tagging rule and the per-backend evidence table live in +[adding-gallery-models.md](adding-gallery-models.md). A backend never needs to +appear in the token table itself; it ranks builds, not engines. + +Leaving your backend out is a valid choice when no ordering can be justified for +it. It then ranks below every known engine and selection falls back to size, +which is the behaviour that predates preference. + +**Leaving a whole capability out is not.** A missing row gives that host an +empty preference list, so size alone decides among everything that survives the +filters, and the filter will not save you: `IsBackendCompatible` derives hardware +support from the engine NAME, so `vllm` and `sglang` carry no darwin, cuda, rocm +or sycl token and are never dropped on a host with no GPU. That is why `default` +(no usable accelerator, including a GPU under the 4 GiB VRAM floor) and +`darwin-x86` both have rows putting `llama-cpp` first. Every capability +`getSystemCapabilities()` can return needs a row unless every engine really is +equally at home there. When you add one, enumerate the engines you are demoting +rather than relying on them falling through unmatched: unmatched engines all tie +with each other, so size decides among them. + ## Documenting the backend (README + docs) A backend is not "added" until it is discoverable. Update the user-facing docs: @@ -254,6 +317,7 @@ After adding a new backend, verify: - [ ] No Makefile syntax errors (check with linter) - [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern) - [ ] **`Load` validates its input and refuses models it can't serve.** When a model config has no explicit `backend:`, the model loader greedily probes *every* installed backend with the model's name and binds to the first `Load` that succeeds — an accept-anything `Load` will capture arbitrary LLMs (issue #9287). Backends that load a real artefact get this for free (the load fails); backends with no artefact must gate on the name: `opus` accepts only its own name (or none), `local-store` requires the `store.NamespacePrefix` namespace marker sent by `core/backend/stores.go`. +- [ ] **Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`, NOT `servingFeaturePreferenceTokens`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets the next sort key decide. - [ ] Documented: added to the category list in `docs/content/features/backends.md` (and any new endpoint/realtime capability documented under `docs/content/`) - [ ] If it is an in-house native C/C++/GGML engine, added to the maintained-engines table in the top-level `README.md` diff --git a/.agents/adding-gallery-models.md b/.agents/adding-gallery-models.md index fe5a57e87..51221608f 100644 --- a/.agents/adding-gallery-models.md +++ b/.agents/adding-gallery-models.md @@ -91,6 +91,108 @@ To add a variant (e.g., different quantization), use YAML merge: uri: huggingface:////-Q8_0.gguf ``` +## Offering several builds of one model (`variants`) + +When the same model is published in more than one quantization, or is also +servable by another engine, add each build as its own ordinary gallery entry and +then point one of them at the others with `variants`: + +```yaml +- !!merge <<: *chatml + name: "nanbeige4.1-3b-q4" + # ... the usual urls / overrides / files for the Q4 build ... + variants: + - model: nanbeige4.1-3b-q8 +``` + +Rules: + +- The declaring entry is a **complete, normal entry**. It keeps its own + `files`/`overrides` and stays installable on every host and by every older + LocalAI release, which simply ignore `variants`. +- A variant references another gallery entry **by name**. That entry must exist + and must not declare `variants` of its own. +- **A referenced entry keeps its own gallery row by default.** It is hidden only + in the collapsed listing (`collapse_variants=true`, which the web UI requests + by default), where the declaring entry stands in for it. Searching there still + matches the referenced entry and answers with the entry declaring it, so + referencing an entry never makes it unfindable; turning the collapse off + returns it under its own name. +- **Order carries no meaning.** Do not try to encode a preference; write the + list in whatever order reads best. +- **A variant may be smaller than the declaring entry.** Offering a downgrade + for small hosts is a normal shape: the declaring entry's own build competes + like every other candidate, so a large host keeps the large build. +- **Do not describe hardware.** At install time LocalAI drops variants whose + backend cannot run on the host, then drops those that do not fit available + memory. The declaring entry's own build is exempt from both filters, so + selection always terminates on something installable. Sizes are measured live + from the weights and cached, so nothing has to be written down. +- **Engine preference outranks size.** Among the builds that survive the + filters, the host's preferred engine wins first and only then does the larger + footprint win. On NVIDIA a vLLM build beats a larger llama.cpp one; on Apple + silicon an MLX build beats a larger GGUF one; on a host with no preference for + either engine the larger build wins, since a bigger footprint is a higher + quality quantization of the same weights. Predict what a user gets by asking + which engine the host prefers before asking which build is biggest. The + per-capability order lives in `engineNamePreferenceRules` + (`pkg/system/capabilities.go`); see + [adding-backends.md](adding-backends.md) for how a backend gets into it. +- **Serving feature preference sits between engine and size.** Among builds on + an equally preferred engine, one that speculates or predicts several tokens + per step beats the plain build of the same weights, because it answers faster + for the same output: a `dflash` build beats an `mtp` one, and either beats a + plain build. The order lives in `servingFeaturePreferenceTokens` + (`pkg/system/capabilities.go`) and is matched against the entry's `tags:` and + **nothing else**: not the entry name, not `overrides.options`. See + [the tagging rule](#the-dflash--mtp-tagging-rule) below. Engine deliberately + outranks it: a serving feature makes the right engine faster, it does not make + a wrong engine right. Fit still outranks both, so a drafter pairing (strictly + larger than the plain build, since it ships a drafter alongside it) is dropped + on a host too small for it before this order is ever consulted. +- A variant is nothing but a name; there is no per-variant memory field. When + the measured size for a build is wrong, correct it on the referenced entry by + setting that entry's own `size:` (e.g. `size: "20GiB"`). The estimator prefers + a declared size over its own guesswork, so the fix applies everywhere the size + is shown or compared rather than only to variant selection. + +Users can override the automatic choice with `variant` on `POST /models/apply`, +`local-ai models install --variant`, or the `install_model` MCP tool. See +`docs/content/features/model-gallery.md`. + +The gallery lint specs live in `core/gallery`, so run that suite after adding a +`variants` list. + +### The `dflash` / `mtp` tagging rule + +**Tag an entry `dflash` or `mtp` when the entry actually configures that +feature. Variant ranking reads the tag and nothing else.** + +Decide by looking at what the entry configures, in whatever vocabulary its +backend uses: + +| Backend | Configures the feature when it declares | +|---------|------------------------------------------| +| `llama-cpp` | `overrides.options` contains `spec_type:draft-dflash` or `spec_type:draft-mtp` | +| `ds4` | `overrides.options` contains `mtp_path:` / `mtp_draft:` | +| `sglang` | the referenced `gallery/*.yaml` sets `speculative_algorithm:` | + +That check is curation-time only. `spec_type` is llama.cpp's config vocabulary, +and a cross-backend ranking decision must not depend on one backend's option +syntax, which is precisely why the ranker reads the tag instead of the options. + +Two mistakes the rule exists to prevent: + +- **Weights that carry the heads are not an entry that enables them.** The + NVFP4 GGUF entries ship MTP-bearing weights but set only `use_jinja:true`, so + they enable no speculative decoding and must NOT be tagged. Tagging them wins + them the feature axis without being any faster. +- **A name is not a declaration.** An entry whose name spells `-mtp` while + configuring nothing gets no tag, and an entry that configures the feature is + tagged even when its name says nothing (`hy3`, `glm-5.2`). Ranking never reads + the name, so an untagged build that does enable the feature is simply ranked + as plain rather than promoted on a marker nobody meant. + ## Available template configs Look at existing `.yaml` files in `gallery/` to find the right prompt template for your model architecture: diff --git a/AGENTS.md b/AGENTS.md index c03db98e2..393c07845 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -44,4 +44,5 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants] - **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map. - **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds - **Backend OS coverage**: a new backend must target every OS it can build for, not just Linux. `.github/backend-matrix.yml` has two matrices — `include:` (Linux) and `includeDarwin:` (macOS / Apple Silicon). Most C/C++/GGML and many Python backends build on Darwin too — wire the `includeDarwin` entry + `backend/index.yaml` `metal:` entries, or say in the PR why an OS is unsupported. See the darwin checklist in [.agents/adding-backends.md](.agents/adding-backends.md). +- **Gallery variant ranking**: a gallery entry can declare `variants` (alternative builds of the same weights), and LocalAI ranks the ones a host can run by engine preference first, size second. A new backend that should be preferred on some hardware must be listed in `engineNamePreferenceRules` in `pkg/system/capabilities.go`; the sibling `backendBuildTagPreferenceRules` speaks build tags rather than engine names, and using the wrong table matches nothing without erroring. See [.agents/adding-backends.md](.agents/adding-backends.md). - **UI**: The active UI is the React app in `core/http/react-ui/`. The older Alpine.js/HTML UI in `core/http/static/` is pending deprecation — all new UI work goes in the React UI diff --git a/core/cli/models.go b/core/cli/models.go index a95462da1..e2a832b0c 100644 --- a/core/cli/models.go +++ b/core/cli/models.go @@ -39,6 +39,7 @@ type ModelsInstall struct { RequireBackendIntegrity bool `env:"LOCALAI_REQUIRE_BACKEND_INTEGRITY,REQUIRE_BACKEND_INTEGRITY" help:"If true, reject backend installs without a configured signature verification policy (OCI URIs) or SHA256 (tarball/HTTP URIs)." group:"hardening" default:"false"` AutoloadBackendGalleries bool `env:"LOCALAI_AUTOLOAD_BACKEND_GALLERIES" help:"If true, automatically loads backend galleries" group:"backends" default:"true"` ModelArgs []string `arg:"" optional:"" name:"models" help:"Model configuration URLs to load"` + Variant string `name:"variant" help:"Install a specific variant of a gallery entry that declares them, by the variant's model name. Leave unset to let LocalAI auto-select: builds this hardware cannot run or cannot fit are dropped, the engine this hardware prefers wins, and size decides among equals." group:"models"` ModelsCMDFlags `embed:""` } @@ -147,7 +148,11 @@ func (mi *ModelsInstall) Run(ctx *cliContext.Context) error { } modelLoader := model.NewModelLoader(systemState) - err = startup.InstallModels(context.Background(), galleryService, galleries, backendGalleries, systemState, modelLoader, !mi.DisablePredownloadScan, mi.AutoloadBackendGalleries, mi.RequireBackendIntegrity, progressCallback, modelName) + var installOptions []gallery.InstallOption + if mi.Variant != "" { + installOptions = append(installOptions, gallery.WithVariant(mi.Variant)) + } + err = startup.InstallModelsWithOptions(context.Background(), galleryService, galleries, backendGalleries, systemState, modelLoader, !mi.DisablePredownloadScan, mi.AutoloadBackendGalleries, mi.RequireBackendIntegrity, progressCallback, installOptions, modelName) if err != nil { return err } diff --git a/core/gallery/available_memory_internal_test.go b/core/gallery/available_memory_internal_test.go new file mode 100644 index 000000000..e62b13800 --- /dev/null +++ b/core/gallery/available_memory_internal_test.go @@ -0,0 +1,73 @@ +package gallery + +import ( + "os" + + "github.com/mudler/LocalAI/pkg/system" + "github.com/mudler/LocalAI/pkg/xsysinfo" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("availableModelMemory", func() { + // hostRAM is the figure the fallback resolves to. A host that cannot report + // its own RAM cannot exercise the fallback at all, so those specs skip + // rather than assert on a number that is legitimately unavailable. + hostRAM := func() uint64 { + ram, err := xsysinfo.GetSystemRAMInfo() + if err != nil || ram == nil || ram.Total == 0 { + Skip("this host does not report system RAM; the RAM fallback cannot be exercised here") + } + return ram.Total + } + + // stateWithCapability forces DetectedCapability for one spec. The state is + // built fresh so nothing is served from the capability cache. + stateWithCapability := func(capability string, vram uint64) *system.SystemState { + previous, had := os.LookupEnv("LOCALAI_FORCE_META_BACKEND_CAPABILITY") + Expect(os.Setenv("LOCALAI_FORCE_META_BACKEND_CAPABILITY", capability)).To(Succeed()) + DeferCleanup(func() { + if had { + Expect(os.Setenv("LOCALAI_FORCE_META_BACKEND_CAPABILITY", previous)).To(Succeed()) + return + } + Expect(os.Unsetenv("LOCALAI_FORCE_META_BACKEND_CAPABILITY")).To(Succeed()) + }) + + s := &system.SystemState{} + s.VRAM = vram + Expect(s.DetectedCapability()).To(Equal(capability), "the capability override did not take") + return s + } + + // A unified-memory host reports a GPU capability but no discrete VRAM pool. + // Apple Silicon is the case that matters: metal is reported unconditionally + // on arm64 macs while TotalAvailableVRAM finds nothing to add up. Reading + // that zero as the budget drops every sized variant and strands the host on + // the base build however much memory it has, which is what turned the macOS + // runner red. + It("falls back to system RAM when a detected GPU reports no VRAM", func() { + ram := hostRAM() + + got := availableModelMemory(stateWithCapability("metal", 0)) + + Expect(got).ToNot(BeZero(), "a unified-memory host was given a zero budget; every sized variant would be dropped") + Expect(got).To(Equal(ram)) + }) + + // A discrete GPU reports real VRAM, and that stays the budget: the model + // lives on the card, so system RAM is not what bounds it. + It("prefers VRAM when the GPU reports it", func() { + const vram = uint64(24) << 30 + + Expect(availableModelMemory(stateWithCapability("nvidia-cuda-12", vram))).To(Equal(vram)) + }) + + // With no GPU at all the model lives in system RAM. + It("uses system RAM when no GPU is detected", func() { + ram := hostRAM() + + Expect(availableModelMemory(stateWithCapability("default", 0))).To(Equal(ram)) + }) +}) diff --git a/core/gallery/collapse_variants.go b/core/gallery/collapse_variants.go new file mode 100644 index 000000000..3e308b602 --- /dev/null +++ b/core/gallery/collapse_variants.go @@ -0,0 +1,109 @@ +package gallery + +import ( + "os" + "strings" +) + +// VariantReferencedIDs reports the IDs of entries that another entry already +// offers as one of its variants, and which therefore need no row of their own +// in a deduplicated listing: they are reachable by installing the entry that +// references them. +// +// The returned keys are GalleryModel.ID() values, so an entry is identified by +// gallery and name rather than by name alone. Two galleries may legitimately +// ship an entry of the same name, and only the one actually referenced should +// disappear. +// +// This is the key set of VariantParents; see there for the resolution rules. +func VariantReferencedIDs(models []*GalleryModel) map[string]struct{} { + parents := VariantParents(models) + referenced := make(map[string]struct{}, len(parents)) + for id := range parents { + referenced[id] = struct{}{} + } + return referenced +} + +// VariantParents maps the ID of every entry another entry already offers as one +// of its variants to the entry that offers it. It is what a collapsed listing +// needs in order to answer a match on a hidden build with the row the user can +// actually act on: the parent. +// +// An entry that declares variants of its own is never reported, even when +// something else references it. That guard is what keeps a chain from hiding a +// row the user cannot otherwise reach: parents are always visible, so every +// hidden entry is guaranteed to have a visible entry that offers it, and no +// parent this returns is itself a key. Such a reference is an authoring error +// anyway, and variant resolution already refuses to install it, but the listing +// must stay coherent in the presence of a gallery that has one rather than +// silently swallowing entries. +// +// When two entries reference the same build, the first in gallery order wins, +// so the listing is deterministic for a gallery the linter would reject. +// +// This is a pure pass over metadata already in memory. It resolves nothing over +// the network and probes no weight files: whether an entry is referenced is +// answerable from the declared names alone. +func VariantParents(models []*GalleryModel) map[string]*GalleryModel { + // FindGalleryElement resolves a reference by scanning every entry, which + // would make this quadratic over the whole gallery. The index below is the + // same lookup precomputed once, so the matching rules have to agree with + // it: case-insensitive, path separators folded to "__", and a reference + // carrying an "@" addressed against "gallery@name" instead of the bare name. + byName := make(map[string]*GalleryModel, len(models)) + byQualified := make(map[string]*GalleryModel, len(models)) + for _, m := range models { + if m == nil { + continue + } + // First match wins, mirroring FindGalleryElement's scan order, so a + // name colliding across galleries resolves the same way here as it + // does at install time. + name := variantLookupKey(m.Name) + if _, seen := byName[name]; !seen { + byName[name] = m + } + qualified := variantLookupKey(m.ID()) + if _, seen := byQualified[qualified]; !seen { + byQualified[qualified] = m + } + } + + referenced := map[string]*GalleryModel{} + for _, m := range models { + if m == nil { + continue + } + for _, v := range m.Variants { + key := variantLookupKey(v.Model) + target, ok := byQualified[key] + if !strings.Contains(key, "@") { + target, ok = byName[key] + } + // A dangling reference names nothing to hide. Reporting it is the + // linter's job, not the listing's. + if !ok || target == nil { + continue + } + if target.HasVariants() { + continue + } + // An entry naming itself would otherwise erase itself from the + // gallery entirely. + if target.ID() == m.ID() { + continue + } + if _, claimed := referenced[target.ID()]; claimed { + continue + } + referenced[target.ID()] = m + } + } + + return referenced +} + +func variantLookupKey(name string) string { + return strings.ToLower(strings.ReplaceAll(name, string(os.PathSeparator), "__")) +} diff --git a/core/gallery/collapse_variants_test.go b/core/gallery/collapse_variants_test.go new file mode 100644 index 000000000..7c5f63eac --- /dev/null +++ b/core/gallery/collapse_variants_test.go @@ -0,0 +1,157 @@ +package gallery_test + +import ( + "github.com/mudler/LocalAI/core/config" + . "github.com/mudler/LocalAI/core/gallery" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// VariantReferencedIDs answers the only question the deduplicated listing asks: +// which entries does another entry already offer, and so need no row of their +// own. Everything else is shown. +var _ = Describe("VariantReferencedIDs", func() { + entry := func(gal, name string, variants ...string) *GalleryModel { + m := &GalleryModel{} + m.Name = name + m.Gallery = config.Gallery{Name: gal} + for _, v := range variants { + m.Variants = append(m.Variants, Variant{Model: v}) + } + return m + } + + It("reports the builds a parent references and nothing else", func() { + parent := entry("test", "parent", "build-a", "build-b") + models := []*GalleryModel{ + parent, + entry("test", "build-a"), + entry("test", "build-b"), + entry("test", "standalone"), + } + + Expect(VariantReferencedIDs(models)).To(HaveLen(2)) + Expect(VariantReferencedIDs(models)).To(HaveKey("test@build-a")) + Expect(VariantReferencedIDs(models)).To(HaveKey("test@build-b")) + // The parent is installable in its own right, and a standalone entry + // is nobody's variant. + Expect(VariantReferencedIDs(models)).NotTo(HaveKey("test@parent")) + Expect(VariantReferencedIDs(models)).NotTo(HaveKey("test@standalone")) + }) + + It("never hides an entry that declares variants of its own", func() { + // A parent referencing another parent is an authoring error that + // variant resolution refuses to install. The listing still has to stay + // coherent: if the chain hid the middle entry, the build it offers + // would be unreachable, since nothing visible would mention it. + top := entry("test", "top", "middle") + middle := entry("test", "middle", "leaf") + models := []*GalleryModel{top, middle, entry("test", "leaf")} + + referenced := VariantReferencedIDs(models) + Expect(referenced).NotTo(HaveKey("test@middle"), + "hiding a parent would strand the builds only it offers") + Expect(referenced).To(HaveKey("test@leaf")) + }) + + It("ignores an entry that references itself", func() { + // Self-reference would otherwise erase the entry from the gallery. + models := []*GalleryModel{entry("test", "loop", "loop", "build")} + models = append(models, entry("test", "build")) + + Expect(VariantReferencedIDs(models)).NotTo(HaveKey("test@loop")) + Expect(VariantReferencedIDs(models)).To(HaveKey("test@build")) + }) + + It("ignores a reference that names no existing entry", func() { + // A dangling reference hides nothing; reporting it is the linter's job. + models := []*GalleryModel{entry("test", "parent", "does-not-exist")} + Expect(VariantReferencedIDs(models)).To(BeEmpty()) + }) + + It("resolves references the way install-time lookup does", func() { + // Matching has to agree with FindGalleryElement, otherwise a reference + // that installs fine would fail to hide its target. Case-insensitive, + // and an "@" addresses gallery and name together. + models := []*GalleryModel{ + entry("test", "parent", "BUILD-A", "other@build-b"), + entry("test", "build-a"), + entry("other", "build-b"), + entry("test", "build-b"), + } + + referenced := VariantReferencedIDs(models) + Expect(referenced).To(HaveKey("test@build-a")) + Expect(referenced).To(HaveKey("other@build-b")) + // The qualified reference names the other gallery's entry, so the + // same-named local entry keeps its row. + Expect(referenced).NotTo(HaveKey("test@build-b")) + }) + + It("returns an empty set for a gallery declaring no variants at all", func() { + models := []*GalleryModel{entry("test", "a"), entry("test", "b")} + Expect(VariantReferencedIDs(models)).To(BeEmpty()) + }) + + // The collapsed listing needs more than "is this hidden": to report a match + // on a hidden build it has to name the row the user can act on. + Context("VariantParents", func() { + It("names the entry that offers each hidden build", func() { + parent := entry("test", "parent", "build-a", "build-b") + models := []*GalleryModel{ + parent, + entry("test", "build-a"), + entry("test", "build-b"), + entry("test", "standalone"), + } + + parents := VariantParents(models) + Expect(parents).To(HaveLen(2)) + Expect(parents["test@build-a"]).To(BeIdenticalTo(parent)) + Expect(parents["test@build-b"]).To(BeIdenticalTo(parent)) + Expect(parents).NotTo(HaveKey("test@standalone")) + }) + + It("never names a parent that is itself hidden", func() { + // What makes substitution safe in one hop: whatever a hidden build + // resolves to is guaranteed to be a row the listing still shows, so + // no chain can substitute a user into an entry that is not there. + models := []*GalleryModel{ + entry("test", "top", "middle"), + entry("test", "middle", "leaf"), + entry("test", "leaf"), + } + + parents := VariantParents(models) + for hidden, parent := range parents { + Expect(parents).NotTo(HaveKey(parent.ID()), + "the parent of "+hidden+" is itself hidden, so substituting lands on a row nobody can see") + } + }) + + It("gives a build claimed twice a single parent, the first in order", func() { + // A gallery the linter rejects, but the listing must still be + // deterministic rather than ordered by map iteration. + first := entry("test", "first", "shared") + second := entry("test", "second", "shared") + models := []*GalleryModel{first, second, entry("test", "shared")} + + Expect(VariantParents(models)["test@shared"]).To(BeIdenticalTo(first)) + }) + + It("agrees with VariantReferencedIDs on which entries are hidden", func() { + models := []*GalleryModel{ + entry("test", "parent", "build-a"), + entry("test", "build-a"), + entry("test", "standalone"), + } + + parents := VariantParents(models) + referenced := VariantReferencedIDs(models) + Expect(parents).To(HaveLen(len(referenced))) + for id := range referenced { + Expect(parents).To(HaveKey(id)) + } + }) + }) +}) diff --git a/core/gallery/describe_variants.go b/core/gallery/describe_variants.go new file mode 100644 index 000000000..3b579d7df --- /dev/null +++ b/core/gallery/describe_variants.go @@ -0,0 +1,120 @@ +package gallery + +// VariantView is one selectable build of an entry, as a client sees it. +// +// It is the read-only mirror of what selection decides at install time, so a +// picker can show the user the same facts the installer acts on rather than a +// second, independently-derived opinion that could disagree with it. +type VariantView struct { + // Model is the name to send back as the install request's `variant`. + Model string `json:"model"` + // Backend is the engine the referenced entry resolves to. A client renders + // it, and it is also the reason Fits may be false on a host whose memory + // would be ample. + Backend string `json:"backend"` + // MemoryBytes is the measured footprint. It is omitted from the JSON + // entirely when the size could not be determined, rather than serialized as + // a zero that a client would have to know to read as unknown. An absent key + // never means "needs nothing". + MemoryBytes uint64 `json:"memory_bytes,omitempty"` + // Fits reports whether auto-selection would consider this variant on this + // host: its backend can run here and its known footprint is within budget. + // An unknown footprint counts as fitting, exactly as selection treats it. + Fits bool `json:"fits"` + // IsBase marks the entry's own payload. It is always installable and is + // what auto-selection falls back to, which is why it is listed alongside + // the declared variants rather than hidden. + IsBase bool `json:"is_base"` + // Quantization is the weight format the referenced entry installs, e.g. + // "Q2_G64", "PQ2_0", "F16". It is the fact that actually separates two + // builds of one model: name, backend and probed size routinely agree + // between variants that differ entirely in precision. + // + // Derived server-side from the referenced entry's model filename rather + // than parsed by clients, so every client reads the same format out of the + // same file the installer will point the backend at, and a naming + // convention change is one edit here rather than one per client. + // + // Omitted when the entry names no recognisable format. That is a normal + // outcome for a backend served from a directory of weights, and clients + // must render its absence rather than an empty or invented value. + Quantization string `json:"quantization,omitempty"` + // Features are the serving features this build declares, best first, e.g. + // ["dflash"]. They mean the same weights are served faster, which is a + // reason to prefer a variant that neither its size nor its precision + // conveys. + // + // This is the SAME tag-against-vocabulary match servingFeatureRank ranks + // on, over the same host preference list, so what a client shows as a speed + // advantage cannot disagree with what selection actually rewarded. A tag + // outside that vocabulary is not a serving feature and is not reported + // here: the entry's own tag list already carries it. + Features []string `json:"features,omitempty"` +} + +// EntryVariants is the variant surface of a single gallery entry. +type EntryVariants struct { + Variants []VariantView `json:"variants"` + // AutoSelected is the variant auto-selection would install on this host + // right now. Clients show it as the default choice. + AutoSelected string `json:"auto_selected"` +} + +// DescribeVariants reports an entry's selectable builds and which one +// auto-selection would currently pick. +// +// It runs the SAME variantOptions + SelectVariant pass the installer runs, so +// the reported auto-selection cannot drift from what installing would actually +// do. The env carries the same probe seam too, so the size shown here and the +// size the installer compares against come from one cache and one round trip. +// +// An entry that declares no variants returns nil, nil WITHOUT touching the +// probe. That is load-bearing: the gallery listing walks entries by the +// thousand and the overwhelming majority declare nothing, so the no-variant +// case must cost nothing at all. +func DescribeVariants(models []*GalleryModel, entry *GalleryModel, env ResolveEnv) (*EntryVariants, error) { + if entry == nil || !entry.HasVariants() { + return nil, nil + } + + options, err := variantOptions(models, entry, env) + if err != nil { + return nil, err + } + + selection, err := SelectVariant(options, env, "") + if err != nil { + return nil, err + } + + views := make([]VariantView, 0, len(options)) + for _, o := range options { + memory, known := o.EffectiveMemory() + view := VariantView{ + Model: o.Variant.Model, + Backend: o.Backend, + // The base is exempt from every filter and always installs, so + // reporting it as anything but fitting would misdescribe it. + Fits: o.IsBase || (env.backendRuns(o.Backend) && (!known || memory <= env.AvailableMemory)), + IsBase: o.IsBase, + // o.Tags is already the REFERENCED entry's tag list, which is the + // only correct source: the feature belongs to the build that would + // be installed, not to the family the parent describes. + Features: env.servingFeaturesOf(o), + } + if known { + view.MemoryBytes = memory + } + // The base option's payload is the entry itself; every other option + // names a gallery entry variantOptions has already proved resolvable, + // so this second lookup is an in-memory map hit and cannot fail. + source := entry + if !o.IsBase { + source = FindGalleryElement(models, o.Variant.Model) + } + view.Quantization = quantizationOfEntry(source) + views = append(views, view) + } + + return &EntryVariants{Variants: views, AutoSelected: selection.Option.Variant.Model}, nil +} diff --git a/core/gallery/describe_variants_test.go b/core/gallery/describe_variants_test.go new file mode 100644 index 000000000..fb6ead9dc --- /dev/null +++ b/core/gallery/describe_variants_test.go @@ -0,0 +1,369 @@ +package gallery_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/gallery" +) + +var _ = Describe("DescribeVariants", func() { + gib := func(n uint64) uint64 { return n * 1024 * 1024 * 1024 } + + newModel := func(name, backend string) *gallery.GalleryModel { + m := &gallery.GalleryModel{} + m.Name = name + m.Backend = backend + return m + } + + var models []*gallery.GalleryModel + var base *gallery.GalleryModel + // probed records every entry the probe was asked about, so a spec can + // assert on the ABSENCE of a round trip and not merely on the result. + var probed []string + + // probing builds an env whose sizes are injected rather than measured, so + // these specs pin exact footprints without reaching the network. + probing := func(available uint64, sizes map[string]uint64) gallery.ResolveEnv { + return gallery.ResolveEnv{ + AvailableMemory: available, + BackendCompatible: func(string) bool { return true }, + ProbeMemory: func(target *gallery.GalleryModel) uint64 { + probed = append(probed, target.Name) + return sizes[target.Name] + }, + } + } + + byName := func(view *gallery.EntryVariants, name string) gallery.VariantView { + GinkgoHelper() + for _, v := range view.Variants { + if v.Model == name { + return v + } + } + Fail("no variant named " + name + " in the reported view") + return gallery.VariantView{} + } + + BeforeEach(func() { + probed = nil + big := newModel("qwen3-8b-vllm-awq", "vllm") + small := newModel("qwen3-8b-gguf-q8", "llama-cpp") + base = newModel("qwen3-8b-gguf-q4", "llama-cpp") + base.Variants = []gallery.Variant{ + {Model: "qwen3-8b-vllm-awq"}, + {Model: "qwen3-8b-gguf-q8"}, + } + models = []*gallery.GalleryModel{big, small, base} + }) + + Describe("an entry that declares no variants", func() { + It("reports nothing and issues no probe at all", func() { + // This is the performance contract of the gallery listing: the + // listing walks over a thousand entries and virtually none of them + // declare variants, so the ordinary entry must cost zero round + // trips. Asserting on `probed` rather than on timing makes that + // contract testable. + plain := newModel("plain-entry", "llama-cpp") + models = append(models, plain) + + view, err := gallery.DescribeVariants(models, plain, probing(gib(24), map[string]uint64{ + "plain-entry": gib(4), + })) + + Expect(err).ToNot(HaveOccurred()) + Expect(view).To(BeNil()) + Expect(probed).To(BeEmpty()) + }) + + It("reports nothing for a nil entry rather than panicking", func() { + view, err := gallery.DescribeVariants(models, nil, probing(gib(24), nil)) + Expect(err).ToNot(HaveOccurred()) + Expect(view).To(BeNil()) + Expect(probed).To(BeEmpty()) + }) + }) + + Describe("an entry that declares variants", func() { + It("reports every declared variant plus the entry's own build", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(24), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + + names := []string{} + for _, v := range view.Variants { + names = append(names, v.Model) + } + // The base is listed so a picker can offer "decline the upgrade", + // which is a real choice an operator makes. + Expect(names).To(ConsistOf("qwen3-8b-vllm-awq", "qwen3-8b-gguf-q8", "qwen3-8b-gguf-q4")) + Expect(byName(view, "qwen3-8b-gguf-q4").IsBase).To(BeTrue()) + Expect(byName(view, "qwen3-8b-vllm-awq").IsBase).To(BeFalse()) + }) + + It("reports the backend of the referenced entry, not of the declaring one", func() { + // A picker renders this, and it is also the reason a variant can be + // unavailable on a host with ample memory. + view, err := gallery.DescribeVariants(models, base, probing(gib(24), nil)) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Backend).To(Equal("vllm")) + Expect(byName(view, "qwen3-8b-gguf-q8").Backend).To(Equal("llama-cpp")) + }) + + It("reports the probed size of each variant", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(24), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").MemoryBytes).To(Equal(gib(20))) + Expect(byName(view, "qwen3-8b-gguf-q8").MemoryBytes).To(Equal(gib(9))) + }) + + It("reports a size it could not determine as unknown rather than as free", func() { + // Zero on the wire means unknown. Rendering it as a zero-byte model + // would advertise the largest download on offer as costless. + view, err := gallery.DescribeVariants(models, base, probing(gib(24), map[string]uint64{})) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").MemoryBytes).To(Equal(uint64(0))) + Expect(byName(view, "qwen3-8b-vllm-awq").Fits).To(BeTrue()) + }) + + It("marks a variant too large for this host as not fitting", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(10), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Fits).To(BeFalse()) + Expect(byName(view, "qwen3-8b-gguf-q8").Fits).To(BeTrue()) + }) + + It("marks a variant whose backend cannot run here as not fitting, however much memory there is", func() { + env := probing(gib(1024), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + }) + env.BackendCompatible = func(backend string) bool { return backend != "vllm" } + + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Fits).To(BeFalse()) + Expect(byName(view, "qwen3-8b-gguf-q8").Fits).To(BeTrue()) + }) + + It("always reports the entry's own build as fitting", func() { + // The base is exempt from every filter and always installs, so + // showing it as unavailable would misdescribe it. Neither a hostile + // backend gate nor a zero memory budget may change that. + env := probing(0, map[string]uint64{}) + env.BackendCompatible = func(string) bool { return false } + + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-gguf-q4").Fits).To(BeTrue()) + }) + + It("surfaces a variant that references an entry no gallery declares", func() { + base.Variants = []gallery.Variant{{Model: "does-not-exist"}} + _, err := gallery.DescribeVariants(models, base, probing(gib(24), nil)) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("does-not-exist")) + }) + }) + + Describe("the reported auto-selection", func() { + // These are the specs that keep a picker honest: what the listing shows + // as the default must be what installing with no variant actually does. + // They assert against ResolveVariant rather than against a hardcoded + // name, so a change to the selection rules cannot make the two drift + // without failing here. + agreesWithInstall := func(env gallery.ResolveEnv) { + GinkgoHelper() + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + + _, chosen, err := gallery.ResolveVariant(models, base, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal(chosen.Model)) + } + + It("matches what installing would pick when everything fits", func() { + agreesWithInstall(probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + }) + + It("matches what installing would pick when only the smaller variant fits", func() { + agreesWithInstall(probing(gib(12), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + }) + + It("matches what installing would pick when nothing fits and the base wins", func() { + agreesWithInstall(probing(gib(2), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + }) + + It("names the largest fitting variant, not the first authored", func() { + // Pinned separately from agreesWithInstall so that a mutation making + // BOTH functions pick the wrong variant still fails a spec. + view, err := gallery.DescribeVariants(models, base, probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal("qwen3-8b-vllm-awq")) + }) + + It("matches what installing would pick when the host prefers an engine", func() { + // The picker and the installer both have to apply engine + // preference, or a Mac would be shown the GGUF build and handed the + // MLX one. Asserting the agreement AND the name pins both halves. + mlx := newModel("qwen3-8b-mlx-4bit", "mlx") + models = append(models, mlx) + base.Variants = append(base.Variants, gallery.Variant{Model: "qwen3-8b-mlx-4bit"}) + + env := probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + "qwen3-8b-mlx-4bit": gib(5), + }) + // Engine names as SystemState.EnginePreferenceTokens reports them for + // metal. Build tags would match no gallery `backend:` value here. + env.EnginePreference = []string{"mlx", "llama-cpp"} + + agreesWithInstall(env) + + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal("qwen3-8b-mlx-4bit")) + }) + + It("matches what installing would pick when the host prefers vLLM", func() { + // The NVIDIA rule the user asked for, checked through the listing so + // the picker and the installer cannot drift on it. The GGUF build is + // deliberately the LARGER one, so only preference can produce this + // answer: size alone would name the llama.cpp build. + env := probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(9), + "qwen3-8b-gguf-q8": gib(20), + }) + env.EnginePreference = []string{"vllm", "sglang", "llama-cpp"} + + agreesWithInstall(env) + + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal("qwen3-8b-vllm-awq")) + }) + + It("names the entry itself when no variant fits", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(2), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal("qwen3-8b-gguf-q4")) + }) + }) + + Describe("the facts that tell two builds of one model apart", func() { + // servedAs points an entry at a weight file the way the gallery does, + // so the reported quantization comes from the same field the installer + // hands the backend. + servedAs := func(m *gallery.GalleryModel, filename string) { + m.Overrides = map[string]any{ + "parameters": map[string]any{"model": filename}, + } + } + + It("reports each variant's quantization from the entry it references", func() { + // The whole point of the field: these three rows share a backend + // and sit within a gigabyte of each other, so quantization is the + // only thing a user can actually choose on. + servedAs(models[0], "models/Qwen3-8B-AWQ-4bit.safetensors") + models[0].Backend = "llama-cpp" + servedAs(models[1], "models/Qwen3-8B-Q8_0.gguf") + servedAs(base, "models/Qwen3-8B-Q4_K_M.gguf") + + view, err := gallery.DescribeVariants(models, base, probing(gib(64), map[string]uint64{})) + + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Quantization).To(Equal("4BIT")) + Expect(byName(view, "qwen3-8b-gguf-q8").Quantization).To(Equal("Q8_0")) + // The base is described from the entry's own payload, not skipped: + // it is a selectable build like any other. + Expect(byName(view, "qwen3-8b-gguf-q4").Quantization).To(Equal("Q4_K_M")) + }) + + It("leaves the quantization unset when the referenced entry names none", func() { + // A vLLM build served from a directory of safetensors declares no + // format. The field must be absent rather than empty-but-present, + // so a client renders "unknown" instead of a blank cell. + servedAs(models[0], "models/Qwen3-8B/") + servedAs(base, "models/Qwen3-8B-Q4_K_M.gguf") + + view, err := gallery.DescribeVariants(models, base, probing(gib(64), map[string]uint64{})) + + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Quantization).To(BeEmpty()) + Expect(byName(view, "qwen3-8b-gguf-q4").Quantization).To(Equal("Q4_K_M")) + }) + + It("names the serving features a build declares, best first", func() { + models[0].Tags = []string{"llm", "MTP", "gguf", "dflash"} + + env := probing(gib(64), map[string]uint64{}) + env.ServingFeaturePreference = []string{"dflash", "mtp"} + + view, err := gallery.DescribeVariants(models, base, env) + + Expect(err).ToNot(HaveOccurred()) + // Preference order, not the author's order, and case-folded the way + // the ranker folds it: "MTP" and "mtp" declare one feature. + Expect(byName(view, "qwen3-8b-vllm-awq").Features).To(Equal([]string{"dflash", "mtp"})) + // A tag outside the vocabulary is not a serving feature. + Expect(byName(view, "qwen3-8b-gguf-q8").Features).To(BeEmpty()) + }) + + It("agrees with the ranking: the build shown as faster is the one selection rewards", func() { + // The contract that keeps the badge honest. Both builds fit and are + // runnable, so only the serving feature can decide, and the variant + // carrying the reported feature must be the one auto-selected. + models[1].Tags = []string{"dflash"} + + env := probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(9), + "qwen3-8b-gguf-q8": gib(9), + }) + env.ServingFeaturePreference = []string{"dflash", "mtp"} + + view, err := gallery.DescribeVariants(models, base, env) + + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-gguf-q8").Features).To(Equal([]string{"dflash"})) + Expect(view.AutoSelected).To(Equal("qwen3-8b-gguf-q8")) + }) + + It("reports no features on a host with no serving feature vocabulary", func() { + // An env with no preference list ranks every build equally on this + // axis, so claiming a speed advantage would describe an advantage + // nothing acted on. + models[0].Tags = []string{"dflash"} + + view, err := gallery.DescribeVariants(models, base, probing(gib(64), map[string]uint64{})) + + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Features).To(BeEmpty()) + }) + }) +}) diff --git a/core/gallery/empty_base_install_test.go b/core/gallery/empty_base_install_test.go new file mode 100644 index 000000000..6b104e233 --- /dev/null +++ b/core/gallery/empty_base_install_test.go @@ -0,0 +1,243 @@ +package gallery_test + +import ( + "context" + "fmt" + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/gallery" + "github.com/mudler/LocalAI/pkg/system" +) + +// An entry declaring neither url: nor config_file: is installed on an empty +// base config, with overrides: and files: supplying everything. These specs +// cover that path, the payload rule that still rejects an entry with nothing to +// install, and the two older paths, which must be untouched. +// +// Nothing here reaches the network. The whole point of the empty-base path is +// that it fetches nothing, and a spec that quietly went to GitHub would be +// asserting the opposite of the change. +var _ = Describe("InstallModelFromGallery with an empty base config", func() { + var tempdir string + var galleries []config.Gallery + var systemState *system.SystemState + // The gallery listing is cached on the name and URL pair, so every spec + // needs a gallery of its own or it reads the previous spec's catalog. + galleryRevision := 0 + + newGallery := func(entries ...gallery.GalleryModel) { + out, err := yaml.Marshal(entries) + Expect(err).ToNot(HaveOccurred()) + name := fmt.Sprintf("empty-base-%d", galleryRevision) + galleryRevision++ + galleryPath := filepath.Join(tempdir, name+".yaml") + Expect(os.WriteFile(galleryPath, out, 0600)).To(Succeed()) + galleries = []config.Gallery{{Name: name, URL: "file://" + galleryPath}} + } + + install := func(name string, req gallery.GalleryModel, options ...gallery.InstallOption) error { + return gallery.InstallModelFromGallery( + context.TODO(), galleries, []config.Gallery{}, systemState, nil, + name, req, func(string, string, string, float64) {}, false, false, false, options...) + } + + installedConfig := func(name string) map[string]any { + dat, err := os.ReadFile(filepath.Join(tempdir, name+".yaml")) + Expect(err).ToNot(HaveOccurred()) + content := map[string]any{} + Expect(yaml.Unmarshal(dat, &content)).To(Succeed()) + return content + } + + // localWeights lets a spec carry a files: list without leaving the + // filesystem. The downloader treats an already-present destination with no + // declared sha256 as fetched and skips it, so seeding the file is what keeps + // these specs off the network. The URI is still authored, because the + // installer walks the list either way and a missing one would not exercise + // the same code. + localWeights := func(name string) gallery.File { + Expect(os.WriteFile(filepath.Join(tempdir, name), []byte("weights for "+name), 0600)).To(Succeed()) + return gallery.File{Filename: name, URI: "https://example.com/" + name} + } + + BeforeEach(func() { + var err error + tempdir, err = os.MkdirTemp("", "empty-base-install") + Expect(err).ToNot(HaveOccurred()) + DeferCleanup(func() { Expect(os.RemoveAll(tempdir)).To(Succeed()) }) + + systemState, err = system.GetSystemState(system.WithModelPath(tempdir)) + Expect(err).ToNot(HaveOccurred()) + }) + + It("installs an entry that declares only overrides and files", func() { + e := gallery.GalleryModel{Overrides: map[string]any{ + "backend": "ds4", + "parameters": map[string]any{"model": "weights.gguf"}, + }} + e.Name = "overrides-only" + e.Description = "an entry with no base config" + e.AdditionalFiles = []gallery.File{localWeights("weights.gguf")} + newGallery(e) + + Expect(install("overrides-only", gallery.GalleryModel{})).To(Succeed()) + + cfg := installedConfig("overrides-only") + Expect(cfg["backend"]).To(Equal("ds4")) + Expect(cfg["parameters"]).To(HaveKeyWithValue("model", "weights.gguf")) + // The name comes from the install, never from a base config, which is + // what the several hundred virtual.yaml entries were getting wrong: they + // inherited the stub's name. + Expect(cfg["name"]).To(Equal("overrides-only")) + Expect(filepath.Join(tempdir, "weights.gguf")).To(BeAnExistingFile()) + }) + + It("installs an entry that declares only files", func() { + e := gallery.GalleryModel{} + e.Name = "files-only" + e.AdditionalFiles = []gallery.File{localWeights("plain.gguf")} + newGallery(e) + + Expect(install("files-only", gallery.GalleryModel{})).To(Succeed()) + Expect(filepath.Join(tempdir, "plain.gguf")).To(BeAnExistingFile()) + }) + + // Half the point of the change is that the empty-base path costs no fetch, + // and an assertion that nothing was fetched is worthless unless something + // could have been. So the control runs the same install with a url: pointing + // at a base config that is not there: it must fail, proving a url IS read on + // this path, and only then does the same fixture without the url passing + // mean the read was skipped rather than silently succeeding. + Describe("the base config fetch", func() { + var missing string + + BeforeEach(func() { + missing = "file://" + filepath.Join(tempdir, "no-such-base.yaml") + Expect(filepath.Join(tempdir, "no-such-base.yaml")).ToNot(BeAnExistingFile()) + }) + + It("is attempted when the entry declares a url", func() { + e := gallery.GalleryModel{Overrides: map[string]any{"backend": "ds4"}} + e.Name = "with-url" + e.URL = missing + newGallery(e) + + // If this ever starts passing, the control has stopped controlling + // and the spec below proves nothing. + Expect(install("with-url", gallery.GalleryModel{})).To(HaveOccurred()) + }) + + It("is skipped entirely when the entry declares none", func() { + e := gallery.GalleryModel{Overrides: map[string]any{"backend": "ds4"}} + e.Name = "without-url" + newGallery(e) + + Expect(install("without-url", gallery.GalleryModel{})).To(Succeed()) + Expect(installedConfig("without-url")["backend"]).To(Equal("ds4")) + }) + }) + + Describe("an entry with nothing to install", func() { + // No url, no config_file, no overrides and no files. Accepting this + // would write an empty model directory and report success, which is the + // authoring mistake the relaxation would otherwise hide. + emptyEntry := func() gallery.GalleryModel { + e := gallery.GalleryModel{} + e.Name = "hollow" + // urls: is the informational link list. It reads like a payload and + // is not one. + e.URLs = []string{"https://huggingface.co/example/hollow"} + return e + } + + It("is refused rather than installed empty", func() { + newGallery(emptyEntry()) + + err := install("hollow", gallery.GalleryModel{}) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("installs nothing")) + Expect(err.Error()).To(ContainSubstring("hollow")) + Expect(filepath.Join(tempdir, "hollow.yaml")).ToNot(BeAnExistingFile()) + }) + + It("is accepted when the caller's request supplies the payload", func() { + // The request's overrides are merged into the install exactly as the + // entry's own are, so a caller who brings them really has asked for + // something installable. + newGallery(emptyEntry()) + + req := gallery.GalleryModel{Overrides: map[string]any{"backend": "llama-cpp"}} + Expect(install("hollow", req)).To(Succeed()) + Expect(installedConfig("hollow")["backend"]).To(Equal("llama-cpp")) + }) + }) + + // The two older paths are meant to be untouched by the relaxation. + Describe("the pre-existing paths", func() { + It("still installs an entry described by an inline config_file", func() { + e := gallery.GalleryModel{ConfigFile: map[string]any{"backend": "llama-cpp"}} + e.Name = "inline" + newGallery(e) + + Expect(install("inline", gallery.GalleryModel{})).To(Succeed()) + Expect(installedConfig("inline")["backend"]).To(Equal("llama-cpp")) + }) + + It("still installs an entry described by a url", func() { + payload, err := yaml.Marshal(gallery.ModelConfig{ + Name: "fetched", + ConfigFile: "backend: vllm\n", + }) + Expect(err).ToNot(HaveOccurred()) + payloadPath := filepath.Join(tempdir, "payload.yaml") + Expect(os.WriteFile(payloadPath, payload, 0600)).To(Succeed()) + + e := gallery.GalleryModel{} + e.Name = "from-url" + e.URL = "file://" + payloadPath + newGallery(e) + + Expect(install("from-url", gallery.GalleryModel{})).To(Succeed()) + Expect(installedConfig("from-url")["backend"]).To(Equal("vllm")) + }) + }) + + // One of the entries that shipped uninstallable, driven through the real + // install path rather than only checked as text. Its files: are swapped for + // a local one because the real ones are gigabytes on HuggingFace; its + // overrides: are the catalog's own, so this proves the authored payload + // lands. + It("installs a previously broken index entry once the empty base is allowed", func() { + entries, err := loadGalleryIndex() + Expect(err).ToNot(HaveOccurred()) + + var e gallery.GalleryModel + for _, candidate := range entries { + if candidate.Name == "liquidai_lfm2-1.2b-rag" { + e = candidate + break + } + } + Expect(e.Name).To(Equal("liquidai_lfm2-1.2b-rag")) + // The defect: no base config of any kind, which used to be fatal. + Expect(e.URL).To(BeEmpty()) + Expect(e.ConfigFile).To(BeEmpty()) + Expect(e.Overrides).ToNot(BeEmpty()) + + e.AdditionalFiles = []gallery.File{localWeights("LiquidAI_LFM2-1.2B-RAG-Q4_K_M.gguf")} + newGallery(e) + + Expect(install(e.Name, gallery.GalleryModel{})).To(Succeed()) + cfg := installedConfig(e.Name) + Expect(cfg["name"]).To(Equal(e.Name)) + // The catalog's own overrides, verbatim, laid over the empty base. + Expect(cfg["parameters"]).To(Equal(e.Overrides["parameters"])) + Expect(cfg["known_usecases"]).To(Equal(e.Overrides["known_usecases"])) + }) +}) diff --git a/core/gallery/gallery.go b/core/gallery/gallery.go index d3dbc6d2f..12a038577 100644 --- a/core/gallery/gallery.go +++ b/core/gallery/gallery.go @@ -334,6 +334,27 @@ var ( // invalidate entries when the gallery data changes. func GalleryGeneration() uint64 { return galleryGeneration.Load() } +// ResetGalleryModelCache drops the cached model list, once any background +// refresh already in flight has finished writing to it. +// +// It exists for tests. The cache is a package global keyed by nothing, which is +// right for a process serving one gallery configuration and wrong for a suite +// where each spec stands up its own: a refresh one spec triggered can land in +// the middle of the next and answer it with the previous spec's entries, so +// whichever assertion happens to straddle it fails at random. +// +// Waiting for the in-flight refresh rather than only clearing is the point. The +// refresh publishes its result after this call would otherwise have returned, +// so clearing without waiting just narrows the window. +func ResetGalleryModelCache() { + for refreshing.Load() { + time.Sleep(time.Millisecond) + } + availableModelsMu.Lock() + availableModelsCache = nil + availableModelsMu.Unlock() +} + // AvailableGalleryModelsCached returns gallery models from an in-memory cache. // Local-only fields (installed status) are refreshed on every call. A background // goroutine is triggered to re-fetch the full model list (including network diff --git a/core/gallery/model_artifacts.go b/core/gallery/model_artifacts.go index 78bd88905..fb302157c 100644 --- a/core/gallery/model_artifacts.go +++ b/core/gallery/model_artifacts.go @@ -19,6 +19,7 @@ type ArtifactMaterializer interface { type installOptions struct { materializer ArtifactMaterializer + variant string } type InstallOption func(*installOptions) @@ -31,6 +32,16 @@ func WithArtifactMaterializer(materializer ArtifactMaterializer) InstallOption { } } +// WithVariant pins a gallery entry to a specific variant by name, bypassing +// hardware-based selection. The entry's own name is a valid pin, since the +// entry is itself the last resort. Ignored for entries that declare no +// variants. +func WithVariant(variant string) InstallOption { + return func(options *installOptions) { + options.variant = variant + } +} + func applyInstallOptions(options ...InstallOption) installOptions { result := installOptions{materializer: modelartifacts.NewDefaultManager()} for _, option := range options { diff --git a/core/gallery/models.go b/core/gallery/models.go index 4d39cc0cc..f3184648b 100644 --- a/core/gallery/models.go +++ b/core/gallery/models.go @@ -9,6 +9,7 @@ import ( "path/filepath" "slices" "strings" + "time" "dario.cat/mergo" lconfig "github.com/mudler/LocalAI/core/config" @@ -17,6 +18,8 @@ import ( "github.com/mudler/LocalAI/pkg/modelartifacts" "github.com/mudler/LocalAI/pkg/system" "github.com/mudler/LocalAI/pkg/utils" + "github.com/mudler/LocalAI/pkg/vram" + "github.com/mudler/LocalAI/pkg/xsysinfo" "github.com/mudler/xlog" "gopkg.in/yaml.v3" @@ -60,6 +63,13 @@ type ModelConfig struct { ConfigFile string `yaml:"config_file"` Files []File `yaml:"files"` PromptTemplates []PromptTemplate `yaml:"prompt_templates"` + + // The fields below record how an entry carrying variants was resolved, + // so a reinstall or upgrade can honor the same pin and so operators can + // see which variant a stable model name is actually backed by. + EntryName string `yaml:"entry_name,omitempty"` + ResolvedVariant string `yaml:"resolved_variant,omitempty"` + PinnedVariant string `yaml:"pinned_variant,omitempty"` } type File struct { @@ -73,6 +83,292 @@ type PromptTemplate struct { Content string `yaml:"content"` } +// variantOptions turns an entry's declared variants into selectable options, +// resolving each referenced gallery entry so the selector gets the backend it +// filters on, and appends the entry itself as the base option. +// +// The base is appended rather than special-cased downstream so there is exactly +// one selection path, and so an operator can pin the entry's own name to +// decline an upgrade their hardware would otherwise take. +func variantOptions(models []*GalleryModel, entry *GalleryModel, env ResolveEnv) ([]VariantOption, error) { + options := make([]VariantOption, 0, len(entry.Variants)+1) + for _, v := range entry.Variants { + // Every referenced entry is looked up here rather than lazily after + // selection, because the backend that decides whether a variant is even + // eligible lives on that entry, not on the variant. + target := FindGalleryElement(models, v.Model) + if target == nil { + return nil, fmt.Errorf("model %q references variant %q which does not exist in any configured gallery", entry.Name, v.Model) + } + if target.HasVariants() { + return nil, fmt.Errorf("model %q references variant %q which declares variants of its own; resolution is a single pass, so those would be silently ignored", entry.Name, v.Model) + } + + // Tags come from the REFERENCED entry, not from the declaring one: the + // serving feature being ranked is a property of the build that would be + // installed, and a parent's tags describe the model family. + option := VariantOption{Variant: v, Backend: target.Backend, Tags: target.Tags} + if env.ProbeMemory != nil { + option.ProbedMemory = env.ProbeMemory(target) + } + options = append(options, option) + } + + // The base is probed like any other candidate. It is never filtered out, but + // it IS ranked against the variants, and an unsized base would lose every + // contest to a variant whose size nothing could measure. + base := VariantOption{ + Variant: Variant{Model: entry.Name}, + Backend: entry.Backend, + IsBase: true, + Tags: entry.Tags, + } + if env.ProbeMemory != nil { + base.ProbedMemory = env.ProbeMemory(entry) + } + return append(options, base), nil +} + +// probeContextLength is the context size the footprint estimate is taken at. +// Selection compares whole models against whole hosts, so this only has to be a +// consistent, realistic default rather than the context a user will eventually +// configure. +const probeContextLength = 8192 + +// probeTimeout bounds a single entry's probe. An entry with several variants +// probes each of them, so an unreachable host has to give up quickly: an +// unknown size only costs a variant its ranking, whereas a stalled probe costs +// the user the whole install. +const probeTimeout = 5 * time.Second + +// probeEntryMemory measures what a gallery entry will occupy, without +// downloading it. +// +// pkg/vram does the work and caches its results across calls: it range-fetches +// the GGUF header for a real estimate, falls back to an HTTP HEAD for the +// content length, and finally to any declared size:. A HuggingFace repo listing +// is deliberately NOT used as a further fallback, unlike the gallery UI's +// estimator: a quantization entry's urls routinely point at the base model's +// repo, and summing every weight file there would overstate one variant badly +// enough to wrongly filter it out. +// +// Zero means the size could not be determined. Callers must read that as +// unknown rather than as zero, so an unreachable network downgrades a variant's +// ranking instead of failing the install. +func probeEntryMemory(ctx context.Context, entry *GalleryModel) uint64 { + input := vram.ModelEstimateInput{Size: entry.Size} + for _, f := range entry.AdditionalFiles { + if vram.IsWeightFile(f.URI) { + input.Files = append(input.Files, vram.FileInput{URI: f.URI}) + } + } + if len(input.Files) == 0 && input.Size == "" { + return 0 + } + + ctx, cancel := context.WithTimeout(ctx, probeTimeout) + defer cancel() + + estimate, err := vram.EstimateModelMultiContext(ctx, input, []uint32{probeContextLength}) + if err != nil { + xlog.Debug("Could not probe a model variant's size; treating it as unknown", "model", entry.Name, "error", err) + return 0 + } + return estimate.VRAMForContext(probeContextLength) +} + +// ResolveVariant picks the gallery entry to install for a host, from the +// variants an entry declares plus the entry itself, and returns it renamed to +// the entry's name and carrying the entry's presentation metadata. +// +// Why the metadata split: the payload (url, config_file, files, overrides) +// must come from the chosen variant because that is what actually gets +// downloaded, while the presentation (name, description, icon, tags) must come +// from the entry the user asked for, so the installed model presents as that +// model rather than as one of its variants. +// +// The base entry always resolves, whatever the host has. It is a complete entry +// that every older LocalAI release installs unconditionally, so refusing it here +// would make the gallery behave worse the newer the client is. Being exempt from +// the filters does not make it exempt from ranking: it is measured and compared +// like every declared variant, and wins by default only once they have all been +// ruled out. +func ResolveVariant(models []*GalleryModel, entry *GalleryModel, env ResolveEnv, pin string) (*GalleryModel, Variant, error) { + options, err := variantOptions(models, entry, env) + if err != nil { + return nil, Variant{}, err + } + + selection, err := SelectVariant(options, env, pin) + if err != nil { + return nil, Variant{}, fmt.Errorf("resolving variant for model %q: %w", entry.Name, err) + } + selected := selection.Option + + if selection.FellBackToBase && len(entry.Variants) > 0 { + xlog.Warn("No declared variant of this model fits this system; installing the entry's own build", + "model", entry.Name, "available_memory", env.AvailableMemory, "reasons", strings.Join(selection.Reasons, "; ")) + } + + // A pin is an operator override and deliberately bypasses the hardware + // checks, but a silent bypass makes a later out-of-memory failure + // impossible to trace back to the pin, so it is recorded loudly here. + if pin != "" { + if need, known := selected.EffectiveMemory(); known && env.AvailableMemory < need { + xlog.Warn("Pinned model variant declares more memory than this system reports; installing anyway because the pin overrides hardware resolution", + "model", entry.Name, "variant", selected.Variant.Model, "required_memory", need, "available_memory", env.AvailableMemory) + } + } + + // Selecting the base means installing the entry's own payload, which is the + // entry itself; there is no second entry to look up. + source := entry + if !selected.IsBase { + // variantOptions already proved this lookup succeeds. + source = FindGalleryElement(models, selected.Variant.Model) + } + + resolved := *source + resolved.Name = entry.Name + resolved.Description = entry.Description + resolved.Icon = entry.Icon + resolved.License = entry.License + // The resolved entry is a concrete install target, so it must not carry the + // variant list any more; leaving it would let a second pass resolve the + // already-resolved entry all over again. + resolved.Variants = nil + + // The struct copy above is shallow, so every reference-typed field still + // aliases the gallery's own entries. The install path mutates Overrides in + // place (mergo merges the caller's request overrides into it) and appends to + // the URL and tag slices, which would write the caller's request into the + // gallery catalog itself and leak between installs the moment this path + // reads from a cached, long-lived gallery listing. Detach them here. + // + // Overrides and ConfigFile are copied all the way down rather than cloned at + // the top level only: gallery overrides are nested in practice (a + // parameters.model map is near-universal), and mergo recurses into nested + // maps and overwrites them in place, so a top-level clone would still hand + // the caller the gallery's own inner maps. The slices below hold value types, + // so cloning them once fully detaches them. + resolved.Overrides = deepCopyStringMap(source.Overrides) + resolved.ConfigFile = deepCopyStringMap(source.ConfigFile) + resolved.AdditionalFiles = slices.Clone(source.AdditionalFiles) + resolved.URLs = slices.Clone(entry.URLs) + resolved.Tags = slices.Clone(entry.Tags) + + return &resolved, selected.Variant, nil +} + +// HostResolveEnv describes this machine to variant selection. +// +// It exists so the install path and every read-only surface that reports what +// selection WOULD choose derive the host from one place. Two copies of this +// wiring would eventually disagree, and a picker that shows a different answer +// than the installer produces is worse than no picker at all. +func HostResolveEnv(ctx context.Context, systemState *system.SystemState) ResolveEnv { + return ResolveEnv{ + AvailableMemory: availableModelMemory(systemState), + // The whole hardware gate. IsBackendCompatible already derives + // Darwin-only, NVIDIA-only, ROCm-only and SYCL-only from the backend + // name, so a gallery author never has to describe hardware. + // + // The uri argument is deliberately empty: it exists for backend OCI + // images, and passing a model's gallery url here would let an unrelated + // substring in a download link decide hardware compatibility. + BackendCompatible: func(backend string) bool { + return systemState.IsBackendCompatible(backend, "") + }, + // The ranking half of the hardware story. IsBackendCompatible only rules + // out what cannot run; on a Mac both an MLX and a llama.cpp build can, + // and this is what makes the native runtime win. + // + // EnginePreferenceTokens, NOT BackendPreferenceTokens: a variant is + // matched on its gallery `backend:` engine name, and the latter reports + // build tags that no engine name contains. + EnginePreference: systemState.EnginePreferenceTokens(), + // The third vocabulary, and the only one that is not host derived: no + // hardware prefers a plain build over an equivalent faster one, so this + // comes from a package function rather than from systemState. + ServingFeaturePreference: system.ServingFeaturePreferenceTokens(), + ProbeMemory: func(target *GalleryModel) uint64 { + return probeEntryMemory(ctx, target) + }, + } +} + +// noGPUDetected is what SystemState.DetectedCapability() reports when no +// usable GPU was found. The constant is unexported in pkg/system. +const noGPUDetected = "default" + +// availableModelMemory reports how much memory a model may occupy on this host. +// +// With a discrete GPU the model lives in VRAM. Otherwise it lives in system +// RAM, read through xsysinfo because that path is cgroup-aware: under +// Kubernetes the container's limit, not the node's physical RAM, is what the +// model actually gets. +// +// A detected GPU whose VRAM reads as zero falls back to RAM rather than to +// zero. That is the normal shape of a unified-memory host, not an error: +// arm64 macs report the metal capability unconditionally, yet have no discrete +// VRAM pool for TotalAvailableVRAM to find, so the model's real budget is +// system RAM. Returning the zero here would strand every Mac on the base +// build no matter how much memory it has. +// +// An unreadable RAM figure yields 0, which drops every variant with a known +// requirement and installs the base. That is the safe direction: an unknown +// host should not be talked into a larger download. +func availableModelMemory(systemState *system.SystemState) uint64 { + if systemState.DetectedCapability() != noGPUDetected && systemState.VRAM > 0 { + return systemState.VRAM + } + ram, err := xsysinfo.GetSystemRAMInfo() + if err != nil || ram == nil { + xlog.Warn("Could not read system RAM; treating this host as having no memory budget for variant selection", "error", err) + return 0 + } + return ram.Total +} + +// deepCopyStringMap copies a decoded YAML map so no part of the result, at any +// depth, is reachable from the original. +func deepCopyStringMap(m map[string]any) map[string]any { + if m == nil { + return nil + } + out := make(map[string]any, len(m)) + for k, v := range m { + out[k] = deepCopyYAMLValue(v) + } + return out +} + +// deepCopyYAMLValue recurses through the only container shapes a YAML decoder +// produces. Scalars are returned as-is because they cannot be mutated through +// the copy. map[any]any is handled as well because gopkg.in/yaml.v2 decodes +// non-string keys into it, and gallery documents are not guaranteed to have +// passed through the v3 decoder. +func deepCopyYAMLValue(v any) any { + switch t := v.(type) { + case map[string]any: + return deepCopyStringMap(t) + case map[any]any: + out := make(map[any]any, len(t)) + for k, val := range t { + out[k] = deepCopyYAMLValue(val) + } + return out + case []any: + out := make([]any, len(t)) + for i, val := range t { + out[i] = deepCopyYAMLValue(val) + } + return out + default: + return v + } +} + // Installs a model from the gallery func InstallModelFromGallery( ctx context.Context, @@ -81,7 +377,9 @@ func InstallModelFromGallery( modelLoader *model.ModelLoader, name string, req GalleryModel, downloadStatus func(string, string, string, float64), enforceScan, automaticallyInstallBackend, requireBackendIntegrity bool, options ...InstallOption) error { - applyModel := func(model *GalleryModel) error { + installOpts := applyInstallOptions(options...) + + applyModel := func(model *GalleryModel, record *ModelConfig) error { name = strings.ReplaceAll(name, string(os.PathSeparator), "__") var config ModelConfig @@ -104,13 +402,46 @@ func InstallModelFromGallery( ConfigFile: string(reYamlConfig), Description: model.Description, License: model.License, - URLs: model.URLs, Name: model.Name, Files: make([]File, 0), // Real values get added below, must be blank + // URLs are deliberately not seeded here: they are appended once + // below for both branches, and seeding them too would write every + // URL twice into the persisted gallery file. // Prompt Template Skipped for now - I expect in this mode that they will be delivered as files. } } else { - return fmt.Errorf("invalid gallery model %+v", model) + // An entry that names no base config describes itself entirely + // through overrides: and files:, so the base is simply empty. + // + // This is what the several hundred entries pointing at + // gallery/virtual.yaml were already getting. That stub carries only + // a name, a description and a license, all three of which are + // overwritten from the gallery entry a few lines up, and overrides + // reach InstallModel as a separate argument rather than being merged + // into the fetched config. So the fetch bought nothing beyond a + // round trip to GitHub on every install, and an author who omits the + // key is asking for exactly the same thing. + if !model.installsSomething(req) { + return fmt.Errorf("gallery model %q installs nothing: it declares no url, no config_file, no overrides and no files", model.Name) + } + config = ModelConfig{ + Description: model.Description, + License: model.License, + Name: model.Name, + Files: make([]File, 0), // Real values get added below, must be blank + // URLs are appended once below for every branch, as in the + // config_file case above. + } + } + + if record != nil { + config.EntryName = record.EntryName + config.ResolvedVariant = record.ResolvedVariant + config.PinnedVariant = record.PinnedVariant + // The variant's own name would otherwise be persisted here, which + // contradicts the whole point of variants: the model is known by + // the entry's stable name regardless of which variant backs it. + config.Name = record.EntryName } installName := model.Name @@ -157,7 +488,77 @@ func InstallModelFromGallery( return fmt.Errorf("no model found with name %q", name) } - return applyModel(model) + // An entry without variants is installed directly. An entry with them is + // still installable as-is; selection below only decides whether one of its + // declared alternatives suits this host better. + if !model.HasVariants() { + // A caller who named a variant asked for something this entry cannot + // give. Installing the entry anyway would look like success while + // quietly ignoring the choice, so it is refused by name instead. + if installOpts.variant != "" { + return fmt.Errorf("%w: %q was requested but model %q declares no variants", ErrPinNotFound, installOpts.variant, model.Name) + } + return applyModel(model, nil) + } + + pin := installOpts.variant + // A previously recorded pin survives reinstalls and upgrades, so a user + // who deliberately chose a variant is not silently re-resolved onto a + // different one by a hardware or gallery change. + // + // The record is keyed by the name the model was installed under, not by the + // gallery entry name: applyModel writes it to ._gallery_.yaml, + // where installName is req.Name whenever the caller supplied one. Reading it + // back under the entry's own name would miss the record for every custom-named + // install and silently re-resolve a deliberately pinned model onto a + // different variant, possibly swapping its backend. + // + // recalledPin tracks where the pin came from, because the two sources must + // fail differently when the name no longer resolves. See below. + recalledPin := "" + if pin == "" { + installName := model.Name + if req.Name != "" { + installName = req.Name + } + if previous, err := GetLocalModelConfiguration(systemState.Model.ModelsPath, installName); err == nil && previous != nil { + recalledPin = previous.PinnedVariant + pin = recalledPin + } + } + + env := HostResolveEnv(ctx, systemState) + + resolved, variant, err := ResolveVariant(models, model, env, pin) + // A pin the caller supplied on this request must stay fatal: they named + // something this entry cannot give, and installing anything else would + // report success for a request that was not honored. + // + // A pin recalled from disk is different. The gallery can rename or withdraw + // a variant long after it was pinned, and the user is not asking for it + // again on this call. Failing here would turn one gallery edit into a + // permanently unrepairable model whose only remedy is deleting a dotfile + // they have never heard of, so the stale pin is dropped and selection runs + // as if it had never been recorded. + if err != nil && recalledPin != "" && errors.Is(err, ErrPinNotFound) { + xlog.Warn("The recorded variant pin for this model no longer exists in the gallery; re-selecting automatically", + "model", model.Name, "dropped_pin", recalledPin) + pin = "" + resolved, variant, err = ResolveVariant(models, model, env, "") + } + if err != nil { + return err + } + + xlog.Info("Resolved model to variant", + "model", model.Name, "variant", variant.Model, + "available_memory", env.AvailableMemory, "pinned", pin != "") + + return applyModel(resolved, &ModelConfig{ + EntryName: model.Name, + ResolvedVariant: variant.Model, + PinnedVariant: pin, + }) } func InstallModel(ctx context.Context, systemState *system.SystemState, nameOverride string, galleryConfig *ModelConfig, configOverrides map[string]any, downloadStatus func(string, string, string, float64), enforceScan bool, options ...InstallOption) (*lconfig.ModelConfig, error) { diff --git a/core/gallery/models_types.go b/core/gallery/models_types.go index 24e32e03e..a623ff535 100644 --- a/core/gallery/models_types.go +++ b/core/gallery/models_types.go @@ -15,6 +15,37 @@ type GalleryModel struct { ConfigFile map[string]any `json:"config_file,omitempty" yaml:"config_file,omitempty"` // Overrides are used to override the configuration of the model located at URL Overrides map[string]any `json:"overrides,omitempty" yaml:"overrides,omitempty"` + // Variants is an optional, UNORDERED list of alternative builds of the same + // model (other backends such as MLX or vLLM, other quantizations) that the + // installer may pick instead of this entry's own payload. Authoring is + // deliberately dumb: name the models, and the selector works out which one + // this host should get. + // + // The entry itself is always the last resort, so an entry carrying variants + // stays a complete, installable entry and older LocalAI releases, which drop + // this key, install it exactly as before. + Variants []Variant `json:"variants,omitempty" yaml:"variants,omitempty"` +} + +// installsSomething reports whether this entry, combined with the caller's +// request, would put anything on disk. +// +// It exists to keep an authoring mistake loud. Once an entry with no url and no +// config_file is accepted as an empty base, the only thing separating a +// deliberate overrides-only entry from a half-written stanza is whether it +// carries a payload at all. Without this check the stub would install cleanly +// and leave a model directory holding a config naming no weights, which fails +// far from the entry that caused it. +// +// The request is counted because its overrides and files are merged into the +// install exactly as the entry's own are, so a caller supplying them really has +// asked for something installable. +// +// An entry with a url or a config_file never reaches this: it has a base config +// to install, however thin. +func (m *GalleryModel) installsSomething(req GalleryModel) bool { + return len(m.Overrides) > 0 || len(m.AdditionalFiles) > 0 || + len(req.Overrides) > 0 || len(req.AdditionalFiles) > 0 } func (m *GalleryModel) GetInstalled() bool { @@ -45,6 +76,13 @@ func (m GalleryModel) ID() string { return fmt.Sprintf("%s@%s", m.Gallery.Name, m.Name) } +// HasVariants reports whether this entry declares alternative builds of itself. +// It says nothing about the entry being installable: an entry with variants is +// a complete entry that can always be installed as-is. +func (m GalleryModel) HasVariants() bool { + return len(m.Variants) > 0 +} + func (m *GalleryModel) GetTags() []string { return m.Tags } diff --git a/core/gallery/resolve_variant.go b/core/gallery/resolve_variant.go new file mode 100644 index 000000000..e3f1b4699 --- /dev/null +++ b/core/gallery/resolve_variant.go @@ -0,0 +1,435 @@ +package gallery + +import ( + "errors" + "fmt" + "slices" + "sort" + "strings" +) + +var ( + // ErrNoVariantMatch is returned when nothing at all is selectable, which + // only happens for a caller that supplies no base option. + ErrNoVariantMatch = errors.New("no model variant matches this system") + // ErrPinNotFound is returned when a pinned variant is absent from the list. + ErrPinNotFound = errors.New("pinned model variant not found") +) + +// VariantOption is a variant paired with the facts a host is matched against. +// +// The install layer builds these by looking the referenced entries up in the +// live gallery. The selector never touches the catalog, which is what keeps it +// pure and testable across hardware shapes without touching the machine. +type VariantOption struct { + Variant Variant + // Backend is the engine the referenced entry resolves to (e.g. "llama-cpp", + // "mlx", "vllm"). Hardware support is derived from this name alone, which is + // precisely why a gallery author never has to describe hardware. + Backend string + // IsBase marks the declaring entry's own payload. It is exempt from every + // filter, because the entry must stay installable on every host and for + // every client, but it is otherwise an ordinary candidate: it is ranked + // against the declared variants rather than consulted only once they have + // all been rejected. + IsBase bool + // ProbedMemory is the footprint measured live from the referenced entry's + // weights, in bytes. It is the only source of a variant's size. An author + // who needs to correct a bad estimate sets `size:` on the referenced entry, + // which the estimator behind the probe already prefers over its own + // guesswork, so the correction lands everywhere the size is used. + // + // Zero means the probe could not determine a size. That is an unknown, not + // a zero requirement: a probe that cannot reach the network must never be + // able to break an install. + ProbedMemory uint64 + // Tags are the referenced entry's declared tags, and they are the SOLE + // serving feature signal. A tag is something an author wrote down on + // purpose, whereas an entry name only happens to contain a marker, so + // nothing else is consulted. + // + // Empty therefore means "declares no serving feature", and an entry that + // enables one without saying so ranks as a plain build. That is the cost of + // making the signal a declaration, and it is the right cost: an install + // silently downgraded to the plain build is recoverable, whereas a build + // promoted on a marker nobody meant is not visible at all. + // .agents/adding-gallery-models.md states the tagging rule. + Tags []string +} + +// EffectiveMemory returns this option's memory requirement in bytes and whether +// one is known at all. +func (o VariantOption) EffectiveMemory() (uint64, bool) { + if o.ProbedMemory > 0 { + return o.ProbedMemory, true + } + return 0, false +} + +// ResolveEnv describes the host a variant is selected for. +type ResolveEnv struct { + // AvailableMemory is what a model may occupy on this host: VRAM when a + // usable GPU was detected, system RAM otherwise. A model's footprint is + // roughly the same in either, so one number and one comparison cover both. + AvailableMemory uint64 + // BackendCompatible reports whether a backend can run here. The install + // layer wires SystemState.IsBackendCompatible, which already derives + // Darwin-only, NVIDIA-only, ROCm-only and SYCL-only from the backend name. + // + // A nil func treats every backend as runnable, the right default for a + // caller with no view of the hardware. + BackendCompatible func(backend string) bool + // EnginePreference lists the ENGINE NAMES this host prefers, best first, as + // SystemState.EnginePreferenceTokens reports them (e.g. NVIDIA gives + // ["vllm", "sglang", "llama-cpp"], metal gives ["mlx", "llama-cpp"]). A + // token is matched as a substring of a variant's backend name. + // + // The vocabulary is load bearing. VariantOption.Backend is a gallery entry's + // `backend:` value, which is an engine name and never carries a build tag, + // so build tags like "cuda" or "rocm" match NOTHING here. Do not wire + // SystemState.BackendPreferenceTokens into this field: that reports build + // tags for installed-build alias resolution, and the mismatch does not + // error, it scores every candidate equally and silently reduces selection to + // size alone. pkg/system/capabilities.go documents both tables. + // + // BackendCompatible answers "can this run here at all" and is a filter. + // This answers "which of the things that CAN run here should win" and is + // only a ranking. The two are deliberately separate: a Mac can run both an + // MLX build and a llama.cpp build, so nothing is filtered, yet the native + // accelerated runtime should still be installed even when the GGUF build is + // the larger download. + // + // An empty list ranks every backend equally, which reduces selection to + // ordering by size alone. That is the right default for a caller with no + // view of the hardware, and it is what every host looked like before + // preference existed. + EnginePreference []string + // ServingFeaturePreference lists the SERVING FEATURES to prefer, best first, + // as system.ServingFeaturePreferenceTokens reports them (currently + // ["dflash", "mtp"]). A token matches when it equals one of the variant's + // declared TAGS, compared whole and case-insensitively. Nothing else is + // matched: not the entry name, not its backend options. + // + // A third vocabulary, matched against a third thing. It is neither a build + // tag nor an engine name: these name a way of serving the same weights + // faster. pkg/system/capabilities.go documents all three tables together + // and justifies why this one reads a declaration rather than free text. + // + // An empty list ranks every build equally on this axis, which is what every + // host looked like before serving features were ranked at all. + ServingFeaturePreference []string + // ProbeMemory measures how much memory a referenced gallery entry needs, + // without downloading it. A zero result means "could not tell", never + // "needs nothing". + // + // It is a func field rather than a live network handle so specs can pin an + // exact size, or an exact failure, without reaching the internet. A nil func + // leaves every variant unknown, which selection already handles. + // + // SelectVariant never calls this: the install layer resolves every size into + // VariantOption.ProbedMemory first, so the selector stays pure. + ProbeMemory func(entry *GalleryModel) uint64 +} + +func (e ResolveEnv) backendRuns(backend string) bool { + if e.BackendCompatible == nil { + return true + } + return e.BackendCompatible(backend) +} + +// preferenceRank scores a backend against this host's preferred engine order, +// lower being better. +// +// It walks EnginePreference generically and names no engine and no capability: +// everything a new runtime needs is expressed by the table in pkg/system, so +// adding one never reaches this function. +// +// A backend matching no token scores just below the least preferred known one. +// It is never dropped, and a field where nothing matches scores uniformly, so +// an unrecognised backend, an unrecognised capability and an absent preference +// list all degrade to the same predictable place: ordering by size alone. +func (e ResolveEnv) preferenceRank(backend string) int { + name := strings.ToLower(backend) + return preferenceIndex(e.EnginePreference, func(token string) bool { + return strings.Contains(name, token) + }) +} + +// servingFeatureRank scores a variant against the preferred serving feature +// order, lower being better. +// +// It names no feature, exactly as preferenceRank names no engine: the ordered +// list comes from pkg/system, so teaching LocalAI about a new serving feature +// is a one-line edit to that table and never reaches this file. +// +// A declared TAG is the ONLY signal, compared whole and case-insensitively. An +// entry name is deliberately not consulted: a naming convention is not a +// contract, and an author is free to spell a build however they like, so +// reading a marker out of a name infers a capability nobody declared. Tags are +// a LocalAI-level vocabulary the gallery curators control and they mean the +// same thing on every backend. `overrides.options` was considered as the +// signal and rejected for the opposite reason: spec_type:draft-mtp is what +// actually enables the feature, but that spelling is llama.cpp's config +// vocabulary, and a cross-backend ranking decision must not depend on one +// backend's option syntax. Curators check the options at tagging time; the +// ranker reads only the tag. +// +// A build declaring no feature is a plain build and scores just below the +// least preferred known one, which is the whole point: whenever a faster way +// to serve the same weights survived the filters, it outranks the plain build. +func (e ResolveEnv) servingFeatureRank(o VariantOption) int { + tags := lowercased(o.Tags) + return preferenceIndex(e.ServingFeaturePreference, func(token string) bool { + return slices.Contains(tags, token) + }) +} + +// servingFeaturesOf lists the serving features a variant declares, best first. +// +// It is the descriptive half of servingFeatureRank: that function collapses the +// same match into one score for ordering, this one names what matched so a +// picker can tell the user WHY one build is preferred over another. Both read +// the single vocabulary on ResolveEnv, so a feature can never be shown as an +// advantage the ranker did not reward, nor rewarded without being shown. +// +// Preference order rather than the author's tag order, so the list reads best +// thing first. +// +// nil rather than an empty slice for a plain build, so the view field it feeds +// is omitted from the wire entirely. +func (e ResolveEnv) servingFeaturesOf(o VariantOption) []string { + if len(e.ServingFeaturePreference) == 0 || len(o.Tags) == 0 { + return nil + } + tags := lowercased(o.Tags) + var features []string + for _, token := range e.ServingFeaturePreference { + if slices.Contains(tags, strings.ToLower(strings.TrimSpace(token))) { + features = append(features, token) + } + } + return features +} + +// lowercased folds a tag list for comparison, so a gallery author writing +// "MTP" declares the same feature as one writing "mtp". +func lowercased(tags []string) []string { + if len(tags) == 0 { + return nil + } + out := make([]string, 0, len(tags)) + for _, t := range tags { + out = append(out, strings.ToLower(strings.TrimSpace(t))) + } + return out +} + +// preferenceIndex returns the position of the first token `matches` accepts. +// +// A subject matching no token scores just below the least preferred known one. +// It is never dropped, and a list where nothing matches scores uniformly, so an +// unrecognised subject and an absent preference list degrade to the same +// predictable place: this axis stops discriminating and the next key decides. +func preferenceIndex(tokens []string, matches func(token string) bool) int { + if len(tokens) == 0 { + return 0 + } + for i, token := range tokens { + if token != "" && matches(token) { + return i + } + } + return len(tokens) +} + +// VariantSelection is the outcome of a selection pass. +type VariantSelection struct { + Option VariantOption + // FellBackToBase reports that no declared variant survived the filters and + // the entry's own payload was all that remained. Callers log this, because a + // host quietly taking the base when upgrades were on offer is worth being + // able to see. + // + // It is deliberately narrower than "the base was selected": the base also + // wins on merit whenever it outranks every surviving variant, and that is an + // ordinary, uninteresting outcome rather than something to warn about. + FellBackToBase bool + // Reasons explains, one line per rejected variant, why it was dropped. + Reasons []string +} + +// SelectVariant picks the option to install for a host. +// +// The algorithm, in order: +// +// 1. An explicit pin wins outright, fit or no fit. It is a deliberate operator +// override, so refusing it would defeat the point; the caller warns. +// 2. Variants whose backend cannot run here are dropped. This is the whole of +// the hardware gate, derived from the backend name. +// 3. Variants whose known memory requirement exceeds what the host has are +// dropped. A variant with an UNKNOWN requirement survives, because nothing +// proves it does not fit and refusing on a size the probe could not read +// would let a network hiccup silently downgrade what gets installed. +// 4. The base survives both filters unconditionally, and then competes. It is +// a candidate like any other, not a last resort: an entry whose own build is +// the largest thing that fits must win against a smaller variant, and a base +// of known size must win against a variant whose size nothing could measure. +// 5. The survivors are ranked and the best one wins: fit tier first (rankOf +// below), then this host's engine preference, then the serving feature +// preference, then size. Both preferences beat size deliberately, so a Mac +// takes an MLX build over a larger GGUF one, and a host that can hold a +// speculative-decoding build of the same weights takes it over the plain +// one. +// 6. With no survivor at all, which can only happen when the caller supplied no +// base, there is nothing to install and this reports ErrNoVariantMatch. +func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (VariantSelection, error) { + if pin != "" { + for _, o := range options { + if strings.EqualFold(o.Variant.Model, pin) { + return VariantSelection{Option: o}, nil + } + } + return VariantSelection{}, fmt.Errorf("%w: %q is not among the %d variants of this model", ErrPinNotFound, pin, len(options)) + } + + type ranked struct { + option VariantOption + memory uint64 + rank int + preference int + feature int + } + + survivors := make([]ranked, 0, len(options)) + reasons := make([]string, 0, len(options)) + survivingVariants := 0 + + for i := range options { + o := options[i] + memory, known := o.EffectiveMemory() + + // The base skips both gates. There is nothing below it, so refusing it + // would make an entry every older LocalAI installs fine uninstallable on + // newer ones. + if !o.IsBase { + if !env.backendRuns(o.Backend) { + reasons = append(reasons, fmt.Sprintf("%s needs backend %q, which cannot run on this system", o.Variant.Model, o.Backend)) + continue + } + if known && memory > env.AvailableMemory { + reasons = append(reasons, fmt.Sprintf("%s needs %s of memory", o.Variant.Model, humanBytes(memory))) + continue + } + survivingVariants++ + } + + survivors = append(survivors, ranked{ + option: o, + memory: memory, + rank: rankOf(o, env), + preference: env.preferenceRank(o.Backend), + feature: env.servingFeatureRank(o), + }) + } + + if len(survivors) == 0 { + return VariantSelection{}, fmt.Errorf( + "%w: %s of memory available; variants: %s", + ErrNoVariantMatch, humanBytes(env.AvailableMemory), strings.Join(reasons, "; "), + ) + } + + // Four keys, in this order, and the order is the whole design: + // + // 1. rank, the fit tier. Fit is a fact about whether the host can hold the + // build at all, so nothing outranks it. + // 2. preference, the host's runtime order. Among builds the host can + // equally hold, the one on the more native runtime wins even when it is + // the smaller download. A Mac given an MLX build and a larger llama.cpp + // build should run MLX; ranking by size first would install the GGUF and + // leave the accelerated build unused. Do NOT move this below size. + // 3. feature, the serving feature order. Among builds on an equally + // preferred runtime, one that speculates or predicts several tokens per + // step beats the plain build of the same weights, because it answers + // faster for the same output. + // + // Engine outranks this deliberately. A serving feature makes the right + // engine faster; it does not make a wrong engine right. Wearing the + // wrong engine costs the whole GPU, whereas a plain build on the native + // engine is merely slower than it could be, so a DFlash build on the + // portable engine must not beat a plain build on the engine this host + // actually serves well. + // 4. memory, largest first, because among builds on equally preferred + // runtimes with the same serving feature a bigger build is a higher + // quality quantization of the same model. Neither preference key may + // flatten this: two plain builds on one backend are still ordered by + // size. + // + // Stable so that options equal on all four keep their authored order, which + // is the only thing order still decides. + sort.SliceStable(survivors, func(i, j int) bool { + if survivors[i].rank != survivors[j].rank { + return survivors[i].rank < survivors[j].rank + } + if survivors[i].preference != survivors[j].preference { + return survivors[i].preference < survivors[j].preference + } + if survivors[i].feature != survivors[j].feature { + return survivors[i].feature < survivors[j].feature + } + return survivors[i].memory > survivors[j].memory + }) + + winner := survivors[0].option + return VariantSelection{ + Option: winner, + // Only a base that won by default is worth reporting. A base that + // outranked live competition is an ordinary selection. + FellBackToBase: winner.IsBase && survivingVariants == 0, + Reasons: reasons, + }, nil +} + +// Fit tiers, best first. Within a tier the host's preferred runtime wins, then +// the preferred serving feature, then the larger footprint; see the sort in +// SelectVariant. +const ( + // rankProvenFit is a measured size that the host is measured to satisfy. + rankProvenFit = iota + // rankBase is the entry's own build when it is not a proven fit: either it + // needs more memory than the host reports, or its size could not be + // measured either. It still outranks any unsized variant, because it is the + // payload the entry is guaranteed to be able to install and a variant of + // unmeasurable size is a guess. Taking the guess on a host that cannot be + // shown to accommodate it is how an unreachable network silently changes + // what gets installed. + rankBase + // rankUnknownFit is a variant whose size nothing could measure. Nothing + // proves it does not fit, so it is not dropped, but nothing proves it does + // either, so it ranks last. + rankUnknownFit +) + +func rankOf(o VariantOption, env ResolveEnv) int { + if memory, known := o.EffectiveMemory(); known && memory <= env.AvailableMemory { + return rankProvenFit + } + if o.IsBase { + return rankBase + } + return rankUnknownFit +} + +func humanBytes(b uint64) string { + const unit = 1024 + if b < unit { + return fmt.Sprintf("%dB", b) + } + div, exp := uint64(unit), 0 + for n := b / unit; n >= unit; n /= unit { + div *= unit + exp++ + } + return fmt.Sprintf("%.1f%ciB", float64(b)/float64(div), "KMGTPE"[exp]) +} diff --git a/core/gallery/resolve_variant_test.go b/core/gallery/resolve_variant_test.go new file mode 100644 index 000000000..c5bd86481 --- /dev/null +++ b/core/gallery/resolve_variant_test.go @@ -0,0 +1,1089 @@ +package gallery_test + +import ( + "context" + "os" + "path/filepath" + "runtime" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/gallery" + "github.com/mudler/LocalAI/pkg/system" +) + +var _ = Describe("VariantOption.EffectiveMemory", func() { + It("reports no requirement when nothing was probed", func() { + o := gallery.VariantOption{Variant: gallery.Variant{Model: "x"}} + size, known := o.EffectiveMemory() + Expect(known).To(BeFalse()) + Expect(size).To(Equal(uint64(0))) + }) + + It("uses the probed size", func() { + o := gallery.VariantOption{Variant: gallery.Variant{Model: "x"}, ProbedMemory: 6 * 1024 * 1024 * 1024} + size, known := o.EffectiveMemory() + Expect(known).To(BeTrue()) + Expect(size).To(Equal(uint64(6 * 1024 * 1024 * 1024))) + }) + + It("treats a failed probe as unknown rather than as a zero requirement", func() { + // A probe that could not reach the network reports 0. Reading that as + // "needs nothing" would make an unreachable host look like the perfect + // fit and hand the user the largest download on offer. + o := gallery.VariantOption{Variant: gallery.Variant{Model: "x"}, ProbedMemory: 0} + _, known := o.EffectiveMemory() + Expect(known).To(BeFalse()) + }) +}) + +var _ = Describe("SelectVariant", func() { + gib := func(n uint64) uint64 { return n * 1024 * 1024 * 1024 } + + // Every size here is a probed one, because the probe is now the only source + // a variant's footprint can come from. A zero stands for the probe having + // been unable to tell, which is an unknown rather than a zero requirement. + option := func(model, backend string, probed uint64) gallery.VariantOption { + return gallery.VariantOption{ + Variant: gallery.Variant{Model: model}, + Backend: backend, + ProbedMemory: probed, + } + } + + // The base is exempt from every filter, which the fallback specs below pin + // down, but it is ranked against the variants like any other candidate, so + // its size is load-bearing. + base := func(model string, probed uint64) gallery.VariantOption { + o := option(model, "llama-cpp", probed) + o.IsBase = true + return o + } + + // linuxNvidia mirrors what SystemState.IsBackendCompatible does on a Linux + // box with an NVIDIA card: Darwin-only engines are out, everything else runs. + linuxNvidia := func(backend string) bool { + return backend != "mlx" && backend != "mlx-vlm" + } + + Describe("hardware filtering", func() { + It("never selects a variant whose backend cannot run on this host", func() { + // The MLX build is both the largest and the only thing that would + // otherwise win, so nothing but the backend gate can reject it. + options := []gallery.VariantOption{ + option("m-mlx-8bit", "mlx", gib(24)), + option("m-gguf-q8", "llama-cpp", gib(12)), + base("m-gguf-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(80), + BackendCompatible: linuxNvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q8")) + }) + + It("selects the same variant on a host whose backend gate does admit it", func() { + // The mirror image of the spec above, so the rejection is proven to + // come from the host and not from something intrinsic to the entry. + options := []gallery.VariantOption{ + option("m-mlx-8bit", "mlx", gib(24)), + option("m-gguf-q8", "llama-cpp", gib(12)), + base("m-gguf-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(80), + BackendCompatible: func(string) bool { return true }, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-mlx-8bit")) + }) + + It("treats every backend as runnable when the host cannot be inspected", func() { + options := []gallery.VariantOption{option("m-mlx-8bit", "mlx", gib(24)), base("m-gguf-q4", gib(6))} + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(80)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-mlx-8bit")) + }) + }) + + Describe("ranking", func() { + It("picks the largest variant that fits, not the first authored", func() { + // Authored smallest-first, so first-match would take m-q4 and any + // ranking that ignores size would too. + options := []gallery.VariantOption{ + option("m-q4", "llama-cpp", gib(6)), + option("m-q8", "llama-cpp", gib(12)), + option("m-f16", "llama-cpp", gib(24)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + Expect(selection.FellBackToBase).To(BeFalse()) + }) + + It("picks the largest variant regardless of authored order", func() { + // Same set, authored largest-first. Order must make no difference at + // all, which is the entire point of dropping ordered first-match. + options := []gallery.VariantOption{ + option("m-f16", "llama-cpp", gib(24)), + option("m-q8", "llama-cpp", gib(12)), + option("m-q4", "llama-cpp", gib(6)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + }) + + It("prefers a known fit over a variant of unknown size", func() { + // An unknown requirement is a guess. It survives the filter, because + // nothing proves it does not fit, but it must never displace a + // variant that is known to fit. + options := []gallery.VariantOption{ + option("m-unknown", "llama-cpp", 0), + option("m-q8", "llama-cpp", gib(12)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + }) + + It("keeps a variant of unknown size rather than dropping it", func() { + options := []gallery.VariantOption{ + option("m-unknown", "llama-cpp", 0), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(4)}, "") + Expect(err).ToNot(HaveOccurred()) + // Surviving is observable through the rejection reasons: a dropped + // variant is always accounted for there, and this one is not. + Expect(selection.Reasons).ToNot(ContainElement(ContainSubstring("m-unknown"))) + // It survives, but it does not win: the base is a sized, guaranteed + // payload and an unmeasurable variant is a guess. + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + Expect(selection.FellBackToBase).To(BeFalse()) + }) + + It("installs the base rather than an unsized variant on a host too small for either", func() { + // The exact shape 241 of the current index entries have: a referenced + // entry with no files and no size, whose probe can only answer + // "unknown". Ranking it above the base would install an unmeasured + // download on a machine with 2GiB, and would do so silently. + options := []gallery.VariantOption{ + option("m-unknown", "llama-cpp", 0), + base("m-base-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(2)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base-q4")) + Expect(selection.Option.IsBase).To(BeTrue()) + }) + + It("selects the base when the base is the largest option that fits", func() { + // A Q8 base offering a Q4 downgrade for small hosts is a natural + // authoring shape. Treating the base as a last resort would install + // the Q4 on every host large enough for the Q8 and permanently + // downgrade the user. + options := []gallery.VariantOption{ + option("m-q4", "llama-cpp", gib(4)), + base("m-base-q8", gib(8)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base-q8")) + // The Q4 survived every filter, so this is the base winning on rank + // and not the base being fallen back to. + Expect(selection.FellBackToBase).To(BeFalse()) + Expect(selection.Reasons).To(BeEmpty()) + }) + + It("selects a smaller variant when the base does not fit but the variant does", func() { + // The mirror of the spec above: the base competes, it does not win by + // default, so a host that cannot hold it must still take the downgrade + // the entry offers for exactly that case. + options := []gallery.VariantOption{ + option("m-q4", "llama-cpp", gib(4)), + base("m-base-q8", gib(8)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(6)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q4")) + }) + + It("reports why a probed size that does not fit was rejected", func() { + // Ranking and filtering both run off the probe, so a rejection has to + // be traceable back to the figure the probe returned. + options := []gallery.VariantOption{ + option("m-q4", "llama-cpp", gib(6)), + option("m-f16", "llama-cpp", gib(24)), + option("m-q8", "llama-cpp", gib(12)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-f16"))) + }) + + It("admits a variant needing exactly the memory available", func() { + options := []gallery.VariantOption{option("m-q8", "llama-cpp", gib(12)), base("m-base", gib(2))} + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(12)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + }) + }) + + Describe("ranking by host engine preference", func() { + // These are ENGINE NAMES, exactly what SystemState.EnginePreferenceTokens + // reports for these hosts and exactly what a gallery entry's `backend:` + // field holds. Build tags ("cuda", "rocm", "metal") belong to + // BackendPreferenceTokens and would match no engine name here, which is + // why every backend below is spelled as a real gallery engine. + // + // They are spelled out rather than read from the live machine so the + // specs pin the intended behaviour on every CI runner. + darwinMetal := []string{"mlx", "llama-cpp"} + nvidia := []string{"vllm", "sglang", "llama-cpp"} + + // A Mac runs both engines, so nothing is filtered here and preference is + // the only thing that can decide. + darwinRunsEverything := func(string) bool { return true } + + It("prefers a vLLM build to a larger llama.cpp build on nvidia", func() { + // The rule the engine table exists to express, and the one that was + // silently inert while the ranker was fed build tags: no gallery + // engine name contains "cuda", so every candidate scored equal and + // the larger llama.cpp build won. Emptying the nvidia rule in + // pkg/system fails this spec. + options := []gallery.VariantOption{ + option("m-vllm-awq", "vllm", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-gguf-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm-awq")) + }) + + It("takes the larger llama.cpp build on the same nvidia host once preference is unknown", func() { + // The mirror of the spec above, proving the vLLM win comes from the + // preference list and not from anything intrinsic to the option set. + // This is the state the whole feature was stuck in before the two + // vocabularies were separated. + options := []gallery.VariantOption{ + option("m-vllm-awq", "vllm", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-gguf-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q8")) + }) + + It("ranks a vllm-omni build with vllm, since the token is a substring", func() { + // Substring matching is deliberate: vllm-omni is a vLLM build and + // must inherit vLLM's rank rather than fall through to unranked. + options := []gallery.VariantOption{ + option("m-omni", "vllm-omni", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-base", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-omni")) + }) + + It("prefers an MLX build to a larger llama.cpp build on darwin", func() { + options := []gallery.VariantOption{ + option("m-mlx-4bit", "mlx", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-gguf-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + BackendCompatible: darwinRunsEverything, + EnginePreference: darwinMetal, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-mlx-4bit")) + }) + + It("takes the larger llama.cpp build on the same darwin host once preference is unknown", func() { + // The mirror of the spec above, proving the MLX win comes from the + // preference list and not from anything intrinsic to the option set. + options := []gallery.VariantOption{ + option("m-mlx-4bit", "mlx", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-gguf-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + BackendCompatible: darwinRunsEverything, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q8")) + }) + + It("still picks the largest fitting build among equally preferred engines", func() { + // Preference must not flatten size ordering: with one engine in + // play there is nothing left for it to decide. + options := []gallery.VariantOption{ + option("m-gguf-q4", "llama-cpp", gib(6)), + option("m-gguf-q8", "llama-cpp", gib(12)), + option("m-gguf-f16", "llama-cpp", gib(48)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q8")) + }) + + It("does not let a preferred engine rescue a build that does not fit", func() { + // Fit is a filter and preference is only a ranking among survivors. + // The vLLM build is both preferred and too large, so the llama.cpp + // build the host can actually hold has to win. + options := []gallery.VariantOption{ + option("m-vllm-fp16", "vllm", gib(48)), + option("m-gguf-q4", "llama-cpp", gib(6)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q4")) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-vllm-fp16"))) + }) + + It("orders engines absent from the table by size rather than dropping them", func() { + // Neither engine appears in the nvidia rule. Nothing is discarded + // and nothing is arbitrarily favoured; the host falls back to the + // size-only behaviour it had before preference existed. + // + // The base is left unsized so it ranks in the tier below a proven + // fit. It is a llama.cpp build, which the nvidia rule DOES rank, and + // letting it compete in the same tier would prove preference works + // rather than proving unlisted engines degrade to size. + options := []gallery.VariantOption{ + option("m-diffusers", "diffusers", gib(12)), + option("m-transformers", "transformers", gib(6)), + base("m-base", 0), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-diffusers")) + Expect(selection.Reasons).To(BeEmpty()) + }) + + It("ranks sglang between vllm and llama-cpp on nvidia", func() { + // A judgement call worth pinning: sglang is a GPU serving engine of + // the same class as vllm, so it outranks the portable engine, but it + // sits behind vllm. + options := []gallery.VariantOption{ + option("m-vllm", "vllm", gib(4)), + option("m-sglang", "sglang", gib(6)), + option("m-gguf", "llama-cpp", gib(8)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm")) + + withoutVLLM := []gallery.VariantOption{ + option("m-sglang", "sglang", gib(6)), + option("m-gguf", "llama-cpp", gib(8)), + base("m-base", gib(2)), + } + selection, err = gallery.SelectVariant(withoutVLLM, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-sglang")) + }) + + It("still installs the base when nothing fits, whatever the host prefers", func() { + options := []gallery.VariantOption{ + option("m-mlx-8bit", "mlx", gib(48)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-base", gib(64)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(4), + BackendCompatible: darwinRunsEverything, + EnginePreference: darwinMetal, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + Expect(selection.Option.IsBase).To(BeTrue()) + Expect(selection.FellBackToBase).To(BeTrue()) + }) + + It("honors a pin for a less preferred backend", func() { + // A pin is an operator override, so preference must not quietly + // redirect it any more than the memory filter does. + options := []gallery.VariantOption{ + option("m-mlx-4bit", "mlx", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-base", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + BackendCompatible: darwinRunsEverything, + EnginePreference: darwinMetal, + }, "m-gguf-q8") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q8")) + }) + }) + + Describe("ranking by serving feature", func() { + // These are SERVING FEATURES, exactly what + // system.ServingFeaturePreferenceTokens reports, and a third vocabulary + // after build tags and engine names. They are matched against a + // variant's declared TAGS and against nothing else. + // + // Spelled out rather than read from pkg/system so these specs pin the + // intended ordering rather than restating whatever the table says. + features := []string{"dflash", "mtp"} + nvidia := []string{"vllm", "sglang", "llama-cpp"} + + // A tag is the whole signal, so every candidate that is meant to carry + // a feature declares it here. Candidates built with `option` carry no + // tags and are therefore plain builds no matter what their name says, + // which several specs below rely on. + tagged := func(model string, probed uint64, tags ...string) gallery.VariantOption { + o := option(model, "llama-cpp", probed) + o.Tags = tags + return o + } + + It("prefers a speculative build to the plain build of the same weights", func() { + // The rule the feature table exists to express. Both builds fit and + // both run on the same engine, and the plain build is deliberately + // the larger one, so without this axis size would take it and the + // drafter pairing would never be installed. Emptying + // ServingFeaturePreference fails this spec. + options := []gallery.VariantOption{ + tagged("m-turbo-q4", gib(14), "llm", "dflash"), + tagged("m-q8", gib(20), "llm"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4")) + }) + + It("takes the larger plain build once the feature preference is unknown", func() { + // The mirror of the spec above on the identical option set, proving + // the DFlash win comes from the feature list rather than from + // anything intrinsic to the set. + options := []gallery.VariantOption{ + tagged("m-turbo-q4", gib(14), "llm", "dflash"), + tagged("m-q8", gib(20), "llm"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + }) + + It("prefers dflash to mtp", func() { + // Both are speculative pairings, so only the table's order can + // separate them, and the MTP build is deliberately the larger one so + // size cannot be what decides. + options := []gallery.VariantOption{ + tagged("m-alpha-q8", gib(20), "llm", "mtp"), + tagged("m-beta-q4", gib(14), "llm", "dflash"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-beta-q4")) + }) + + It("does not let a preferred feature rescue a build that does not fit", func() { + // A drafter pairing is strictly larger than the plain build, so this + // is the ordinary case on a small host rather than a corner one. Fit + // is a filter; the feature preference only ranks survivors. + options := []gallery.VariantOption{ + tagged("m-turbo-f16", gib(48), "llm", "dflash"), + tagged("m-q8", gib(12), "llm"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-turbo-f16"))) + }) + + It("lets the host engine preference outrank the serving feature", func() { + // A serving feature makes the right engine faster; it does not make + // a wrong engine right. On nvidia the plain vLLM build therefore + // beats a DFlash llama.cpp build even though both fit and the + // llama.cpp one is larger. + options := []gallery.VariantOption{ + tagged("m-turbo-q8", gib(24), "llm", "dflash"), + option("m-vllm-awq", "vllm", gib(8)), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm-awq")) + }) + + It("still picks the largest fitting build among equally featured builds", func() { + // Neither preference key may flatten size ordering: with one engine + // and one feature in play there is nothing left for them to decide. + options := []gallery.VariantOption{ + tagged("m-turbo-q4", gib(8), "llm", "dflash"), + tagged("m-turbo-q8", gib(14), "llm", "dflash"), + tagged("m-turbo-f16", gib(48), "llm", "dflash"), + base("m-q4", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8")) + }) + + It("leaves an unfeatured build ranked last rather than dropping it", func() { + // Ranking never filters. With only plain builds on offer the axis + // scores them uniformly and selection degrades to size alone. + options := []gallery.VariantOption{ + option("m-q8", "llama-cpp", gib(12)), + option("m-q4", "llama-cpp", gib(6)), + base("m-q2", gib(3)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + Expect(selection.Reasons).To(BeEmpty()) + }) + + Describe("reading the declared tags", func() { + It("prefers a build whose tag declares the feature its name does not", func() { + // The case tags exist for. "m-turbo-q8" carries MTP heads but + // spells nothing in its name, which is the shape most of the + // gallery's MTP entries had before they were tagged. It is also + // the smaller build, so only the feature axis can lift it. + options := []gallery.VariantOption{ + tagged("m-turbo-q8", gib(14), "llm", "gguf", "mtp"), + tagged("m-plain-q8", gib(20), "llm", "gguf"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8")) + }) + + It("does NOT prefer a build that only its name declares, with no tags at all", func() { + // The inversion of the old name-fallback spec, and the + // regression this change exists to guard. A name is + // author-supplied free text and a naming convention is not a + // contract: "m-nvfp4-mtp" is exactly the shape of the gallery's + // NVFP4 entries, whose weights carry MTP heads while the entry + // enables no speculative decoding at all, so ranking it as a + // speculative build made it win the feature axis without being + // any faster. With no tag it is a plain build and the larger + // plain build takes it on size. + options := []gallery.VariantOption{ + option("m-nvfp4-mtp", "llama-cpp", gib(14)), + tagged("m-plain-q8", gib(20), "llm", "gguf"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-plain-q8")) + }) + + It("lets a tagged build beat a larger one whose name says the same feature", func() { + // The sharper form of the spec above: the name half is not + // merely unnecessary, it is not consulted. The untagged + // "m-dflash" would outrank the tagged MTP build on the old + // name-first ordering, and is the larger build besides, so it + // wins on either of the two ways the name could still be read. + options := []gallery.VariantOption{ + option("m-dflash", "llama-cpp", gib(20)), + tagged("m-turbo-q4", gib(14), "llm", "mtp"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4")) + }) + + It("matches a tag regardless of the case it was written in", func() { + // Gallery tags are author-supplied, so the same declaration + // arrives in whatever case the author typed. A curator who + // writes "MTP" has declared the feature just as plainly as one + // who writes "mtp", and the sole signal must not turn on that. + options := []gallery.VariantOption{ + tagged("m-turbo-q8", gib(14), "LLM", "MTP"), + tagged("m-plain-q8", gib(20), "llm", "gguf"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8")) + }) + + It("prefers dflash to mtp when both are declared by tag", func() { + // The table's order has to survive on the tag path with both + // features spelled out explicitly, and the MTP build is + // deliberately the larger one so size cannot be what decides. + options := []gallery.VariantOption{ + tagged("m-alpha-q8", gib(20), "llm", "mtp"), + tagged("m-beta-q8", gib(14), "llm", "dflash"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-beta-q8")) + }) + + It("does not read a feature out of an unrelated tag", func() { + // Tags are compared whole, so a tag that merely contains a + // feature token declares nothing. "multimodal" contains no + // feature; "smtp" does contain "mtp" as a substring and must + // not count either. The mail model is the larger build, so a + // false positive would hand it the win. + options := []gallery.VariantOption{ + tagged("m-mail-q8", gib(24), "llm", "multimodal", "smtp"), + tagged("m-turbo-q4", gib(14), "llm", "mtp"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4")) + }) + + It("does not let a tag rescue a build that does not fit", func() { + // Fit still outranks the feature axis, whichever signal + // declared it. + options := []gallery.VariantOption{ + tagged("m-turbo-f16", gib(48), "llm", "mtp"), + tagged("m-plain-q8", gib(12), "llm"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-plain-q8")) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-turbo-f16"))) + }) + + It("lets the host engine preference outrank a tagged serving feature", func() { + // Engine beats feature no matter which signal carried the + // feature: a tag does not make a wrong engine right. + options := []gallery.VariantOption{ + tagged("m-turbo-q8", gib(24), "llm", "mtp"), + option("m-vllm-awq", "vllm", gib(8)), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm-awq")) + }) + + It("ignores tags entirely when no feature preference is configured", func() { + // The axis stops discriminating with an empty list, tags or no + // tags, and selection degrades to size alone as it always did. + options := []gallery.VariantOption{ + tagged("m-turbo-q8", gib(14), "llm", "mtp"), + tagged("m-plain-q8", gib(20), "llm"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-plain-q8")) + }) + }) + }) + + Describe("falling back to the base", func() { + It("selects the base when nothing else fits", func() { + options := []gallery.VariantOption{ + option("m-q8", "llama-cpp", gib(12)), + option("m-f16", "llama-cpp", gib(24)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(4)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + Expect(selection.Option.IsBase).To(BeTrue()) + Expect(selection.FellBackToBase).To(BeTrue()) + Expect(selection.Reasons).To(HaveLen(2)) + }) + + It("selects the base even when the base does not fit either", func() { + // The base is exempt from the memory filter, not merely favoured by + // it: there is nothing below it, so refusing here would make an entry + // every older client installs fine uninstallable on newer ones. + options := []gallery.VariantOption{option("m-q8", "llama-cpp", gib(12)), base("m-base", gib(2))} + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: 0}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + Expect(selection.Option.IsBase).To(BeTrue()) + Expect(selection.FellBackToBase).To(BeTrue()) + }) + + It("prefers the base to an unsized variant even when the base itself is unsized", func() { + // Neither can be shown to fit, so nothing separates them on size. The + // base is still the payload the entry is guaranteed to install. + options := []gallery.VariantOption{ + option("m-unknown", "llama-cpp", 0), + base("m-base", 0), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(8)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + }) + + It("explains why each variant was rejected", func() { + options := []gallery.VariantOption{ + option("m-mlx", "mlx", gib(8)), + option("m-f16", "llama-cpp", gib(24)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(4), + BackendCompatible: linuxNvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("cannot run on this system"))) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("24.0GiB"))) + }) + + It("errors when the caller supplies no base at all", func() { + options := []gallery.VariantOption{option("m-f16", "llama-cpp", gib(24))} + + _, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(4)}, "") + Expect(err).To(MatchError(gallery.ErrNoVariantMatch)) + }) + }) + + Describe("explicit selection", func() { + It("honors a pin the hardware would never have chosen", func() { + options := []gallery.VariantOption{ + option("m-mlx", "mlx", gib(64)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(4), + BackendCompatible: linuxNvidia, + }, "m-mlx") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-mlx")) + }) + + It("honors a pin naming the base, declining an upgrade that fits", func() { + options := []gallery.VariantOption{option("m-f16", "llama-cpp", gib(8)), base("m-base", gib(2))} + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(64)}, "m-base") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + }) + + It("matches a pin case-insensitively", func() { + options := []gallery.VariantOption{option("m-f16", "llama-cpp", gib(8)), base("m-base", gib(2))} + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(64)}, "M-F16") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-f16")) + }) + + It("fails loudly when the pin names nothing in the list", func() { + options := []gallery.VariantOption{option("m-f16", "llama-cpp", gib(8)), base("m-base", gib(2))} + + _, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(64)}, "m-gone") + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("m-gone")) + }) + }) +}) + +var _ = Describe("HostResolveEnv engine preference wiring", func() { + // The specs above feed SelectVariant a hand-written token list, which proves + // the ranker but not that the host actually reaches the engine table. These + // drive the REAL table through the REAL wiring, so emptying + // engineNamePreferenceRules in pkg/system, or reverting this field to + // BackendPreferenceTokens, fails here. + var origEnv, origRunFileEnv string + const capabilityEnv = "LOCALAI_FORCE_META_BACKEND_CAPABILITY" + const capabilityRunFileEnv = "LOCALAI_FORCE_META_BACKEND_CAPABILITY_RUN_FILE" + // What getSystemCapabilities reports for a host with no usable accelerator. + const noGPUCapability = "default" + + BeforeEach(func() { + origEnv = os.Getenv(capabilityEnv) + origRunFileEnv = os.Getenv(capabilityRunFileEnv) + }) + + AfterEach(func() { + if origEnv != "" { + Expect(os.Setenv(capabilityEnv, origEnv)).To(Succeed()) + } else { + Expect(os.Unsetenv(capabilityEnv)).To(Succeed()) + } + if origRunFileEnv != "" { + Expect(os.Setenv(capabilityRunFileEnv, origRunFileEnv)).To(Succeed()) + } else { + Expect(os.Unsetenv(capabilityRunFileEnv)).To(Succeed()) + } + }) + + envFor := func(capability string) gallery.ResolveEnv { + GinkgoHelper() + Expect(os.Setenv(capabilityEnv, capability)).To(Succeed()) + return gallery.HostResolveEnv(context.Background(), &system.SystemState{}) + } + + It("hands the ranker engine names an NVIDIA host's gallery entries can match", func() { + preference := envFor("nvidia-cuda-12").EnginePreference + Expect(preference).To(Equal([]string{"vllm", "sglang", "llama-cpp"})) + }) + + It("installs the vLLM build over a larger llama.cpp one on a real NVIDIA host", func() { + // End to end on the live table: the exact behaviour that was silently + // dead while the ranker was fed build tags. + env := envFor("nvidia-cuda-12") + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-vllm"}, Backend: "vllm", ProbedMemory: 8 * 1024 * 1024 * 1024}, + {Variant: gallery.Variant{Model: "m-gguf"}, Backend: "llama-cpp", ProbedMemory: 24 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm")) + }) + + It("installs the MLX build over a larger llama.cpp one on a real darwin host", func() { + env := envFor("metal") + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + // The real gate rejects mlx off darwin, and this spec is about ranking. + env.BackendCompatible = func(string) bool { return true } + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-mlx"}, Backend: "mlx", ProbedMemory: 8 * 1024 * 1024 * 1024}, + {Variant: gallery.Variant{Model: "m-gguf"}, Backend: "llama-cpp", ProbedMemory: 24 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-mlx")) + }) + + It("installs the llama.cpp build over a larger vLLM one on a host with no GPU", func() { + // The hole this rule closes. IsBackendCompatible keys on the engine + // name, and "vllm" carries no darwin/cuda/rocm/sycl token, so a vLLM + // build is NOT filtered out here. Emptying the default rule in + // pkg/system puts the larger vLLM build back on a CPU-only box. + env := envFor(noGPUCapability) + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-vllm"}, Backend: "vllm", ProbedMemory: 24 * 1024 * 1024 * 1024}, + {Variant: gallery.Variant{Model: "m-gguf"}, Backend: "llama-cpp", ProbedMemory: 8 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf")) + }) + + It("installs the llama.cpp build over a larger vLLM one on an intel mac", func() { + env := envFor("darwin-x86") + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-vllm"}, Backend: "vllm", ProbedMemory: 24 * 1024 * 1024 * 1024}, + {Variant: gallery.Variant{Model: "m-gguf"}, Backend: "llama-cpp", ProbedMemory: 8 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf")) + }) + + It("still installs a vLLM build on a host with no GPU when it is the only one offered", func() { + // Preference ORDERS survivors, it never filters them. Demoting vLLM must + // not make a model published only as a vLLM build uninstallable. + env := envFor(noGPUCapability) + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-vllm"}, Backend: "vllm", ProbedMemory: 8 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm")) + }) + + It("prefers llama.cpp on a GPU host with too little VRAM to serve from", func() { + // This host has a GPU, yet getSystemCapabilities reports "default" + // because it is under the 4 GiB floor. Driven through the real detector + // rather than a forced capability, so that mapping is exercised too. + if runtime.GOOS == "darwin" { + Skip("darwin reports metal or darwin-x86 before the VRAM floor is consulted") + } + Expect(os.Unsetenv(capabilityEnv)).To(Succeed()) + // A capability run file on the machine would override detection. + Expect(os.Setenv(capabilityRunFileEnv, filepath.Join(GinkgoT().TempDir(), "absent"))).To(Succeed()) + + state := &system.SystemState{GPUVendor: system.Nvidia, VRAM: 2 * 1024 * 1024 * 1024} + Expect(state.DetectedCapability()).To(Equal(noGPUCapability)) + + env := gallery.HostResolveEnv(context.Background(), state) + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-vllm"}, Backend: "vllm", ProbedMemory: 24 * 1024 * 1024 * 1024}, + {Variant: gallery.Variant{Model: "m-gguf"}, Backend: "llama-cpp", ProbedMemory: 8 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf")) + }) + + It("carries no build tag into the ranker, whatever the host", func() { + // The wiring-level lock. BackendPreferenceTokens returns build tags for + // every capability, so if it is ever wired back into this field, one of + // these tags shows up here. + for _, capability := range []string{"nvidia-cuda-12", "amd", "intel", "metal", "vulkan", "default"} { + Expect(envFor(capability).EnginePreference).ToNot( + ContainElements("cuda", "rocm", "hip", "sycl", "metal", "cpu", "darwin-x86"), + "capability %q leaked a build tag into the variant ranker", capability) + } + }) +}) diff --git a/core/gallery/variant.go b/core/gallery/variant.go new file mode 100644 index 000000000..ad7ff0932 --- /dev/null +++ b/core/gallery/variant.go @@ -0,0 +1,18 @@ +package gallery + +// Variant is one option in a gallery entry's variant list. It references an +// existing gallery entry by name, and that is all an author has to write. +// +// Authored order carries no meaning. Selection filters out what this host +// cannot run and then ranks what is left, so hardware knowledge lives in the +// selector rather than being pushed onto whoever edits the gallery. +// +// There is deliberately no authored memory figure here. When the live probe +// misreads a variant's footprint, the fix is the referenced entry's own `size:` +// field: the estimator already prefers a declared size over its own guesswork, +// so correcting it there fixes the figure for every consumer rather than only +// for this one selection pass. +type Variant struct { + // Model is the name of a gallery entry that declares no variants of its own. + Model string `json:"model" yaml:"model"` +} diff --git a/core/gallery/variant_quantization.go b/core/gallery/variant_quantization.go new file mode 100644 index 000000000..176620d48 --- /dev/null +++ b/core/gallery/variant_quantization.go @@ -0,0 +1,147 @@ +package gallery + +import ( + "path" + "regexp" + "strings" +) + +// quantizationToken matches one filename segment that names a weight format. +// +// The alternatives, in order: the GGUF k-quant/legacy family with any number of +// trailing qualifiers (`q8_0`, `q4_k_m`, `iq2_xxs`, `pq2_0`, `q2_0_g128`), the +// float formats (`f16`, `fp16`, `bf16`, `f32`) including the vendor 4-bit ones +// (`nvfp4`, `mxfp4`), and the bit-count spellings the MLX and AWQ ecosystems +// use (`4bit`, `int8`). +// +// It is anchored at both ends because it is applied to an already-split +// segment. Matching mid-string would let a repo or model name containing a +// quant-shaped substring be reported as the quantization. +var quantizationToken = regexp.MustCompile(`^(?:p?i?q[0-9]+(?:_[0-9a-z]+)*|f(?:p)?(?:8|16|32)|bf16|(?:nv|mx)fp[0-9]+|int[0-9]+|[0-9]+bit)$`) + +// weightExtensions are stripped before a filename is split into segments, so a +// model whose quantization is the last thing in its name (`...-Q8_0.gguf`) is +// not read as the segment `gguf`. +var weightExtensions = []string{".gguf", ".safetensors", ".bin", ".pt", ".pth", ".onnx"} + +// quantizationFromFilename reports the weight format named in a model +// filename, uppercased, or "" when the name does not declare one. +// +// Segments are scanned from the END because that is where the qualifier sits by +// near-universal convention (`Ternary-Bonsai-27B-Q2_g64.gguf`), and because a +// repo path may itself carry a quant-shaped segment that describes the +// repository rather than this file. +// +// Only `-` and `.` split segments; `_` deliberately does not, because it is the +// separator INSIDE a quant token (`Q4_K_M`) and splitting on it would report +// `Q4` for a build that is not Q4. +// +// Two passes, in decreasing order of confidence. A whole segment naming a +// format is the unambiguous case and always wins. Only when no segment does is +// the `_`-delimited tail of a segment considered, which catches the authoring +// style that runs the format into the model name (`gemma-4-E2B_q4_0-it.gguf`, +// a real and populous gallery family). Running the loose pass second rather +// than inline keeps a precise match from ever losing to a fuzzy one further +// right in the name. +// +// The result is uppercased so one gallery does not show `Q2_g64` next to +// `Q2_G64` for what is the same format spelled two ways by two authors. +func quantizationFromFilename(filename string) string { + if filename == "" { + return "" + } + + base := path.Base(strings.TrimSpace(filename)) + lower := strings.ToLower(base) + for _, ext := range weightExtensions { + if strings.HasSuffix(lower, ext) { + base = base[:len(base)-len(ext)] + break + } + } + + segments := strings.FieldsFunc(base, func(r rune) bool { return r == '-' || r == '.' }) + + for i := len(segments) - 1; i >= 0; i-- { + if quantizationToken.MatchString(strings.ToLower(segments[i])) { + return strings.ToUpper(segments[i]) + } + } + + for i := len(segments) - 1; i >= 0; i-- { + if tail := quantizationTail(segments[i]); tail != "" { + return tail + } + } + return "" +} + +// quantizationTail reports a weight format run into the tail of a segment, +// uppercased, or "" when there is none. +// +// It walks the `_`-delimited tails from the LONGEST first, so `e2b_q4_0` yields +// `Q4_0` rather than the `0` a shortest-first walk would reach. The full +// segment is not retried here; the caller has already ruled it out. +func quantizationTail(segment string) string { + for i, r := range segment { + if r != '_' { + continue + } + tail := segment[i+1:] + if quantizationToken.MatchString(strings.ToLower(tail)) { + return strings.ToUpper(tail) + } + } + return "" +} + +// quantizationOfEntry reports the weight format a gallery entry installs. +// +// `overrides.parameters.model` is preferred over the file list because it names +// the file the backend is actually pointed at. An entry routinely ships more +// than one weight file (a vision tower alongside the language model, for +// instance), and those companions carry their own, different quantization: the +// Bonsai entries pair a Q2_0 model with a Q8_0 mmproj, so reading the file list +// first would report Q8_0 for a Q2_0 build. +// +// Returning "" is a normal outcome, not a failure. A backend served from a +// directory of safetensors, or an entry whose name simply does not encode a +// format, has no quantization to report and callers must render its absence +// rather than a guess. +func quantizationOfEntry(entry *GalleryModel) string { + if entry == nil { + return "" + } + + if q := quantizationFromFilename(overrideModelParameter(entry)); q != "" { + return q + } + + // The file list is the fallback for entries that install a config_file + // naming no model parameter of their own. First file wins: the language + // model is listed first by convention, and there is nothing better to go on + // once the authoritative pointer is absent. + for _, f := range entry.AdditionalFiles { + if q := quantizationFromFilename(f.Filename); q != "" { + return q + } + } + return "" +} + +// overrideModelParameter digs `overrides.parameters.model` out of an entry. +// +// Overrides are free-form YAML decoded into map[string]any, so every level has +// to be type-asserted; a gallery author who writes a list or a scalar where a +// map belongs gets "" here rather than a panic in the listing handler. +func overrideModelParameter(entry *GalleryModel) string { + parameters, ok := entry.Overrides["parameters"].(map[string]any) + if !ok { + return "" + } + model, ok := parameters["model"].(string) + if !ok { + return "" + } + return model +} diff --git a/core/gallery/variant_quantization_internal_test.go b/core/gallery/variant_quantization_internal_test.go new file mode 100644 index 000000000..70f5a1b7f --- /dev/null +++ b/core/gallery/variant_quantization_internal_test.go @@ -0,0 +1,126 @@ +package gallery + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("quantizationFromFilename", func() { + DescribeTable("reads the weight format out of a model filename", + func(filename, expected string) { + Expect(quantizationFromFilename(filename)).To(Equal(expected)) + }, + Entry("a plain legacy quant", "Ternary-Bonsai-27B-Q2_0.gguf", "Q2_0"), + Entry("a packed quant", "Ternary-Bonsai-27B-PQ2_0.gguf", "PQ2_0"), + // The pair the whole feature exists for: two builds of one model whose + // names differ only here, and whose sizes are close enough that a user + // cannot tell them apart from the size column. + Entry("a group-size qualified quant", "Ternary-Bonsai-27B-Q2_g64.gguf", "Q2_G64"), + Entry("a multi-qualifier quant", "Ternary-Bonsai-27B-Q2_0_g128.gguf", "Q2_0_G128"), + Entry("a k-quant", "Qwen3-8B-Q4_K_M.gguf", "Q4_K_M"), + Entry("an i-quant", "Qwen3-8B-IQ2_XXS.gguf", "IQ2_XXS"), + Entry("a float format", "Qwen3-8B-f16.gguf", "F16"), + Entry("bfloat16", "Qwen3-8B-bf16.safetensors", "BF16"), + Entry("an MLX bit count", "Qwen3-8B-4bit.safetensors", "4BIT"), + // A real gallery shape: the format sits mid-name with serving-feature + // and repo-suffix segments after it, so the scan cannot simply take + // the last segment. + Entry("a vendor 4-bit float behind later segments", "Qwen3.6-27B-NVFP4-MTP-GGUF.gguf", "NVFP4"), + Entry("the other vendor 4-bit float", "Qwen3-8B-MXFP4.gguf", "MXFP4"), + // The gemma QAT authoring style: the format is run into the model name + // with an underscore rather than set off by a dash. + Entry("a format run into the model name", "gemma-4-E2B_q4_0-it.gguf", "Q4_0"), + Entry("a format run in at the end", "model_q8_0.gguf", "Q8_0"), + Entry("an integer width", "Qwen3-8B-int8.safetensors", "INT8"), + Entry("a dot-separated qualifier", "qwen3-8b.Q8_0.gguf", "Q8_0"), + // A repo path is routinely prefixed to the filename in the gallery, and + // its own segments must not be mistaken for this file's format. + Entry("a path prefix", "bonsai/models/Q4-repo/Model-Q8_0.gguf", "Q8_0"), + ) + + DescribeTable("reports nothing when the name declares no format", + func(filename string) { + Expect(quantizationFromFilename(filename)).To(BeEmpty()) + }, + Entry("an empty name", ""), + // The degrade case the UI has to render: a backend served from a + // directory of weights names no format anywhere. + Entry("a bare model name", "Qwen3-8B.safetensors"), + Entry("a directory", "models/qwen3-8b/"), + // A parameter count is quant-shaped to a careless matcher: it is a + // digit run next to a letter, and reporting "27B" as a quantization + // would be worse than reporting nothing. + Entry("a parameter count", "Ternary-Bonsai-27B.gguf"), + Entry("an extension alone", "model.gguf"), + ) + + It("prefers a whole segment over a tail further right in the name", func() { + // The two passes exist for this: a precise, dash-delimited format must + // never lose to a looser mid-segment match that happens to sit later. + Expect(quantizationFromFilename("Model-Q4_K_M-repo_8bit.gguf")).To(Equal("Q4_K_M")) + }) + + It("takes the longest tail, not the shortest", func() { + // A shortest-first walk over `e2b_q4_0` reaches `0` before `q4_0`, and + // `0` is not a weight format. + Expect(quantizationFromFilename("gemma_q4_0.gguf")).To(Equal("Q4_0")) + }) + + It("does not split on the separator inside a quant token", func() { + // Splitting on `_` as well as `-` would truncate every k-quant to its + // family and report a Q4_K_M build as plain "Q4", which names a + // different format that the entry does not ship. + Expect(quantizationFromFilename("Model-Q4_K_S.gguf")).To(Equal("Q4_K_S")) + }) +}) + +var _ = Describe("quantizationOfEntry", func() { + It("prefers the served model parameter over the file list", func() { + // The Bonsai shape: a Q2_0 language model shipped alongside a Q8_0 + // vision tower. Reading the file list first reports the mmproj's + // format, which describes a companion the user is not choosing. + entry := &GalleryModel{ + Overrides: map[string]any{ + "parameters": map[string]any{"model": "bonsai/models/Bonsai-27B-Q2_0.gguf"}, + }, + } + entry.AdditionalFiles = []File{ + {Filename: "bonsai/mmproj/Bonsai-27B-mmproj-Q8_0.gguf"}, + {Filename: "bonsai/models/Bonsai-27B-Q2_0.gguf"}, + } + + Expect(quantizationOfEntry(entry)).To(Equal("Q2_0")) + }) + + It("falls back to the file list when no model parameter is set", func() { + entry := &GalleryModel{} + entry.AdditionalFiles = []File{{Filename: "models/Qwen3-8B-Q6_K.gguf"}} + + Expect(quantizationOfEntry(entry)).To(Equal("Q6_K")) + }) + + It("reports nothing for a nil entry", func() { + Expect(quantizationOfEntry(nil)).To(BeEmpty()) + }) + + It("reports nothing when the entry ships no recognisable format", func() { + entry := &GalleryModel{} + entry.AdditionalFiles = []File{{Filename: "models/Qwen3-8B/model.safetensors"}} + + Expect(quantizationOfEntry(entry)).To(BeEmpty()) + }) + + DescribeTable("survives overrides that are not shaped like a parameter map", + func(overrides map[string]any) { + entry := &GalleryModel{Overrides: overrides} + // A gallery author's typo must degrade to "unknown format", never + // panic inside the listing handler. + Expect(quantizationOfEntry(entry)).To(BeEmpty()) + }, + Entry("no overrides at all", nil), + Entry("parameters is a scalar", map[string]any{"parameters": "Q8_0"}), + Entry("parameters is a list", map[string]any{"parameters": []any{"model"}}), + Entry("model is not a string", map[string]any{"parameters": map[string]any{"model": 42}}), + Entry("model is absent", map[string]any{"parameters": map[string]any{"context_size": 8192}}), + ) +}) diff --git a/core/gallery/variants_install_test.go b/core/gallery/variants_install_test.go new file mode 100644 index 000000000..f677d0561 --- /dev/null +++ b/core/gallery/variants_install_test.go @@ -0,0 +1,757 @@ +package gallery_test + +import ( + "context" + "fmt" + "os" + "path/filepath" + "runtime" + + "dario.cat/mergo" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/gallery" + "github.com/mudler/LocalAI/pkg/system" +) + +var _ = Describe("GalleryModel variant declarations", func() { + It("declares no variants when the key is absent", func() { + m := gallery.GalleryModel{} + m.Name = "plain" + Expect(m.HasVariants()).To(BeFalse()) + }) + + It("declares variants when the key is present", func() { + m := gallery.GalleryModel{Variants: []gallery.Variant{{Model: "x"}}} + Expect(m.HasVariants()).To(BeTrue()) + }) + + It("parses a variant list", func() { + var m gallery.GalleryModel + err := yaml.Unmarshal([]byte(` +name: qwen3.6-27b +url: "github:example/repo/qwen3.6-27b.yaml@master" +variants: + - model: qwen3.6-27b-mlx-8bit + - model: qwen3.6-27b-gguf-q8 +`), &m) + Expect(err).ToNot(HaveOccurred()) + Expect(m.Name).To(Equal("qwen3.6-27b")) + Expect(m.URL).To(Equal("github:example/repo/qwen3.6-27b.yaml@master")) + Expect(m.HasVariants()).To(BeTrue()) + Expect(m.Variants).To(HaveLen(2)) + // A variant is nothing but a name, which is the whole of the authoring + // surface. + Expect(m.Variants[0].Model).To(Equal("qwen3.6-27b-mlx-8bit")) + Expect(m.Variants[1].Model).To(Equal("qwen3.6-27b-gguf-q8")) + }) +}) + +var _ = Describe("ResolveVariant", func() { + gib := func(n uint64) uint64 { return n * 1024 * 1024 * 1024 } + + // runsEverything keeps these specs about resolution rather than about the + // hardware of whatever machine runs them. + runsEverything := func(string) bool { return true } + + newModel := func(name, url, description, icon string) *gallery.GalleryModel { + m := &gallery.GalleryModel{} + m.Name = name + m.URL = url + m.Description = description + m.Icon = icon + return m + } + + var models []*gallery.GalleryModel + var base *gallery.GalleryModel + + BeforeEach(func() { + upgrade := newModel("qwen3-8b-vllm-awq", "file://vllm.yaml", "AWQ variant", "vllm.png") + upgrade.Backend = "vllm" + // The base is an ordinary, complete entry that happens to list an + // alternative build of itself, which is the whole point of the design. + base = newModel("qwen3-8b-gguf-q4", "file://gguf.yaml", "Qwen3 8B Q4", "qwen.png") + base.Backend = "llama-cpp" + base.Tags = []string{"llm"} + base.Variants = []gallery.Variant{{Model: "qwen3-8b-vllm-awq"}} + models = []*gallery.GalleryModel{upgrade, base} + }) + + // The AWQ build measures 20GiB, so the specs below vary only the host and + // read off which payload resolution lands on. The probe is injected rather + // than performed, so nothing here reaches the network. + env := func(memory uint64) gallery.ResolveEnv { + return gallery.ResolveEnv{ + AvailableMemory: memory, + BackendCompatible: runsEverything, + ProbeMemory: func(target *gallery.GalleryModel) uint64 { + if target.Name == "qwen3-8b-vllm-awq" { + return 20 * 1024 * 1024 * 1024 + } + return 0 + }, + } + } + + It("installs a fitting variant's payload under the entry's name", func() { + resolved, variant, err := gallery.ResolveVariant(models, base, env(gib(24)), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-vllm-awq")) + Expect(resolved.Name).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://vllm.yaml")) + }) + + It("carries the referenced entry's backend into the compatibility check", func() { + // The variant itself says nothing about hardware, so the backend can + // only come from the entry it names. Rejecting exactly that backend must + // therefore be enough to rule the variant out. + resolved, variant, err := gallery.ResolveVariant(models, base, gallery.ResolveEnv{ + AvailableMemory: gib(64), + BackendCompatible: func(backend string) bool { return backend != "vllm" }, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("carries the referenced entry's tags into the serving feature ranking", func() { + // The tags that rank a variant are the REFERENCED entry's, not the + // declaring entry's: the feature belongs to the build that would be + // installed. Only the GGUF build here is tagged with a feature, and it + // is deliberately the smaller of the two, so it can only win if the + // lookup read a tag off the entry rather than off the variant name or + // off the parent. + speculative := gallery.GalleryModel{} + speculative.Name = "qwen3-8b-gguf-turbo" + speculative.URL = "file://turbo.yaml" + speculative.Backend = "llama-cpp" + speculative.Tags = []string{"llm", "mtp"} + + parent := gallery.GalleryModel{} + parent.Name = "qwen3-8b-gguf-q4" + parent.URL = "file://gguf.yaml" + parent.Backend = "llama-cpp" + parent.Variants = []gallery.Variant{{Model: "qwen3-8b-vllm-awq"}, {Model: "qwen3-8b-gguf-turbo"}} + + catalog := []*gallery.GalleryModel{models[0], &speculative, &parent} + + resolved, variant, err := gallery.ResolveVariant(catalog, &parent, gallery.ResolveEnv{ + AvailableMemory: gib(64), + BackendCompatible: runsEverything, + // Only llama-cpp is ranked, so the vLLM build cannot win on engine + // and the contest comes down to the feature axis. + EnginePreference: []string{"llama-cpp"}, + ServingFeaturePreference: []string{"dflash", "mtp"}, + ProbeMemory: func(target *gallery.GalleryModel) uint64 { + if target.Name == "qwen3-8b-gguf-turbo" { + return gib(10) + } + return gib(20) + }, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-turbo")) + Expect(resolved.URL).To(Equal("file://turbo.yaml")) + }) + + It("falls back to the entry's own payload when no variant fits", func() { + resolved, variant, err := gallery.ResolveVariant(models, base, env(gib(8)), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("installs the entry on a host with no memory budget at all", func() { + // The base carries no floor of its own precisely so this can never + // refuse; an unreadable host reports 0 and must still get the entry. + resolved, variant, err := gallery.ResolveVariant(models, base, env(0), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("strips the variant list from the resolved entry", func() { + // A resolved entry is a concrete install target. Leaving the list on it + // would let a second selection pass fire on an already-resolved entry. + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(8)), "") + Expect(err).ToNot(HaveOccurred()) + Expect(resolved.HasVariants()).To(BeFalse()) + }) + + Describe("the live size probe", func() { + // The probe is injected rather than performed, so these specs pin an + // exact size, or an exact failure, without reaching the network. + probing := func(memory uint64, sizes map[string]uint64) gallery.ResolveEnv { + e := env(memory) + e.ProbeMemory = func(target *gallery.GalleryModel) uint64 { return sizes[target.Name] } + return e + } + + It("selects a variant the probe shows to fit", func() { + resolved, variant, err := gallery.ResolveVariant(models, base, + probing(gib(24), map[string]uint64{"qwen3-8b-vllm-awq": gib(20)}), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-vllm-awq")) + Expect(resolved.URL).To(Equal("file://vllm.yaml")) + }) + + It("rejects a variant the probe shows to be too large", func() { + // Same variant, same host, only the probed size differs, so nothing + // but the probe can account for the different answer. + resolved, variant, err := gallery.ResolveVariant(models, base, + probing(gib(24), map[string]uint64{"qwen3-8b-vllm-awq": gib(80)}), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("completes the install on the entry's own payload when no probe answers", func() { + // A failed probe reports 0. That is an unknown, so nothing is filtered + // out and above all the resolution does not fail: a network hiccup + // must never be able to break an install. It must not be able to + // change what gets installed either, so an unmeasurable variant does + // not displace the payload the entry itself ships. + resolved, variant, err := gallery.ResolveVariant(models, base, + probing(gib(24), map[string]uint64{}), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("still installs the base when every probe fails and nothing else survives", func() { + // The one variant this host could otherwise take is ruled out by its + // backend and no probe answers, so selection has to terminate on the + // base rather than on an error. + e := probing(gib(24), map[string]uint64{}) + e.BackendCompatible = func(backend string) bool { return backend != "vllm" } + + resolved, variant, err := gallery.ResolveVariant(models, base, e, "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + }) + + It("presents the entry's metadata, not the variant's", func() { + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(24)), "") + Expect(err).ToNot(HaveOccurred()) + Expect(resolved.Description).To(Equal("Qwen3 8B Q4")) + Expect(resolved.Icon).To(Equal("qwen.png")) + Expect(resolved.Tags).To(ConsistOf("llm")) + }) + + It("honors a pin naming the entry itself", func() { + // The entry is the last resort of its own variant list, so its own name + // has to be a usable pin: it is how an operator declines an upgrade + // their hardware would otherwise take. + resolved, variant, err := gallery.ResolveVariant(models, base, env(gib(24)), "qwen3-8b-gguf-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.Name).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("honors a pin the hardware does not satisfy", func() { + resolved, variant, err := gallery.ResolveVariant(models, base, env(gib(2)), "qwen3-8b-vllm-awq") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-vllm-awq")) + Expect(resolved.URL).To(Equal("file://vllm.yaml")) + }) + + It("detaches the resolved entry from the gallery's own maps and slices", func() { + models[0].Overrides = map[string]any{"f16": true} + models[0].AdditionalFiles = []gallery.File{{Filename: "a.bin"}} + + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(24)), "") + Expect(err).ToNot(HaveOccurred()) + + // The install path merges the caller's request overrides into this map in + // place, so aliasing the gallery's map would write one caller's request + // into the shared catalog and leak it into every later install. + resolved.Overrides["f16"] = false + resolved.Overrides["threads"] = 4 + Expect(models[0].Overrides).To(HaveKeyWithValue("f16", true)) + Expect(models[0].Overrides).ToNot(HaveKey("threads")) + + resolved.AdditionalFiles[0].Filename = "mutated.bin" + Expect(models[0].AdditionalFiles[0].Filename).To(Equal("a.bin")) + + resolved.Tags = append(resolved.Tags, "extra") + Expect(base.Tags).To(ConsistOf("llm")) + }) + + It("detaches the resolved entry even when it resolves to the entry itself", func() { + // Resolving to the base returns a copy of the very entry the gallery + // holds, which is the case most likely to alias it. + base.Overrides = map[string]any{"parameters": map[string]any{"model": "q4.gguf"}} + + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(8)), "") + Expect(err).ToNot(HaveOccurred()) + + resolved.Overrides["parameters"].(map[string]any)["model"] = "callers-choice.gguf" + Expect(base.Overrides["parameters"]).To(HaveKeyWithValue("model", "q4.gguf")) + }) + + It("detaches nested override maps from the gallery's own entry", func() { + models[0].Overrides = map[string]any{ + "parameters": map[string]any{"model": "real-variant.gguf"}, + "stopwords": []any{""}, + } + + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(24)), "") + Expect(err).ToNot(HaveOccurred()) + + // Cloning only the top level would leave this inner map shared with the + // gallery entry, so writing through the resolved copy would rewrite the + // catalog's own payload. + resolved.Overrides["parameters"].(map[string]any)["model"] = "callers-choice.gguf" + resolved.Overrides["stopwords"].([]any)[0] = "<|im_end|>" + + Expect(models[0].Overrides["parameters"]).To(HaveKeyWithValue("model", "real-variant.gguf")) + Expect(models[0].Overrides["stopwords"]).To(Equal([]any{""})) + }) + + It("does not write the caller's overrides back into the gallery entry", func() { + models[0].Overrides = map[string]any{"parameters": map[string]any{"model": "real-variant.gguf"}} + + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(24)), "") + Expect(err).ToNot(HaveOccurred()) + + // This is exactly what the install path does with the caller's request + // overrides, and mergo recurses into nested maps and overwrites them in + // place. Asserting against the in-memory catalog is the only way to + // observe the leak: re-reading the gallery from disk re-unmarshals fresh + // maps and would pass whether or not the resolved entry was detached. + requestOverrides := map[string]any{"parameters": map[string]any{"model": "callers-choice.gguf"}} + Expect(mergo.Merge(&resolved.Overrides, requestOverrides, mergo.WithOverride)).To(Succeed()) + + Expect(resolved.Overrides["parameters"]).To(HaveKeyWithValue("model", "callers-choice.gguf")) + Expect(models[0].Overrides["parameters"]).To(HaveKeyWithValue("model", "real-variant.gguf")) + }) + + It("errors when a variant references a missing entry", func() { + base.Variants = []gallery.Variant{{Model: "does-not-exist"}} + _, _, err := gallery.ResolveVariant(models, base, env(gib(8)), "") + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("does-not-exist")) + }) + + It("refuses a variant that declares variants of its own", func() { + nested := newModel("nested", "file://nested.yaml", "", "") + nested.Variants = []gallery.Variant{{Model: "qwen3-8b-vllm-awq"}} + models = append(models, nested) + base.Variants = []gallery.Variant{{Model: "nested"}} + _, _, err := gallery.ResolveVariant(models, base, env(gib(8)), "") + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("nested")) + }) + + It("surfaces a bad pin", func() { + _, _, err := gallery.ResolveVariant(models, base, env(gib(24)), "nope") + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + }) +}) + +var _ = Describe("InstallModelFromGallery with variant entries", func() { + var tempdir string + var galleries []config.Gallery + var systemState *system.SystemState + // galleryRevision keeps every gallery this suite writes distinct from every + // other, so the process-wide gallery cache cannot serve one spec's catalog + // to another. + galleryRevision := 0 + + // Every entry is described with an inline config_file rather than a URL so + // the whole install runs off the local filesystem with no network access. + // + // Each call writes a fresh path and a fresh gallery name because the gallery + // listing is cached on the name and URL pair. A spec that edits its gallery + // mid-flight would otherwise keep reading the version it started with, and + // would silently prove nothing. + newGallery := func(entries ...gallery.GalleryModel) { + out, err := yaml.Marshal(entries) + Expect(err).ToNot(HaveOccurred()) + name := fmt.Sprintf("test-%d", galleryRevision) + galleryRevision++ + galleryPath := filepath.Join(tempdir, name+".yaml") + Expect(os.WriteFile(galleryPath, out, 0600)).To(Succeed()) + + galleries = []config.Gallery{{Name: name, URL: "file://" + galleryPath}} + } + + entry := func(name, backend string) gallery.GalleryModel { + m := gallery.GalleryModel{ConfigFile: map[string]any{"backend": backend}} + m.Name = name + m.Description = "entry " + name + return m + } + + // sizedEntry declares its footprint through the entry's own size:, which is + // the only way an author influences a variant's measured size. The estimator + // prefers a declared size over its own guesswork and parses it locally, so + // these specs pin an exact figure without touching the network. + sizedEntry := func(name, backend, size string) gallery.GalleryModel { + m := entry(name, backend) + m.Size = size + return m + } + + // urlEntry describes an entry through a url rather than an inline + // config_file. The distinction matters for the recorded name: the url branch + // reads a name out of the fetched config, and that name is the referenced + // entry's own, so only the entry-name overlay can keep the record under the + // name the user asked for. The inline config_file branch seeds the name from + // the already-renamed resolved entry and so cannot observe the overlay. + urlEntry := func(name, backend, size string) gallery.GalleryModel { + payload := gallery.ModelConfig{ + Name: name, + Description: "entry " + name, + ConfigFile: "backend: " + backend + "\n", + } + out, err := yaml.Marshal(payload) + Expect(err).ToNot(HaveOccurred()) + payloadPath := filepath.Join(tempdir, "payload-"+name+".yaml") + Expect(os.WriteFile(payloadPath, out, 0600)).To(Succeed()) + + m := gallery.GalleryModel{} + m.Name = name + m.Description = "entry " + name + m.URL = "file://" + payloadPath + m.Size = size + return m + } + + // withVariants attaches alternative builds to an otherwise ordinary entry. + // Where a spec needs a definite size it declares it on the referenced entry + // with sizedEntry, in absolute terms rather than relative to the host, so + // these specs assert on selection rather than on whatever memory the machine + // running them happens to have. + withVariants := func(m gallery.GalleryModel, variants ...gallery.Variant) gallery.GalleryModel { + m.Variants = variants + return m + } + + install := func(name string, req gallery.GalleryModel, options ...gallery.InstallOption) error { + return gallery.InstallModelFromGallery( + context.TODO(), galleries, []config.Gallery{}, systemState, nil, + name, req, func(string, string, string, float64) {}, false, false, false, options...) + } + + installedBackend := func(name string) string { + dat, err := os.ReadFile(filepath.Join(tempdir, name+".yaml")) + Expect(err).ToNot(HaveOccurred()) + content := map[string]any{} + Expect(yaml.Unmarshal(dat, &content)).To(Succeed()) + return content["backend"].(string) + } + + BeforeEach(func() { + var err error + tempdir, err = os.MkdirTemp("", "variant-install") + Expect(err).ToNot(HaveOccurred()) + DeferCleanup(func() { Expect(os.RemoveAll(tempdir)).To(Succeed()) }) + + systemState, err = system.GetSystemState(system.WithModelPath(tempdir)) + Expect(err).ToNot(HaveOccurred()) + }) + + It("installs the entry's own payload when no variant fits the host", func() { + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "10000GiB"), + ) + + // No machine clears 10000GiB, so this asserts the base is the last + // resort and that missing every variant is not an error. + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("base-backend")) + }) + + It("installs a fitting variant's payload under the entry's own name", func() { + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("upgrade-backend")) + _, err := os.Stat(filepath.Join(tempdir, "qwen3-8b-q8.yaml")) + Expect(os.IsNotExist(err)).To(BeTrue(), "the variant must not be installed under its own name") + }) + + It("completes the install when a variant's size cannot be determined", func() { + // This runs the REAL probe against an entry that declares no size and no + // weight files, which is exactly the shape a probe cannot answer for. + // It must yield an unknown, not an error: the variant survives the + // filter and the install completes, on the variant or on the base, but + // never on a failure. Nothing here touches the network. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + entry("qwen3-8b-q8", "upgrade-backend"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(BeElementOf("base-backend", "upgrade-backend")) + }) + + It("installs the largest fitting variant, not the first authored", func() { + // Authored smallest-first with all three within reach, so first-match + // would take the small one. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-small"}, + gallery.Variant{Model: "qwen3-8b-large"}, + ), + sizedEntry("qwen3-8b-small", "small-backend", "16MiB"), + sizedEntry("qwen3-8b-large", "large-backend", "256MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("large-backend")) + }) + + It("never installs a variant whose backend cannot run on this host", func() { + if runtime.GOOS == "darwin" { + Skip("mlx is a compatible backend on darwin, so it proves nothing here") + } + // The mlx entry is the only variant and fits trivially, so the real + // SystemState.IsBackendCompatible rejecting mlx on a non-darwin host is + // the only thing that can send this to the base. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-mlx"}), + entry("qwen3-8b-mlx", "mlx"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("base-backend")) + }) + + It("round-trips the resolution record to disk under the entry's name", func() { + // The variant is described by url so its payload carries its own name. + // Without the entry-name overlay the record persists as "qwen3-8b-q8", + // and the stable name the entry exists to provide is lost the moment + // anything reads the record back. + // The declared sizes make the upgrade the largest build that fits, so + // resolution lands on the variant and the record it writes is the one + // under test here. + newGallery( + withVariants(urlEntry("qwen3-8b-q4", "base-backend", "16MiB"), + gallery.Variant{Model: "qwen3-8b-q8"}), + urlEntry("qwen3-8b-q8", "upgrade-backend", "256MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + + record, err := gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.EntryName).To(Equal("qwen3-8b-q4")) + Expect(record.ResolvedVariant).To(Equal("qwen3-8b-q8")) + Expect(record.PinnedVariant).To(BeEmpty()) + Expect(record.Name).To(Equal("qwen3-8b-q4")) + Expect(record.Description).To(Equal("entry qwen3-8b-q4")) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("upgrade-backend")) + }) + + It("refuses a variant name the entry does not declare, naming what was asked for", func() { + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + entry("qwen3-8b-q8", "upgrade-backend"), + ) + + err := install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q6")) + + // Auto-selecting instead would report success while silently + // installing something other than what the caller asked for, and a + // typo'd variant name would be indistinguishable from a working one. + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("qwen3-8b-q6")) + Expect(filepath.Join(tempdir, "qwen3-8b-q4.yaml")).ToNot(BeAnExistingFile()) + }) + + It("refuses a variant name when the entry declares no variants at all", func() { + // The entry is installable and selection never runs for it, so without + // an explicit refusal the requested variant would be dropped on the + // floor and the install would look like it honored the choice. + newGallery(entry("qwen3-8b-q4", "base-backend")) + + err := install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q8")) + + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("qwen3-8b-q8")) + Expect(filepath.Join(tempdir, "qwen3-8b-q4.yaml")).ToNot(BeAnExistingFile()) + }) + + It("records a pin and honors it on a plain reinstall", func() { + // The upgrade declares a size and the base does not, so auto-selection + // genuinely prefers the upgrade here. Without that the second install + // below would land on the base whether or not the pin was recalled, and + // the spec would prove nothing. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q4"))).To(Succeed()) + + record, err := gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(Equal("qwen3-8b-q4")) + Expect(record.ResolvedVariant).To(Equal("qwen3-8b-q4")) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("base-backend")) + + // No WithVariant this time: hardware selection would take the upgrade, + // so only the recalled pin can keep this on the base payload. + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("base-backend")) + + record, err = gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(Equal("qwen3-8b-q4")) + }) + + It("honors a pin recorded under a custom install name", func() { + // Sized as above so auto-selection would take the upgrade, which is what + // makes the recall observable in the second install. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + + req := gallery.GalleryModel{} + req.Name = "prod-llm" + + Expect(install("qwen3-8b-q4", req, gallery.WithVariant("qwen3-8b-q4"))).To(Succeed()) + + // The pin is written under the installed name, so the recall must read it + // back under that name too and not under the gallery entry's name. + record, err := gallery.GetLocalModelConfiguration(tempdir, "prod-llm") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(Equal("qwen3-8b-q4")) + Expect(record.EntryName).To(Equal("qwen3-8b-q4")) + Expect(installedBackend("prod-llm")).To(Equal("base-backend")) + + Expect(install("qwen3-8b-q4", req)).To(Succeed()) + Expect(installedBackend("prod-llm")).To(Equal("base-backend")) + }) + + It("re-selects automatically when a recorded pin no longer exists in the gallery", func() { + // A pin is recorded, then the gallery is edited to rename the build it + // named. Without dropping the stale pin every later reinstall and every + // upgrade of this model fails forever, and the only remedy is deleting a + // dotfile the user has never heard of. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + Expect(install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q8"))).To(Succeed()) + + record, err := gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(Equal("qwen3-8b-q8")) + + // The gallery edit: the same build under a new name. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8-v2"}), + sizedEntry("qwen3-8b-q8-v2", "renamed-backend", "16MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + // Auto-selection ran: the renamed build is the largest that fits. + Expect(installedBackend("qwen3-8b-q4")).To(Equal("renamed-backend")) + + // The stale pin is cleared rather than carried forward, so the next + // install does not have to rediscover that it is unusable. + record, err = gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(BeEmpty()) + Expect(record.ResolvedVariant).To(Equal("qwen3-8b-q8-v2")) + }) + + It("still refuses a variant the caller names on this request, recorded pin or not", func() { + // The caller-supplied pin and the recalled one must fail differently. + // This one is a request that cannot be honored, so reporting success + // for it would make a typo indistinguishable from a working choice. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + Expect(install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q8"))).To(Succeed()) + + err := install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q6")) + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("qwen3-8b-q6")) + }) + + It("writes each declared url only once into the persisted gallery file", func() { + base := withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}) + base.URLs = []string{"https://example.invalid/qwen3"} + newGallery(base, entry("qwen3-8b-q8", "upgrade-backend")) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + + record, err := gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.URLs).To(ConsistOf("https://example.invalid/qwen3")) + }) +}) + +var _ = Describe("legacy client compatibility", func() { + It("keeps every entry that declares variants installable by clients that ignore them", func() { + data, err := os.ReadFile(filepath.Join("..", "..", "gallery", "index.yaml")) + Expect(err).ToNot(HaveOccurred()) + + // Parse exactly as an older LocalAI release would: non-strictly, with + // no knowledge of the variants key. Such a client installs whatever + // payload the entry carries directly, so the entry must carry one. + var legacy []struct { + Name string `yaml:"name"` + URL string `yaml:"url"` + ConfigFile map[string]any `yaml:"config_file"` + Files []gallery.File `yaml:"files"` + Overrides map[string]any `yaml:"overrides"` + } + Expect(yaml.Unmarshal(data, &legacy)).To(Succeed()) + + var current []gallery.GalleryModel + Expect(yaml.Unmarshal(data, ¤t)).To(Succeed()) + + legacyByName := map[string]int{} + for i, e := range legacy { + legacyByName[e.Name] = i + } + + withVariants := 0 + for _, e := range current { + if !e.HasVariants() { + continue + } + withVariants++ + + i, ok := legacyByName[e.Name] + Expect(ok).To(BeTrue(), "entry %q vanished under a legacy parse", e.Name) + old := legacy[i] + // An entry whose payload lived only in its variants would install to + // nothing on every released LocalAI, which is precisely what making + // the entry itself the last resort exists to prevent. + Expect(old.URL != "" || len(old.ConfigFile) > 0).To(BeTrue(), + "entry %q carries no payload of its own, so older clients would install nothing", e.Name) + } + Expect(withVariants).To(BeNumerically(">", 0), + "expected at least one entry declaring variants in the gallery index") + }) +}) diff --git a/core/gallery/variants_lint_test.go b/core/gallery/variants_lint_test.go new file mode 100644 index 000000000..c5e0bda30 --- /dev/null +++ b/core/gallery/variants_lint_test.go @@ -0,0 +1,400 @@ +package gallery_test + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "sync" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" + + "github.com/mudler/LocalAI/core/gallery" +) + +// variantViolation is one invariant breach found by a lint helper. Helpers +// return every breach they find instead of stopping at the first, so a single +// run names them all rather than forcing a fix-one-rerun-repeat cycle. +type variantViolation struct { + Entry string + Variant string + Detail string +} + +func (v variantViolation) String() string { + if v.Variant == "" { + return fmt.Sprintf("%s: %s", v.Entry, v.Detail) + } + return fmt.Sprintf("%s -> variant %q: %s", v.Entry, v.Variant, v.Detail) +} + +func formatViolations(violations []variantViolation) string { + lines := make([]string, 0, len(violations)) + for _, v := range violations { + lines = append(lines, " "+v.String()) + } + return "\n" + strings.Join(lines, "\n") +} + +func indexEntriesByName(entries []gallery.GalleryModel) map[string]gallery.GalleryModel { + byName := make(map[string]gallery.GalleryModel, len(entries)) + for _, e := range entries { + byName[e.Name] = e + } + return byName +} + +// checkVariantReferences verifies every variant names an existing entry that +// declares no variants of its own. Selection is a single pass, so a nested +// reference would silently ignore the inner list. +// +// This is the one structural rule left. The rules about authored order, the +// hardware vocabulary and floor relationships stopped describing real hazards +// once selection became a filter plus a ranking: an author writes names, and +// there is no longer anything about hardware they can get wrong. +func checkVariantReferences(entries []gallery.GalleryModel) []variantViolation { + byName := indexEntriesByName(entries) + var violations []variantViolation + for _, e := range entries { + if !e.HasVariants() { + continue + } + for _, v := range e.Variants { + target, ok := byName[v.Model] + if !ok { + violations = append(violations, variantViolation{Entry: e.Name, Variant: v.Model, Detail: "references unknown model"}) + continue + } + if target.HasVariants() { + violations = append(violations, variantViolation{Entry: e.Name, Variant: v.Model, Detail: "references an entry that declares variants of its own; nesting is not allowed"}) + } + } + } + return violations +} + +// checkEntriesInstallSomething verifies every entry would actually put +// something on disk. +// +// It replaces an earlier rule that demanded a url: or a config_file: from every +// variant target. That demand no longer holds: applyModel now treats an entry +// declaring neither as an empty base config, which is precisely what the many +// entries pointing at gallery/virtual.yaml were already getting, minus the +// fetch. Requiring one of the two fields would now reject perfectly good +// authoring. +// +// What survives is the weaker invariant the relaxation left exposed. An entry +// with no base config, no overrides: and no files: names nothing to download +// and nothing to configure, so installing it yields an empty model directory. +// That is an authoring mistake, and it is worth catching in the catalog rather +// than on someone's machine. +// +// The rule covers every entry, not only variant targets, because the hazard has +// nothing to do with variants: it is a half-written stanza, and a parent entry +// can be one just as easily as a target. +// entryInstallsSomething restates applyModel's acceptance rule in the terms an +// author reads: a base config, or a payload to lay over an empty one. +func entryInstallsSomething(e gallery.GalleryModel) bool { + return len(e.URL) > 0 || len(e.ConfigFile) > 0 || len(e.Overrides) > 0 || len(e.AdditionalFiles) > 0 +} + +func checkEntriesInstallSomething(entries []gallery.GalleryModel) []variantViolation { + var violations []variantViolation + for _, e := range entries { + if entryInstallsSomething(e) { + continue + } + violations = append(violations, variantViolation{ + Entry: e.Name, + Detail: fmt.Sprintf("entry %q installs nothing: it declares no url:, no config_file:, no overrides: and no files:, "+ + "so installing it would leave an empty model directory. Give it the payload it is missing. "+ + "Note that urls: (plural) is the informational link list and is not a payload", e.Name), + }) + } + return violations +} + +// loadGalleryIndex parses gallery/index.yaml once for the whole suite. The +// index carries well over a thousand entries, so re-parsing it per spec is +// pure overhead. +var loadGalleryIndex = sync.OnceValues(func() ([]gallery.GalleryModel, error) { + data, err := os.ReadFile(filepath.Join("..", "..", "gallery", "index.yaml")) + if err != nil { + return nil, err + } + var entries []gallery.GalleryModel + if err := yaml.Unmarshal(data, &entries); err != nil { + return nil, err + } + return entries, nil +}) + +func plainEntry(name, url string) gallery.GalleryModel { + e := gallery.GalleryModel{} + e.Name = name + e.URL = url + return e +} + +func entryWithVariants(name, url string, variants ...gallery.Variant) gallery.GalleryModel { + e := gallery.GalleryModel{Variants: variants} + e.Name = name + e.URL = url + return e +} + +// variantFixture builds an entry with variants plus the entries it references. +// Synthetic fixtures keep the invariant logic covered however few entries in +// gallery/index.yaml declare variants; without them every index-driven spec +// below is a no-op that passes while checking nothing. +func variantFixture(base gallery.GalleryModel, referenced ...gallery.GalleryModel) []gallery.GalleryModel { + return append([]gallery.GalleryModel{base}, referenced...) +} + +var _ = Describe("gallery variant lint helpers", func() { + // An entry declares variants solely by carrying the key. GalleryBackend.IsMeta() + // has deliberately different semantics (it requires an EMPTY uri), so a + // well-meaning alignment of the two would make every helper skip every + // entry and pass silently. Assert the distinction directly. + It("treats an entry with variants as such even though it has a url", func() { + Expect(entryWithVariants("base", "u://base", gallery.Variant{Model: "big"}).HasVariants()).To(BeTrue()) + Expect(plainEntry("big", "u://big").HasVariants()).To(BeFalse()) + }) + + It("passes every invariant on a valid entry", func() { + // Authoring is nothing but a list of names, so a bare list of them must + // be entirely valid. + entries := variantFixture( + entryWithVariants("base", "u://base", + gallery.Variant{Model: "big"}, + gallery.Variant{Model: "mid"}, + gallery.Variant{Model: "metal-big"}, + gallery.Variant{Model: "small"}, + ), + plainEntry("big", "u://big"), + plainEntry("mid", "u://mid"), + plainEntry("metal-big", "u://metal-big"), + plainEntry("small", "u://small"), + ) + + Expect(checkVariantReferences(entries)).To(BeEmpty()) + Expect(checkEntriesInstallSomething(entries)).To(BeEmpty()) + }) + + Describe("checkEntriesInstallSomething", func() { + // The shape the rule exists for: a stanza that got as far as a name and + // stopped. + empty := func(name string) gallery.GalleryModel { + e := gallery.GalleryModel{} + e.Name = name + // urls: is the informational HuggingFace link list. It reads like a + // payload and is not one, so a stub carrying only this must still be + // flagged. + e.URLs = []string{"https://huggingface.co/example/" + name} + return e + } + + It("flags an entry with no url, no config_file, no overrides and no files", func() { + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "dead"}), + empty("dead"), + ) + + violations := checkEntriesInstallSomething(entries) + Expect(violations).To(HaveLen(1)) + Expect(violations[0].Entry).To(Equal("dead")) + Expect(violations[0].Detail).To(ContainSubstring("installs nothing")) + }) + + It("accepts an entry carrying only overrides, which applyModel installs on an empty base", func() { + target := gallery.GalleryModel{Overrides: map[string]any{"backend": "ds4"}} + target.Name = "overrides-only" + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "overrides-only"}), + target, + ) + + Expect(checkEntriesInstallSomething(entries)).To(BeEmpty()) + }) + + It("accepts an entry carrying only files", func() { + target := gallery.GalleryModel{} + target.Name = "files-only" + target.AdditionalFiles = []gallery.File{{Filename: "weights.gguf", URI: "u://weights"}} + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "files-only"}), + target, + ) + + Expect(checkEntriesInstallSomething(entries)).To(BeEmpty()) + }) + + It("accepts an entry described by an inline config_file rather than a url", func() { + target := gallery.GalleryModel{ConfigFile: map[string]any{"backend": "llama-cpp"}} + target.Name = "inline" + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "inline"}), + target, + ) + + Expect(checkEntriesInstallSomething(entries)).To(BeEmpty()) + }) + + It("reports every breach in one pass rather than stopping at the first", func() { + entries := variantFixture( + entryWithVariants("base", "u://base", + gallery.Variant{Model: "dead-a"}, + gallery.Variant{Model: "dead-b"}, + ), + empty("dead-a"), + empty("dead-b"), + ) + + Expect(checkEntriesInstallSomething(entries)).To(HaveLen(2)) + }) + }) + + Describe("checkVariantReferences", func() { + It("flags a variant naming an entry that does not exist", func() { + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "ghost"}), + plainEntry("a", "u://a"), + ) + + violations := checkVariantReferences(entries) + Expect(violations).To(HaveLen(1)) + Expect(violations[0].Variant).To(Equal("ghost")) + Expect(violations[0].Detail).To(ContainSubstring("unknown model")) + }) + + It("flags a variant naming an entry that declares variants itself", func() { + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "nested"}), + entryWithVariants("nested", "u://nested", gallery.Variant{Model: "a"}), + plainEntry("a", "u://a"), + ) + + violations := checkVariantReferences(entries) + Expect(violations).To(HaveLen(1)) + Expect(violations[0].Variant).To(Equal("nested")) + Expect(violations[0].Detail).To(ContainSubstring("nesting is not allowed")) + }) + + It("reports every breach in one pass rather than stopping at the first", func() { + entries := variantFixture( + entryWithVariants("base", "u://base", + gallery.Variant{Model: "ghost"}, + gallery.Variant{Model: "phantom"}, + ), + ) + + Expect(checkVariantReferences(entries)).To(HaveLen(2)) + }) + }) + +}) + +var _ = Describe("gallery/index.yaml variant invariants", Ordered, func() { + var entries []gallery.GalleryModel + + BeforeAll(func() { + var err error + entries, err = loadGalleryIndex() + Expect(err).ToNot(HaveOccurred()) + // A truncated or emptied index unmarshals cleanly and would make every + // spec below vacuously pass. + Expect(entries).ToNot(BeEmpty()) + }) + + It("references only existing entries that declare no variants themselves", func() { + v := checkVariantReferences(entries) + Expect(v).To(BeEmpty(), formatViolations(v)) + }) + + // Gallery-wide, not variants-only. The rule this replaced was scoped to + // variant targets because the index carried nine unrelated entries it would + // have failed on. Those nine declared overrides and files but no url, and + // they install cleanly on an empty base now, so there is nothing left to + // exempt and the gate covers the whole catalog in one step. + It("contains no entry that would install nothing", func() { + v := checkEntriesInstallSomething(entries) + Expect(v).To(BeEmpty(), formatViolations(v)) + }) +}) + +// The lint rules above check the catalog as text. This drives the real +// resolution path for the entry a user actually clicked and failed to install, +// so the fix is proven at the layer that broke and not only at the layer that +// should have caught it. +// +// It is a container of its own rather than another spec beside the lint rules +// because an Ordered container stops at its first failure: sharing one would +// let a lint breach skip these silently, which is precisely the kind of +// vacuously-green spec this file exists to avoid. +var _ = Describe("gallery/index.yaml deepseek-v4-flash resolution", Ordered, func() { + var entries []gallery.GalleryModel + var models []*gallery.GalleryModel + var entry *gallery.GalleryModel + + BeforeAll(func() { + var err error + entries, err = loadGalleryIndex() + Expect(err).ToNot(HaveOccurred()) + Expect(entries).ToNot(BeEmpty()) + }) + + BeforeEach(func() { + models = make([]*gallery.GalleryModel, 0, len(entries)) + for i := range entries { + models = append(models, &entries[i]) + } + entry = gallery.FindGalleryElement(models, "deepseek-v4-flash") + Expect(entry).ToNot(BeNil()) + Expect(entry.HasVariants()).To(BeTrue()) + }) + + // The unpinned pass covers the entry the user clicks. It does NOT prove the + // variants are installable: with no probe wired every size is unknown and + // the ranking ties, so the base wins and this only ever exercises the + // parent's own url. The pin spec below is what covers the four targets. + // + // Note that a base selection reports Variant.Model as the ENTRY's name + // rather than as empty, so asserting a non-empty Model here would look like + // a check that a declared variant won while passing on the base every time. + It("yields an installable entry for the entry itself", func() { + env := gallery.ResolveEnv{ + AvailableMemory: 512 << 30, + BackendCompatible: func(string) bool { return true }, + } + + resolved, selected, err := gallery.ResolveVariant(models, entry, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(entryInstallsSomething(*resolved)).To(BeTrue(), + "resolved entry %q (variant %q) carries no payload, so InstallModelFromGallery would refuse it", + resolved.Name, selected.Model) + }) + + // Pinning reaches each target directly, which is what actually proves all + // four are installable: every one of them is resolved, not just whichever + // the ranking happens to prefer. + // + // None of the four declares a url: any more. They describe themselves + // entirely through overrides: and files:, which applyModel lays over an + // empty base, so this also pins that dropping the urls left them + // installable. + It("yields an installable entry for every declared variant pin", func() { + env := gallery.ResolveEnv{ + AvailableMemory: 512 << 30, + BackendCompatible: func(string) bool { return true }, + } + + for _, v := range entry.Variants { + resolved, _, err := gallery.ResolveVariant(models, entry, env, v.Model) + Expect(err).ToNot(HaveOccurred(), "pinning %q", v.Model) + Expect(resolved.URL).To(BeEmpty(), "variant %q should reach the empty-base path, not a fetch", v.Model) + Expect(entryInstallsSomething(*resolved)).To(BeTrue(), "variant %q carries no payload", v.Model) + } + }) +}) diff --git a/core/http/endpoints/localai/gallery.go b/core/http/endpoints/localai/gallery.go index fabae336d..fb0a77bb4 100644 --- a/core/http/endpoints/localai/gallery.go +++ b/core/http/endpoints/localai/gallery.go @@ -25,6 +25,11 @@ type ModelGalleryEndpointService struct { type GalleryModel struct { ID string `json:"id"` + // Variant installs one specific build of an entry that declares variants, + // named as it appears in the entry's `variants` list (see the `variants` + // and `auto_variant` fields of the gallery listing). Leave it empty to let + // LocalAI auto-select the largest build this host can actually run. + Variant string `json:"variant,omitempty"` gallery.GalleryModel } @@ -86,6 +91,7 @@ func (mgs *ModelGalleryEndpointService) ApplyModelGalleryEndpoint() echo.Handler Req: input.GalleryModel, ID: uuid.String(), GalleryElementName: input.ID, + Variant: input.Variant, Galleries: mgs.galleries, BackendGalleries: mgs.backendGalleries, } diff --git a/core/http/react-ui/e2e/backends-management.spec.js b/core/http/react-ui/e2e/backends-management.spec.js index c34d50533..11b179189 100644 --- a/core/http/react-ui/e2e/backends-management.spec.js +++ b/core/http/react-ui/e2e/backends-management.spec.js @@ -26,3 +26,54 @@ test.describe('Backends management page', () => { await expect(page.getByPlaceholder('oci://quay.io/example/backend:latest')).toBeVisible() }) }) + +// Backend gallery descriptions are Markdown too: 40 of the entries in +// backend/index.yaml carry headings, inline code, lists or links, and they used +// to be dumped raw into the truncated table cell. +const MARKDOWN_DESCRIPTION = + '# InsightFace\n\nUse `insightface` for face analysis. See [the docs](https://example.com/docs) for **details**.' +const STRIPPED_DESCRIPTION = + 'InsightFace Use insightface for face analysis. See the docs for details.' + +test.describe('Backends management page - Markdown descriptions', () => { + test.beforeEach(async ({ page }) => { + await page.route('**/api/backends*', (route) => { + route.fulfill({ + contentType: 'application/json', + body: JSON.stringify({ + backends: [ + { name: 'markdown-backend', description: MARKDOWN_DESCRIPTION, installed: false }, + { name: 'plain-backend', description: '', installed: false }, + ], + }), + }) + }) + await page.goto('/app/backends') + await expect(page.locator('th', { hasText: 'Description' })).toBeVisible({ timeout: 10_000 }) + }) + + test('table cell shows the description as clean text, not raw Markdown', async ({ page }) => { + const cell = page.locator('tr', { hasText: 'markdown-backend' }).locator('span[title]', { hasText: 'InsightFace' }) + + await expect(cell).toHaveText(STRIPPED_DESCRIPTION) + // The syntax itself must be gone, not merely rendered somewhere. + await expect(cell).not.toContainText('#') + await expect(cell).not.toContainText('`') + await expect(cell).not.toContainText('**') + await expect(cell).not.toContainText('https://example.com/docs') + // A block element here would blow up the row height. + await expect(cell.locator('h1')).toHaveCount(0) + }) + + test('title tooltip carries the stripped text, not raw Markdown', async ({ page }) => { + const cell = page.locator('tr', { hasText: 'markdown-backend' }).locator('span[title]', { hasText: 'InsightFace' }) + + await expect(cell).toHaveAttribute('title', STRIPPED_DESCRIPTION) + }) + + test('a backend with no description still shows the placeholder', async ({ page }) => { + const row = page.locator('tr', { hasText: 'plain-backend' }) + + await expect(row.locator('span[title=""]')).toHaveText('-') + }) +}) diff --git a/core/http/react-ui/e2e/models-gallery.spec.js b/core/http/react-ui/e2e/models-gallery.spec.js index d45b5eeda..6ec1045be 100644 --- a/core/http/react-ui/e2e/models-gallery.spec.js +++ b/core/http/react-ui/e2e/models-gallery.spec.js @@ -8,6 +8,10 @@ const MOCK_MODELS_RESPONSE = { backend: "llama-cpp", installed: false, tags: ["chat"], + // The listing carries only the declaration flag. Describing variants + // costs the server a network probe each, so the description lives + // behind /api/models/variants/:id and is fetched on demand. + has_variants: true, }, { name: "whisper-model", @@ -173,7 +177,9 @@ test.describe("Models Gallery - Backend Features", () => { // The detail view should show Backend label and value const detail = page.locator('td[colspan="8"]'); await expect(detail.locator("text=Backend")).toBeVisible(); - await expect(detail.locator("text=llama-cpp")).toBeVisible(); + // The Backend DetailRow renders before the Variants section, which lists a + // per-variant backend badge of its own, so scope to the first match. + await expect(detail.locator("text=llama-cpp").first()).toBeVisible(); }); }); @@ -433,3 +439,1285 @@ test.describe("Models Gallery - Empty State", () => { await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); }); }); + +// The variant description the companion endpoint returns for llama-model. +// memory_bytes is omitempty server-side, so the mlx variant deliberately +// carries no key at all: the UI must render that as unknown, never 0 B. +// +// quantization and features are omitempty for the same reason. The mlx build +// carries neither, standing in for a backend served from a directory of +// weights whose name declares no format: those cells must degrade to a stated +// "unknown", never to a blank or an "undefined". +const MOCK_VARIANTS_RESPONSE = { + variants: [ + { + model: "llama-model", + backend: "llama-cpp", + memory_bytes: 4 * 1024 * 1024 * 1024, + fits: true, + is_base: true, + quantization: "Q4_K_M", + }, + { + model: "llama-model-q8", + backend: "llama-cpp", + memory_bytes: 8 * 1024 * 1024 * 1024, + fits: true, + is_base: false, + quantization: "Q8_0", + features: ["dflash"], + }, + { + model: "llama-model-mlx", + backend: "mlx", + fits: true, + is_base: false, + }, + { + model: "llama-model-f16", + backend: "llama-cpp", + memory_bytes: 40 * 1024 * 1024 * 1024, + fits: false, + is_base: false, + quantization: "F16", + }, + ], + auto_selected: "llama-model-q8", +}; + +test.describe("Models Gallery - Variant picker", () => { + // installUrls records every install request so a test can assert both the + // presence and the absence of the ?variant= parameter. + let installUrls; + // variantUrls records every companion-endpoint request. It is what proves + // the description is fetched lazily and cached, rather than being paid for + // by every row on page load. + let variantUrls; + // Held requests let a test observe the in-flight state rather than racing it. + let releaseVariants; + + test.beforeEach(async ({ page }) => { + installUrls = []; + variantUrls = []; + releaseVariants = null; + await page.route("**/api/models*", (route) => { + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(MOCK_MODELS_RESPONSE), + }); + }); + await page.route("**/api/models/install/**", (route) => { + installUrls.push(route.request().url()); + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ jobID: "variant-install" }), + }); + }); + await page.route("**/api/models/variants/**", async (route) => { + variantUrls.push(route.request().url()); + if (releaseVariants) await releaseVariants; + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify(MOCK_VARIANTS_RESPONSE), + }); + }); + await page.goto("/app/models"); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + }); + + const variantRow = (page) => page.locator("tr", { hasText: "llama-model" }).first(); + const plainRow = (page) => + page.locator("tr", { hasText: "stablediffusion-model" }).first(); + const openMenu = (page) => + variantRow(page).getByRole("button", { name: "Choose a variant" }).click(); + + test("the listing alone fetches no variant descriptions", async ({ page }) => { + // The whole point of the companion endpoint: a page load costs zero + // probes no matter how many entries declare variants. + await expect(page.locator("tbody tr").first()).toBeVisible(); + expect(variantUrls).toHaveLength(0); + }); + + test("an entry that declares variants shows the split-button chevron", async ({ + page, + }) => { + await expect( + variantRow(page).getByRole("button", { name: "Choose a variant" }), + ).toBeVisible(); + }); + + test("an entry without variants renders no chevron", async ({ page }) => { + await expect( + plainRow(page).getByRole("button", { name: "Choose a variant" }), + ).toHaveCount(0); + // and still offers an ordinary install + await expect( + plainRow(page).locator("button.btn-primary"), + ).toHaveCount(1); + }); + + test("an entry without variants fetches nothing even when expanded", async ({ + page, + }) => { + await plainRow(page).click(); + await expect(page.locator('td[colspan="8"]')).toBeVisible(); + expect(variantUrls).toHaveLength(0); + }); + + test("plain Install sends no variant parameter", async ({ page }) => { + await plainRow(page).locator("button.btn-primary").click(); + await expect.poll(() => installUrls.length).toBe(1); + expect(installUrls[0]).not.toContain("variant="); + }); + + test("opening the menu fetches the description once and caches it", async ({ + page, + }) => { + await openMenu(page); + await expect(page.locator(".action-menu")).toBeVisible(); + await expect.poll(() => variantUrls.length).toBe(1); + expect(variantUrls[0]).toContain("/api/models/variants/llama-model"); + + // Close and reopen: the cached answer must be reused. + await page.keyboard.press("Escape"); + await openMenu(page); + await expect( + page.locator(".action-menu__item", { hasText: "llama-model-q8" }), + ).toBeVisible(); + expect(variantUrls).toHaveLength(1); + }); + + test("the menu shows a loading state while the description is in flight", async ({ + page, + }) => { + let unblock; + releaseVariants = new Promise((resolve) => { + unblock = resolve; + }); + await openMenu(page); + await expect(page.locator(".action-menu")).toContainText("Loading variants"); + unblock(); + await expect( + page.locator(".action-menu__item", { hasText: "llama-model-q8" }), + ).toBeVisible(); + await expect(page.locator(".action-menu")).not.toContainText( + "Loading variants", + ); + }); + + test("the auto-selected variant is marked in the menu", async ({ page }) => { + await openMenu(page); + const menu = page.locator(".action-menu"); + await expect(menu).toBeVisible(); + const autoItem = menu.locator(".action-menu__item", { + hasText: "llama-model-q8", + }); + await expect(autoItem.locator(".badge", { hasText: "Auto" })).toBeVisible(); + // the base build is identifiable too + await expect( + menu + .locator(".action-menu__item", { hasText: "llama-model" }) + .first() + .locator(".badge", { hasText: "Base build" }), + ).toBeVisible(); + }); + + test("a variant with no memory_bytes renders as unknown, not 0", async ({ + page, + }) => { + await openMenu(page); + const mlxItem = page.locator(".action-menu__item", { + hasText: "llama-model-mlx", + }); + await expect(mlxItem).toContainText("Unknown size"); + await expect(mlxItem).not.toContainText("0 B"); + }); + + test("a variant that does not fit is still selectable", async ({ page }) => { + await openMenu(page); + const f16 = page.locator(".action-menu__item", { + hasText: "llama-model-f16", + }); + await expect(f16.locator(".badge", { hasText: "Does not fit" })).toBeVisible(); + await expect(f16).toBeEnabled(); + }); + + test("choosing a specific variant sends ?variant= on the install", async ({ + page, + }) => { + await openMenu(page); + await page + .locator(".action-menu__item", { hasText: "llama-model-mlx" }) + .click(); + await expect.poll(() => installUrls.length).toBe(1); + expect(installUrls[0]).toContain("variant=llama-model-mlx"); + }); + + test("the expanded detail row lists every variant", async ({ page }) => { + await variantRow(page).click(); + const detail = page.locator('td[colspan="8"]'); + await expect(detail).toContainText("Variants"); + await expect(detail).toContainText("llama-model-q8"); + await expect(detail).toContainText("llama-model-mlx"); + await expect(detail).toContainText("llama-model-f16"); + await expect(detail).toContainText("Unknown size"); + await expect(detail).toContainText("Auto-selected"); + await expect(detail).toContainText("Base build"); + await expect(detail).toContainText("Does not fit"); + await expect(detail).toContainText("mlx"); + // Expanding is the second trigger point, so it pays for exactly one fetch. + expect(variantUrls).toHaveLength(1); + }); + + test("the variant rows line up as columns", async ({ page }) => { + await variantRow(page).click(); + const rows = page.locator(".variant-row"); + await expect(rows).toHaveCount(4); + const columns = await rows.evaluateAll((els) => + els.map((el) => ({ + backend: el.querySelector(".variant-row__backend").getBoundingClientRect().x, + size: el.querySelector(".variant-row__size").getBoundingClientRect().right, + })), + ); + // Names differ in length, so without shared tracks each row would start + // its backend at a different x. Sub-pixel rounding is the only tolerance. + for (const c of columns) { + expect(Math.abs(c.backend - columns[0].backend)).toBeLessThan(1.5); + expect(Math.abs(c.size - columns[0].size)).toBeLessThan(1.5); + } + }); + + test("only the informative status is badged", async ({ page }) => { + await variantRow(page).click(); + const detail = page.locator('td[colspan="8"]'); + await expect(detail.locator(".variant-row")).toHaveCount(4); + // "Fits" was true of three rows out of four and said nothing; the row that + // does not fit is the one worth marking. + await expect(detail.getByText("Fits", { exact: true })).toHaveCount(0); + const unfit = detail.locator(".variant-row--unfit"); + await expect(unfit).toHaveCount(1); + await expect(unfit).toContainText("llama-model-f16"); + await expect(unfit.locator(".badge-warning")).toHaveText("Does not fit"); + // Auto-selected still answers "what do I get if I just hit Install". + await expect( + detail.locator(".variant-row", { hasText: "llama-model-q8" }), + ).toContainText("Auto-selected"); + }); + + test("clicking a variant row installs that variant", async ({ page }) => { + await variantRow(page).click(); + await page + .locator(".variant-row", { hasText: "llama-model-mlx" }) + .click(); + await expect.poll(() => installUrls.length).toBe(1); + expect(installUrls[0]).toContain("variant=llama-model-mlx"); + }); + + test("the menu names each build's quantization alongside backend and size", async ({ + page, + }) => { + // Without it the meta line reads "llama-cpp - 8 GB" for two builds that + // differ entirely in precision, which describes nothing the user is + // choosing between. + await openMenu(page); + await expect( + page.locator(".action-menu__item", { hasText: "llama-model-q8" }), + ).toContainText("llama-cpp · Q8_0 · 8 GB"); + }); + + test("the menu marks a build that serves faster", async ({ page }) => { + // A compact marker, not a sentence: the dropdown has room for the token + // and the detail row carries the spelled-out name. + await openMenu(page); + await expect( + page + .locator(".action-menu__item", { hasText: "llama-model-q8" }) + .locator(".badge", { hasText: "DFLASH" }), + ).toBeVisible(); + }); + + test("a build naming no quantization drops the segment rather than blanking", async ({ + page, + }) => { + // The degrade contract in the compact surface: no empty segment, no + // dangling separator, and above all no "undefined". + await openMenu(page); + const item = page.locator(".action-menu__item", { + hasText: "llama-model-mlx", + }); + await expect(item).toContainText("mlx · Unknown size"); + await expect(item).not.toContainText("undefined"); + await expect(item).not.toContainText("· ·"); + }); + + test("the detail row gives quantization its own column", async ({ page }) => { + await variantRow(page).click(); + const detail = page.locator(".variant-list"); + + await expect( + detail.locator(".variant-row", { hasText: "llama-model-q8" }).locator(".variant-row__quant"), + ).toHaveText("Q8_0"); + await expect( + detail.locator(".variant-row", { hasText: "llama-model-f16" }).locator(".variant-row__quant"), + ).toHaveText("F16"); + }); + + test("the detail row states an unknown quantization rather than leaving a gap", async ({ + page, + }) => { + // An empty cell in an aligned column reads as a rendering fault, so the + // absent case is spelled out and styled as the exception it is. + await variantRow(page).click(); + const cell = page + .locator(".variant-row", { hasText: "llama-model-mlx" }) + .locator(".variant-row__quant"); + + await expect(cell).toHaveText("Unknown format"); + await expect(cell).toHaveClass(/variant-row__quant--unknown/); + }); + + test("the detail row spells out the serving feature", async ({ page }) => { + // This is the room the detail row has over the dropdown: "DFLASH" names + // nothing to a user who has not met it. + await variantRow(page).click(); + + await expect( + page + .locator(".variant-row", { hasText: "llama-model-q8" }) + .locator(".badge", { hasText: "Faster: DFlash" }), + ).toBeVisible(); + // A build declaring no feature carries no feature badge at all. + await expect( + page + .locator(".variant-row", { hasText: "llama-model-mlx" }) + .locator(".badge", { hasText: "Faster" }), + ).toHaveCount(0); + }); + + test("a variant row is reachable and actionable from the keyboard", async ({ + page, + }) => { + await variantRow(page).click(); + const row = page.locator(".variant-row", { hasText: "llama-model-f16" }); + await row.focus(); + // A build that does not fit stays installable: the explicit choice is an + // override the server honours. + await expect(row).toBeFocused(); + await page.keyboard.press("Enter"); + await expect.poll(() => installUrls.length).toBe(1); + expect(installUrls[0]).toContain("variant=llama-model-f16"); + }); +}); + +// The gallery entries behind two of llama-model's variants, as the listing +// returns them when asked for one by exact name. Every field is deliberately +// unlike the parent's, so a test asserting on them proves the panel resolved +// the variant's own entry rather than re-rendering the row it sits under. +// +// llama-model-mlx is absent on purpose: it stands for a name the listing no +// longer returns, which is a real outcome once a gallery is reloaded between +// describing an entry's variants and asking about one of them. +const VARIANT_ENTRIES = { + "llama-model-q8": { + name: "llama-model-q8", + description: "The eight-bit build, kept for quality-sensitive work.", + backend: "llama-cpp", + installed: false, + license: "q8-only-licence", + tags: ["chat", "q8-only-tag"], + urls: ["https://example.invalid/llama-model-q8"], + additionalFiles: [ + { + filename: "llama-model-q8.gguf", + uri: "https://example.invalid/q8.gguf", + sha256: "q8", + }, + ], + }, + "llama-model-f16": { + name: "llama-model-f16", + description: "The full-precision build.", + backend: "llama-cpp", + installed: false, + license: "f16-only-licence", + tags: ["chat"], + urls: [], + }, +}; + +// The variant list answers "how do these differ". This answers "tell me +// everything about this one", for a build that has no listing row of its own +// while the collapse is on and so is unreachable anywhere else in the page. +test.describe("Models Gallery - Variant details", () => { + let installUrls; + // Requests for a single variant's gallery entry, told apart from the + // gallery's own listing by the page size the detail lookup asks for. + let detailUrls; + + test.beforeEach(async ({ page }) => { + installUrls = []; + detailUrls = []; + + await page.route("**/api/models*", (route) => { + const url = new URL(route.request().url()); + const term = (url.searchParams.get("term") || "").trim(); + const isDetailLookup = + url.pathname.endsWith("/api/models") && + url.searchParams.get("items") === "100" && + term !== ""; + if (isDetailLookup) { + detailUrls.push(url.toString()); + const entry = VARIANT_ENTRIES[term]; + return route.fulfill({ + contentType: "application/json", + body: JSON.stringify({ + ...MOCK_MODELS_RESPONSE, + models: entry ? [entry] : [], + }), + }); + } + return route.fulfill({ + contentType: "application/json", + body: JSON.stringify(MOCK_MODELS_RESPONSE), + }); + }); + await page.route("**/api/models/install/**", (route) => { + installUrls.push(route.request().url()); + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ jobID: "variant-install" }), + }); + }); + await page.route("**/api/models/variants/**", (route) => + route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify(MOCK_VARIANTS_RESPONSE), + }), + ); + await page.goto("/app/models"); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + // Expanding the parent is what puts the variant list on screen. + await page.locator("tr", { hasText: "llama-model" }).first().click(); + await expect(page.locator(".variant-row")).toHaveCount(4); + }); + + // Exact, because the names are prefixes of one another: "llama-model" is a + // substring of every other variant's control. + const infoFor = (page, variant) => + page.getByRole("button", { + name: `Show full details for ${variant}`, + exact: true, + }); + + test("every variant carries its own details control", async ({ page }) => { + await expect(page.locator(".variant-row__info")).toHaveCount(4); + // The accessible name has to say which build it acts on: a column of + // identical "info" buttons is useless to a screen reader user. + for (const variant of [ + "llama-model", + "llama-model-q8", + "llama-model-mlx", + "llama-model-f16", + ]) { + await expect(infoFor(page, variant)).toBeVisible(); + } + }); + + test("no details are fetched until the control is used", async ({ page }) => { + // Expanding lists four variants. If the panel prefetched, this would be + // four requests for data nobody has asked to see. + expect(detailUrls).toHaveLength(0); + await infoFor(page, "llama-model-q8").click(); + await expect.poll(() => detailUrls.length).toBe(1); + expect(detailUrls[0]).toContain("term=llama-model-q8"); + }); + + test("the details shown are the variant's own entry, not the parent's", async ({ + page, + }) => { + await infoFor(page, "llama-model-q8").click(); + const panel = page.locator(".variant-detail"); + await expect(panel).toContainText( + "The eight-bit build, kept for quality-sensitive work.", + ); + await expect(panel).toContainText("q8-only-licence"); + await expect(panel).toContainText("q8-only-tag"); + await expect(panel).toContainText("https://example.invalid/llama-model-q8"); + await expect(panel).toContainText("1 file"); + // The parent's own description renders in the same expanded row. If the + // panel were re-rendering the parent, this would be here too. + await expect(panel).not.toContainText("A llama model"); + }); + + test("a variant's details never nest another variants list", async ({ + page, + }) => { + // Two levels of disclosure is already deep. A picker inside a picker is + // where it stops being legible. + await infoFor(page, "llama-model-q8").click(); + await expect(page.locator(".variant-detail")).toBeVisible(); + await expect(page.locator(".variant-detail .variant-list")).toHaveCount(0); + }); + + test("opening the details does not install anything", async ({ page }) => { + await infoFor(page, "llama-model-q8").click(); + await expect(page.locator(".variant-detail")).toContainText( + "q8-only-licence", + ); + // The panel is fully rendered by now, so an install triggered by the same + // click would have fired. + expect(installUrls).toHaveLength(0); + }); + + test("the variant row still installs with the control alongside it", async ({ + page, + }) => { + await page.locator(".variant-row", { hasText: "llama-model-q8" }).click(); + await expect.poll(() => installUrls.length).toBe(1); + expect(installUrls[0]).toContain("variant=llama-model-q8"); + // And installing is not a request for details either. + expect(detailUrls).toHaveLength(0); + }); + + test("a variant whose entry cannot be resolved says so", async ({ page }) => { + await infoFor(page, "llama-model-mlx").click(); + const panel = page.locator(".variant-detail"); + // Visibly degraded, not silently blank: an empty panel reads as a + // rendering fault rather than as a lookup that came back with nothing. + await expect(panel).toContainText( + "Details for llama-model-mlx could not be loaded.", + ); + await expect(panel.locator(".variant-detail__state--error")).toBeVisible(); + }); + + test("the details are fetched once and reused", async ({ page }) => { + await infoFor(page, "llama-model-q8").click(); + await expect.poll(() => detailUrls.length).toBe(1); + await page + .getByRole("button", { + name: "Hide full details for llama-model-q8", + exact: true, + }) + .click(); + await expect(page.locator(".variant-detail")).toHaveCount(0); + await infoFor(page, "llama-model-q8").click(); + await expect(page.locator(".variant-detail")).toContainText( + "q8-only-licence", + ); + expect(detailUrls).toHaveLength(1); + }); + + test("only one variant's details are open at a time", async ({ page }) => { + // The list is a comparison; two open panels push the rows being compared + // apart. + await infoFor(page, "llama-model-q8").click(); + await infoFor(page, "llama-model-f16").click(); + await expect(page.locator(".variant-detail")).toHaveCount(1); + await expect(page.locator(".variant-detail")).toContainText( + "f16-only-licence", + ); + }); + + test("the control is keyboard reachable, activates, and dismisses", async ({ + page, + }) => { + const info = infoFor(page, "llama-model-q8"); + await info.focus(); + await expect(info).toBeFocused(); + // A visible focus indicator, not merely a focused element. + await expect(info).toHaveCSS("outline-style", "solid"); + await page.keyboard.press("Enter"); + await expect(page.locator(".variant-detail")).toContainText( + "q8-only-licence", + ); + await expect( + page.getByRole("button", { + name: "Hide full details for llama-model-q8", + exact: true, + }), + ).toHaveAttribute("aria-expanded", "true"); + + await page.keyboard.press("Escape"); + await expect(page.locator(".variant-detail")).toHaveCount(0); + // Focus comes back to the control that opened it, rather than being + // dropped at the top of the document. + await expect(infoFor(page, "llama-model-q8")).toBeFocused(); + }); + + test("the variant rows still line up with the control in front of them", async ({ + page, + }) => { + // The extra column must be shared like every other, or the names it sits + // beside stop forming a column. + const rows = page.locator(".variant-row"); + const columns = await rows.evaluateAll((els) => + els.map((el) => ({ + name: el.querySelector(".variant-row__name").getBoundingClientRect().x, + size: el + .querySelector(".variant-row__size") + .getBoundingClientRect().right, + })), + ); + for (const c of columns) { + expect(Math.abs(c.name - columns[0].name)).toBeLessThan(1.5); + expect(Math.abs(c.size - columns[0].size)).toBeLessThan(1.5); + } + }); +}); + +// The collapsed view is the deduplicated gallery: every entry installable in +// its own right, with nothing shown twice. Here whisper-model stands in for a +// build llama-model already offers as a variant, so it is the only row that +// drops; stablediffusion-model is nobody's variant and stays. The filter is +// server-side because the listing paginates, so these specs assert on the +// request the page actually sends, not just on the rows it renders. +const COLLAPSED_RESPONSE = { + ...MOCK_MODELS_RESPONSE, + models: MOCK_MODELS_RESPONSE.models.filter((m) => m.name !== "whisper-model"), + availableModels: 2, + totalPages: 1, + currentPage: 1, +}; + +// What a search for the grouped-away build gets back with the collapse off: +// the build itself. +const SEARCH_HIT_RESPONSE = { + ...MOCK_MODELS_RESPONSE, + models: MOCK_MODELS_RESPONSE.models.filter((m) => m.name === "whisper-model"), + availableModels: 1, + totalPages: 1, + currentPage: 1, +}; + +// And with the collapse on: the same term matched against the same builds, but +// the match reported at the entry that offers it, which is the row the user can +// act on. One row, and a count that says one. +const SEARCH_PARENT_RESPONSE = { + ...MOCK_MODELS_RESPONSE, + models: MOCK_MODELS_RESPONSE.models.filter((m) => m.name === "llama-model"), + availableModels: 1, + totalPages: 1, + currentPage: 1, +}; + +test.describe("Models Gallery - Collapsed Listing", () => { + let listingUrls; + + test.beforeEach(async ({ page }) => { + listingUrls = []; + + await page.route("**/api/models*", (route) => { + const url = new URL(route.request().url()); + // Only the gallery's own listing. Sibling routes like + // /api/models/estimate share the prefix, and the recommended-models + // panel queries /api/models itself with its own page size, so neither + // must pollute the record of what the page sent, nor pick up the + // narrowed bodies below. + const isListing = + url.pathname.endsWith("/api/models") && + url.searchParams.get("items") === "9"; + if (isListing) { + listingUrls.push(url); + } + const term = (url.searchParams.get("term") || "").trim(); + const collapsed = url.searchParams.get("collapse_variants") === "true"; + const tag = url.searchParams.get("tag"); + // Stands in for the server: the term is matched against every build + // either way, and the collapse decides how a match is reported. Grouped, + // a hit on a build a parent already offers comes back as that parent; + // ungrouped, it comes back as itself. + let body = collapsed ? COLLAPSED_RESPONSE : MOCK_MODELS_RESPONSE; + if (isListing && term === "whisper-model") + body = collapsed ? SEARCH_PARENT_RESPONSE : SEARCH_HIT_RESPONSE; + // A term matching no entry, so the empty state is reachable from a + // search as well as from a chip and the two can be told apart. + else if (isListing && term) body = EMPTY_FILTERED_RESPONSE; + // A usecase filter matches nothing in this fixture, so the empty state + // stays reachable and the specs can pin down what it says. + if (isListing && tag) body = EMPTY_FILTERED_RESPONSE; + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(body), + }); + }); + + await page.goto("/app/models"); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + }); + + // The house pattern for these toggles: the checkbox itself is a zero-sized + // opacity-0 input, so state is read through the wrapping label and changed + // by clicking the visible track. + const collapseToggle = (page) => page.getByLabel("One row per model"); + const flipCollapse = (page) => + page.getByTestId("models-collapse-variants").locator(".toggle__track").click(); + + test("the collapse toggle sits in the refinements band, on by default", async ({ + page, + }) => { + // It belongs with the other narrowing controls rather than among the + // taxonomy chips: it refines a listing the user is already reading. + await expect( + page + .getByTestId("models-filters-refine") + .getByTestId("models-collapse-variants"), + ).toBeVisible(); + await expect( + page.getByTestId("models-collapse-variants"), + ).toContainText("One row per model"); + // Default collapsed, so the default view is one row per model. + await expect(collapseToggle(page)).toBeChecked(); + }); + + test("turning the toggle off reveals the builds the collapse hid", async ({ + page, + }) => { + // Browsing, as opposed to finding. Search reaches a build whose name you + // already know; only this enumerates every build the gallery holds. + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount(0); + + await flipCollapse(page); + + await expect(page.locator("tr", { hasText: "whisper-model" })).toBeVisible(); + // Off means the parameter is absent, so opting out asks for exactly the + // listing every other API client gets. + await expect + .poll(() => + listingUrls[listingUrls.length - 1].searchParams.get( + "collapse_variants", + ), + ) + .toBeNull(); + }); + + test("changing the toggle resets to page 1", async ({ page }) => { + await flipCollapse(page); + + await expect + .poll(() => listingUrls[listingUrls.length - 1].searchParams.get("page")) + .toBe("1"); + }); + + test("the choice survives a reload", async ({ page }) => { + await flipCollapse(page); + await expect(page.locator("tr", { hasText: "whisper-model" })).toBeVisible(); + + await page.reload(); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + + await expect(collapseToggle(page)).not.toBeChecked(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toBeVisible(); + }); + + test("browsing collapses: the parent stays, the build it offers drops", async ({ + page, + }) => { + // A filter that kept only the entries declaring variants would wrongly + // drop stablediffusion-model too. + await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + await expect( + page.locator("tr", { hasText: "stablediffusion-model" }), + ).toBeVisible(); + + // Asserted over every listing request, so a first paint that fetched the + // uncollapsed listing before settling would still fail. + expect(listingUrls.length).toBeGreaterThan(0); + for (const url of listingUrls) { + expect(url.searchParams.get("collapse_variants")).toBe("true"); + } + }); + + test("searching a build the collapse groups away surfaces the entry offering it", async ({ + page, + }) => { + // Grouped, a search is still answered, and answered with a row that can be + // acted on. Typing the name of an entry the gallery does hold must never + // produce "no models found", which reads as "that model does not exist"; + // returning the build itself would instead put a row in the listing that + // the view the user asked for has no place for. + await page.locator(".search-bar input").fill("whisper-model"); + + await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + await expect(page.locator(".empty-state")).toHaveCount(0); + }); + + test("the same search returns the build itself with the toggle off", async ({ + page, + }) => { + // The other half of what the toggle now controls. Off, search answers with + // the individual build, exactly as it does for every client that never + // sends the parameter. + await flipCollapse(page); + await page.locator(".search-bar input").fill("whisper-model"); + + await expect(page.locator("tr", { hasText: "whisper-model" })).toBeVisible(); + }); + + test("the search term is sent alongside the collapse, not instead of it", async ({ + page, + }) => { + // The server decides what an active search means. The page keeps asking + // for the collapsed listing so that decision lives in one place, and so + // clearing the box goes straight back to the browsing view. + await page.locator(".search-bar input").fill("whisper-model"); + await expect.poll( + () => listingUrls[listingUrls.length - 1].searchParams.get("term"), + ).toBe("whisper-model"); + + const searched = listingUrls[listingUrls.length - 1]; + expect(searched.searchParams.get("collapse_variants")).toBe("true"); + }); + + test("clearing the search box returns to the collapsed listing", async ({ + page, + }) => { + await page.locator(".search-bar input").fill("whisper-model"); + // The search narrowed to the one surfaced row, so the other browsing rows + // are gone and their return is what proves the term was dropped. + await expect( + page.locator("tr", { hasText: "stablediffusion-model" }), + ).toHaveCount(0); + + await page.locator(".search-bar input").fill(""); + + await expect( + page.locator("tr", { hasText: "stablediffusion-model" }), + ).toBeVisible(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); + }); + + test("a legacy '0' in storage is not read as a choice", async ({ page }) => { + // An older build wrote '1'/'0' from an effect that ran on mount, so those + // values record that the page was opened rather than that anyone picked a + // view. Only 'on'/'off' counts, so a legacy visitor gets the default. + await page.evaluate(() => { + localStorage.setItem("localai-models-collapse-variants-filter", "0"); + }); + await page.reload(); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + const last = listingUrls[listingUrls.length - 1]; + expect(last.searchParams.get("collapse_variants")).toBe("true"); + }); + + test("searching never dead-ends on the default view", async ({ page }) => { + // The regression 462583f38 existed to prevent, re-checked now that search + // respects the toggle instead of switching it off. A user who never touches + // the control must still get an answer for a build the gallery holds; that + // the answer is the entry offering it is the collapse doing its job, not + // the dead end coming back. + await expect(collapseToggle(page)).toBeChecked(); + + await page.locator(".search-bar input").fill("whisper-model"); + + await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); + await expect(page.locator(".empty-state")).toHaveCount(0); + // Still asked for collapsed: the server decides what a term means. + await expect + .poll(() => + listingUrls[listingUrls.length - 1].searchParams.get( + "collapse_variants", + ), + ) + .toBe("true"); + }); + + test("the empty state does not blame the collapse for a chip", async ({ + page, + }) => { + await page.locator(".filter-btn", { hasText: "Chat" }).click(); + + await expect(page.locator(".empty-state-title")).toHaveText( + "No models found", + ); + await expect(page.locator(".empty-state-text")).toHaveText( + "No models match your current search or filters.", + ); + // The chip is applied server-side over every build the gallery holds, and + // a match there is always reported as some row, so an empty result means + // nothing matched rather than that the collapse swallowed the matches. + // Pointing at the toggle would send the user to a control that cannot + // change this result. + await expect(page.locator(".empty-state-hint")).toHaveCount(0); + }); + + test("the empty state does not blame the collapse for a search", async ({ + page, + }) => { + // Same reasoning as the chip: the term is matched against every build, so + // with one typed the collapse cannot be what emptied the listing. + await page.locator(".search-bar input").fill("nothing-matches-this"); + await expect(page.locator(".empty-state")).toBeVisible(); + + await expect(page.locator(".empty-state-hint")).toHaveCount(0); + }); + + test("clear filters returns to the collapsed browsing view", async ({ + page, + }) => { + await page.locator(".filter-btn", { hasText: "Chat" }).click(); + await expect(page.locator(".empty-state")).toBeVisible(); + + await page.getByRole("button", { name: "Clear filters" }).click(); + + await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + }); + + test("clear filters resets the collapse toggle to its default", async ({ + page, + }) => { + // It is a filter like the others, so leaving it behind would make "clear + // filters" a half-truth. + await flipCollapse(page); + await expect(page.locator("tr", { hasText: "whisper-model" })).toBeVisible(); + await page.locator(".filter-btn", { hasText: "Chat" }).click(); + await expect(page.locator(".empty-state")).toBeVisible(); + + await page.getByRole("button", { name: "Clear filters" }).click(); + + await expect(collapseToggle(page)).toBeChecked(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + }); + + test("the clear button appears for the toggle alone", async ({ page }) => { + // Turning the collapse off is a filter change with nothing else set, so + // the empty state must still offer a way back. + await flipCollapse(page); + await page.locator(".filter-btn", { hasText: "Chat" }).click(); + + await expect( + page.getByRole("button", { name: "Clear filters" }), + ).toBeVisible(); + }); +}); + +// Gallery descriptions are third-party Markdown. They used to be dumped raw +// into the UI, so a model whose description opened with an ATX heading showed +// a literal "# Name [](url)" in the list. +const MARKDOWN_DESCRIPTION = + "# Qwen3.6-27B\n\nChat with it at [the Qwen site](https://chat.qwen.ai) for **free**."; +const MARKDOWN_MODELS_RESPONSE = { + ...MOCK_MODELS_RESPONSE, + models: [ + { + name: "markdown-model", + description: MARKDOWN_DESCRIPTION, + backend: "llama-cpp", + installed: false, + tags: ["chat"], + }, + { + name: "headings-model", + description: + "# Top Heading\n\nBody copy.\n\n## Sub Heading\n\nMore body copy.", + backend: "llama-cpp", + installed: false, + tags: ["chat"], + }, + { + name: "no-description-model", + description: "", + backend: "llama-cpp", + installed: false, + tags: ["chat"], + }, + ], + availableModels: 3, + installedModels: 0, +}; + +test.describe("Models Gallery - Markdown descriptions", () => { + test.beforeEach(async ({ page }) => { + await page.route("**/api/models*", (route) => { + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(MARKDOWN_MODELS_RESPONSE), + }); + }); + await page.goto("/app/models"); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + }); + + test("table cell shows the description as clean text, not raw Markdown", async ({ + page, + }) => { + const row = page.locator("tr", { hasText: "markdown-model" }); + const cell = row.locator("div[title]", { hasText: "Qwen3.6-27B" }); + + await expect(cell).toHaveText( + "Qwen3.6-27B Chat with it at the Qwen site for free.", + ); + // The syntax itself must be gone, not merely rendered somewhere. + await expect(cell).not.toContainText("#"); + await expect(cell).not.toContainText("[]("); + await expect(cell).not.toContainText("**"); + await expect(cell).not.toContainText("https://chat.qwen.ai"); + // A block element here would blow up the row height. + await expect(cell.locator("h1")).toHaveCount(0); + }); + + test("title tooltip carries the stripped text, not raw Markdown", async ({ + page, + }) => { + const row = page.locator("tr", { hasText: "markdown-model" }); + const cell = row.locator("div[title]", { hasText: "Qwen3.6-27B" }); + + await expect(cell).toHaveAttribute( + "title", + "Qwen3.6-27B Chat with it at the Qwen site for free.", + ); + }); + + test("expanded detail row renders the description as real markup", async ({ + page, + }) => { + await page.locator("tr", { hasText: "markdown-model" }).click(); + + const detail = page.locator('td[colspan="8"]'); + await expect(detail.locator("h1", { hasText: "Qwen3.6-27B" })).toBeVisible(); + const link = detail.locator('a[href="https://chat.qwen.ai"]'); + await expect(link).toBeVisible(); + await expect(link).toHaveText("the Qwen site"); + await expect(detail.locator("strong", { hasText: "free" })).toBeVisible(); + }); + + test("a model without a description still shows the placeholder", async ({ + page, + }) => { + const row = page.locator("tr", { hasText: "no-description-model" }); + await expect(row).toBeVisible(); + await expect(row.locator("div[title='']")).toHaveText("—"); + }); + + test("a heading in the description renders on the UI type scale", async ({ + page, + }) => { + await page.locator("tr", { hasText: "headings-model" }).click(); + const prose = page.locator(".detail-prose__body.markdown-body"); + await expect(prose).toBeVisible(); + + const h1 = prose.locator("h1"); + await expect(h1).toHaveText("Top Heading"); + const sizes = await prose.evaluate((el) => { + const px = (sel) => + parseFloat(getComputedStyle(el.querySelector(sel)).fontSize); + return { h1: px("h1"), h2: px("h2"), p: px("p") }; + }); + // The bug: an unscoped h1 inherits the browser default 2em, which is 26px + // inside this 13px surface and swamps the pane. The scale tops out at + // --text-xl (1.25rem / 20px), so anything at or above that is the default + // leaking through rather than a styled heading. + expect(sizes.h1).toBeGreaterThan(sizes.p); + expect(sizes.h1).toBeLessThanOrEqual(20); + expect(sizes.h1).toBeGreaterThanOrEqual(14); + // The inverse defect: a subheading that is indistinguishable from body + // text. It must stay below h1 and at or above the body size. + expect(sizes.h2).toBeLessThanOrEqual(sizes.h1); + expect(sizes.h2).toBeGreaterThanOrEqual(sizes.p); + }); + + test("the description sits outside the label/value grid on a readable measure", async ({ + page, + }) => { + await page.locator("tr", { hasText: "headings-model" }).click(); + const detail = page.locator('td[colspan="8"]'); + // Description is no longer a row of the scalar table. + await expect(detail.locator("table td", { hasText: "Description" })).toHaveCount( + 0, + ); + await expect(detail.locator(".detail-prose__label")).toHaveText( + "Description", + ); + const proseWidth = await page + .locator(".detail-prose__body") + .evaluate((el) => el.getBoundingClientRect().width); + const paneWidth = await detail.evaluate( + (el) => el.getBoundingClientRect().width, + ); + // A measure, not the full pane: the cap is a ch count, so the exact pixel + // value moves with the font, but it must stay well inside the pane. + expect(proseWidth).toBeLessThan(paneWidth * 0.85); + }); + + test("a model without a description renders no prose block", async ({ + page, + }) => { + await page.locator("tr", { hasText: "no-description-model" }).click(); + const detail = page.locator('td[colspan="8"]'); + await expect(detail).toBeVisible(); + await expect(detail.locator(".detail-prose")).toHaveCount(0); + // The scalar rows still render, so the pane is not blank. + await expect(detail).toContainText("Backend"); + }); +}); + +// The filter block is three deliberate bands: query scope (search + backend +// select), the use-case chip row, and the refinements (fits-in-GPU + context). +// These assert the separation holds, because the regression they guard against +// is the refinements being swept back into the chip row's wrap, where their +// position depends on how many chips happen to wrap at the current width. +test.describe("Models Gallery - Filter layout structure", () => { + test.beforeEach(async ({ page }) => { + await page.route("**/api/models*", (route) => { + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(MOCK_MODELS_RESPONSE), + }); + }); + await page.route("**/api/backends/usecases", (route) => { + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(BACKEND_USECASES_MOCK), + }); + }); + await page.route("**/api/resources", (route) => { + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(MOCK_GPU_RESOURCES_RESPONSE), + }); + }); + await page.goto("/app/models"); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + }); + + test("the chip row contains only use-case chips", async ({ page }) => { + const chipRow = page.locator(".filter-bar"); + await expect(chipRow).toHaveCount(1); + // Nothing but .filter-btn children: no toggle, no select, no slider. + const childClasses = await chipRow.evaluate((el) => + Array.from(el.children).map((c) => c.className), + ); + expect(childClasses.length).toBeGreaterThan(0); + for (const cls of childClasses) { + expect(cls).toContain("filter-btn"); + } + await expect(chipRow.locator("input[type='range']")).toHaveCount(0); + await expect(chipRow.locator(".filter-bar-group__toggle")).toHaveCount(0); + await expect(chipRow.getByText("All Backends")).toHaveCount(0); + }); + + test("refinements live in their own band, outside the chip row", async ({ + page, + }) => { + const refine = page.getByTestId("models-filters-refine"); + await expect(refine).toBeVisible(); + await expect(refine.locator(".filter-bar")).toHaveCount(0); + await expect(refine.getByText("Fits in GPU")).toBeVisible(); + await expect(refine.locator("#models-context-size")).toBeVisible(); + // The band is a sibling of the chip row, never a descendant. + const nested = await page + .locator(".filter-bar") + .locator('[data-testid="models-filters-refine"]') + .count(); + expect(nested).toBe(0); + }); + + test("the backend select sits in the query band above the chips", async ({ + page, + }) => { + const selectBtn = page.locator("button", { hasText: "All Backends" }); + await expect(selectBtn).toBeVisible(); + const inChipRow = await page + .locator(".filter-bar") + .locator("button", { hasText: "All Backends" }) + .count(); + expect(inChipRow).toBe(0); + // Reads above the chips it gates. + const selectBox = await selectBtn.boundingBox(); + const chipBox = await page.locator(".filter-bar").boundingBox(); + expect(selectBox.y).toBeLessThan(chipBox.y); + }); + + test("refinements stay grouped and on one band at a narrow width", async ({ + page, + }) => { + await page.setViewportSize({ width: 900, height: 900 }); + const refine = page.getByTestId("models-filters-refine"); + await expect(refine).toBeVisible(); + const chipBox = await page.locator(".filter-bar").boundingBox(); + const refineBox = await refine.boundingBox(); + // Below the chip row, not interleaved with it. + expect(refineBox.y).toBeGreaterThanOrEqual(chipBox.y + chipBox.height - 1); + await expect(refine.getByText("Fits in GPU")).toBeVisible(); + await expect(refine.locator("#models-context-size")).toBeVisible(); + }); + + test("chips expose pressed state and the context slider is labelled", async ({ + page, + }) => { + const chatBtn = page.locator(".filter-btn", { hasText: "Chat" }); + await expect(chatBtn).toHaveAttribute("aria-pressed", "false"); + await chatBtn.click(); + await expect(chatBtn).toHaveAttribute("aria-pressed", "true"); + + const slider = page.locator("#models-context-size"); + // The slider steps over an index, so the announced value must be the size. + await expect(slider).toHaveAttribute("aria-valuetext", /^\d+K$/); + await expect(page.locator("label[for='models-context-size']")).toBeVisible(); + }); + + test("a keyboard-focused chip shows a focus ring", async ({ page }) => { + // The global :focus-visible rule is wrapped in :where(), so it ties with + // .filter-btn on specificity and loses on order. Without an explicit rule + // the chips render their resting shadow while focused, i.e. no indicator. + await page.locator(".filter-bar-group__search input").click(); + await page.keyboard.press("Tab"); // backend select + await page.keyboard.press("Tab"); // first chip + const focused = page.locator(".filter-btn:focus-visible"); + await expect(focused).toHaveCount(1); + // The ring transitions in, so settle before reading the computed value. + await page.waitForTimeout(400); + const shadow = await focused.evaluate( + (el) => getComputedStyle(el).boxShadow, + ); + // A 3px spread ring, not the 1px/2px resting drop shadow. + expect(shadow).toMatch(/0px 0px 0px 3px/); + }); + + test("the context control is keyboard reachable and drives the value", async ({ + page, + }) => { + const slider = page.locator("#models-context-size"); + const before = await slider.inputValue(); + await slider.focus(); + await expect(slider).toBeFocused(); + await page.keyboard.press("ArrowRight"); + await expect(slider).not.toHaveValue(before); + await expect(slider).toHaveAttribute("aria-valuetext", /^\d+K$/); + }); +}); diff --git a/core/http/react-ui/e2e/models-recommended-panel.spec.js b/core/http/react-ui/e2e/models-recommended-panel.spec.js new file mode 100644 index 000000000..7dad5ed9e --- /dev/null +++ b/core/http/react-ui/e2e/models-recommended-panel.spec.js @@ -0,0 +1,148 @@ +import { test, expect } from "./coverage-fixtures.js"; + +// The "Recommended for your hardware" strip defaults its own prominence off the +// installed-model count and remembers both the collapse choice and a dismissal, +// so every assertion here is about state that must survive a reload. + +const DISMISS_KEY = "localai-models-recommended-dismissed"; +const COLLAPSE_KEY = "localai-models-recommended-collapsed"; + +const REC_MODELS = [ + { name: "tiny-chat", description: "Tiny", backend: "llama-cpp", installed: false, tags: ["chat"] }, + { name: "small-chat", description: "Small", backend: "llama-cpp", installed: false, tags: ["chat"] }, +]; + +function listResponse(installedModels) { + return { + models: REC_MODELS, + allBackends: ["llama-cpp"], + allTags: ["chat"], + availableModels: REC_MODELS.length, + installedModels, + totalPages: 1, + currentPage: 1, + }; +} + +const ESTIMATES = { + "tiny-chat": { sizeBytes: 512 * 1024 * 1024, sizeDisplay: "512.0 MB", estimates: { 4096: { vramBytes: 700 * 1024 * 1024, vramDisplay: "700.0 MB" } } }, + "small-chat": { sizeBytes: 1024 * 1024 * 1024, sizeDisplay: "1.00 GB", estimates: { 4096: { vramBytes: 1400 * 1024 * 1024, vramDisplay: "1.40 GB" } } }, +}; + +// installedModels drives the panel's default state, so each test picks its own. +async function mockGallery(page, installedModels) { + // Registered first so the more specific routes below take precedence: + // Playwright matches the most recently added handler. + await page.route("**/api/models*", (route) => + route.fulfill({ contentType: "application/json", body: JSON.stringify(listResponse(installedModels)) }), + ); + await page.route("**/api/models/estimate/*", (route) => { + const name = decodeURIComponent(new URL(route.request().url()).pathname.split("/").pop()); + return route.fulfill({ contentType: "application/json", body: JSON.stringify(ESTIMATES[name] || {}) }); + }); + // CPU-only host, which is the branch that shows the "no GPU detected" note. + await page.route("**/api/resources", (route) => + route.fulfill({ contentType: "application/json", body: JSON.stringify({ type: "cpu", available: false, gpus: [] }) }), + ); +} + +const panel = (page) => page.getByTestId("recommended-models"); +const toggle = (page) => page.getByTestId("recommended-models-toggle"); +const grid = (page) => page.locator("#rec-models-content"); + +async function gotoModels(page) { + await page.goto("/app/models"); + await expect(panel(page)).toBeVisible({ timeout: 20_000 }); +} + +test.describe("Models gallery - recommended panel prominence", () => { + test("first visit with nothing installed shows the panel expanded", async ({ page }) => { + await mockGallery(page, 0); + await gotoModels(page); + + await expect(toggle(page)).toHaveAttribute("aria-expanded", "true"); + await expect(grid(page)).toBeVisible(); + await expect(grid(page).getByText("tiny-chat")).toBeVisible(); + }); + + test("a user with models installed gets it collapsed by default", async ({ page }) => { + await mockGallery(page, 12); + await gotoModels(page); + + await expect(toggle(page)).toHaveAttribute("aria-expanded", "false"); + await expect(grid(page)).toBeHidden(); + // Collapsed is a summary, not a removal: the heading stays on the page. + await expect(panel(page).getByText("Recommended for your hardware")).toBeVisible(); + await expect(panel(page).getByText("2 models suggested")).toBeVisible(); + }); + + test("the collapsed summary expands again on activation", async ({ page }) => { + await mockGallery(page, 12); + await gotoModels(page); + + await expect(grid(page)).toBeHidden(); + await toggle(page).click(); + + await expect(toggle(page)).toHaveAttribute("aria-expanded", "true"); + await expect(grid(page)).toBeVisible(); + await expect(page.evaluate((k) => localStorage.getItem(k), COLLAPSE_KEY)).resolves.toBe("0"); + }); + + test("the collapse choice persists across a reload", async ({ page }) => { + await mockGallery(page, 0); + await gotoModels(page); + await expect(grid(page)).toBeVisible(); + + await toggle(page).click(); + await expect(grid(page)).toBeHidden(); + + await page.reload(); + await expect(panel(page)).toBeVisible({ timeout: 20_000 }); + await expect(toggle(page)).toHaveAttribute("aria-expanded", "false"); + await expect(grid(page)).toBeHidden(); + }); + + test("dismissing it persists across a reload", async ({ page }) => { + await mockGallery(page, 0); + await gotoModels(page); + + await panel(page).getByRole("button", { name: "Dismiss recommendations" }).click(); + await expect(panel(page)).toHaveCount(0); + await expect(page.evaluate((k) => localStorage.getItem(k), DISMISS_KEY)).resolves.toBe("1"); + + await page.reload(); + // The table is the marker that the page finished rendering without the panel. + await expect(page.locator("table tbody tr").first()).toBeVisible({ timeout: 20_000 }); + await expect(panel(page)).toHaveCount(0); + }); + + test("the toggle is keyboard operable and exposes its state", async ({ page }) => { + await mockGallery(page, 12); + await gotoModels(page); + + await toggle(page).focus(); + await expect(toggle(page)).toBeFocused(); + await page.keyboard.press("Enter"); + await expect(toggle(page)).toHaveAttribute("aria-expanded", "true"); + // aria-controls must resolve to the region it actually shows and hides. + await expect(toggle(page)).toHaveAttribute("aria-controls", "rec-models-content"); + await expect(grid(page)).toBeVisible(); + }); + + test("recommendations render and their install buttons still work", async ({ page }) => { + await mockGallery(page, 0); + let installed = null; + await page.route("**/api/models/install/*", (route) => { + installed = decodeURIComponent(new URL(route.request().url()).pathname.split("/").pop()); + return route.fulfill({ contentType: "application/json", body: JSON.stringify({ uuid: "job-1" }) }); + }); + await gotoModels(page); + + const card = grid(page).locator(".rec-models-item", { hasText: "tiny-chat" }); + await expect(card).toBeVisible(); + await expect(card.getByText("512.0 MB")).toBeVisible(); + await card.getByRole("button", { name: "Install" }).click(); + + await expect.poll(() => installed).toBe("tiny-chat"); + }); +}); diff --git a/core/http/react-ui/public/locales/de/models.json b/core/http/react-ui/public/locales/de/models.json index e5929d832..3a2f3e387 100644 --- a/core/http/react-ui/public/locales/de/models.json +++ b/core/http/react-ui/public/locales/de/models.json @@ -1,6 +1,17 @@ { "title": "Modelle installieren", "subtitle": "Durchsuchen und installieren Sie KI-Modelle aus der Galerie", + "recommended": { + "title": "Empfohlen für Ihre Hardware", + "cpuNote": "Keine GPU erkannt - kleine Modelle, die auf der CPU reaktionsschnell bleiben.", + "gpuNote": "Passend zum verfügbaren VRAM, mit Spielraum für den Kontext.", + "install": "Installieren", + "installing": "Installation läuft", + "installStarted": "{{model}} wird installiert…", + "installFailed": "Installation fehlgeschlagen: {{message}}", + "dismiss": "Empfehlungen ausblenden", + "summary": "{{n}} Modelle vorgeschlagen" + }, "stats": { "available": "Verfügbar", "installed": "Installiert" @@ -24,8 +35,12 @@ "embedding": "Embedding", "rerank": "Rerank", "fitsGpu": "Passt in die GPU", + "collapseVariants": "Eine Zeile pro Modell", "allBackends": "Alle Backends", - "searchBackends": "Backends suchen..." + "searchBackends": "Backends suchen...", + "contextSize": "Kontext:", + "useCaseLabel": "Nach Anwendungsfall filtern", + "unavailableForBackend": "Für das gewählte Backend nicht verfügbar" }, "search": { "placeholder": "Modelle suchen...", @@ -71,6 +86,7 @@ "empty": { "title": "Keine Modelle gefunden", "withFilters": "Keine Modelle entsprechen den aktuellen Such- oder Filterkriterien.", + "collapsedVariantsHint": "Alternative Builds, die ein anderer Eintrag bereits anbietet, sind ausgeblendet. Schalte \"Eine Zeile pro Modell\" aus, um sie einzubeziehen.", "noFilters": "Die Modellgalerie ist leer." }, "deleteDialog": { @@ -83,5 +99,27 @@ "loadFailed": "Laden der Modelle fehlgeschlagen: {{message}}", "installFailed": "Installation fehlgeschlagen: {{message}}", "deleteFailed": "Löschen fehlgeschlagen: {{message}}" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "fits": "Fits", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "quantizationTitle": "Gewichtsformat", + "unknownQuantization": "Format unbekannt", + "showDetails": "Alle Details zu {{variant}} anzeigen", + "hideDetails": "Alle Details zu {{variant}} ausblenden", + "detailsLoading": "Details werden geladen...", + "detailsUnavailable": "Details zu {{variant}} konnten nicht geladen werden.", + "features": { + "dflash": "Schneller: DFlash", + "mtp": "Schneller: MTP" + } } } diff --git a/core/http/react-ui/public/locales/en/models.json b/core/http/react-ui/public/locales/en/models.json index bd23d389e..c9a7ab4e9 100644 --- a/core/http/react-ui/public/locales/en/models.json +++ b/core/http/react-ui/public/locales/en/models.json @@ -10,7 +10,8 @@ "installing": "Installing", "installStarted": "Installing {{model}}…", "installFailed": "Install failed: {{message}}", - "dismiss": "Dismiss recommendations" + "dismiss": "Dismiss recommendations", + "summary": "{{n}} models suggested" }, "stats": { "available": "Available", @@ -43,8 +44,12 @@ "vad": "VAD", "ner": "NER", "fitsGpu": "Fits in GPU", + "collapseVariants": "One row per model", "allBackends": "All Backends", - "searchBackends": "Search backends..." + "searchBackends": "Search backends...", + "contextSize": "Context:", + "useCaseLabel": "Filter by use case", + "unavailableForBackend": "Not available for the selected backend" }, "search": { "placeholder": "Search models...", @@ -90,6 +95,7 @@ "empty": { "title": "No models found", "withFilters": "No models match your current search or filters.", + "collapsedVariantsHint": "Alternative builds that another entry already offers are hidden. Turn off \"One row per model\" to include them.", "noFilters": "The model gallery is empty." }, "deleteDialog": { @@ -108,5 +114,27 @@ "selectModel": "Select model...", "searchPlaceholder": "Search models...", "noModels": "No models available" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "installVariant": "Install {{variant}}", + "quantizationTitle": "Weight format", + "unknownQuantization": "Unknown format", + "showDetails": "Show full details for {{variant}}", + "hideDetails": "Hide full details for {{variant}}", + "detailsLoading": "Loading details...", + "detailsUnavailable": "Details for {{variant}} could not be loaded.", + "features": { + "dflash": "Faster: DFlash", + "mtp": "Faster: MTP" + } } } diff --git a/core/http/react-ui/public/locales/es/models.json b/core/http/react-ui/public/locales/es/models.json index ac4275fbb..b7329027a 100644 --- a/core/http/react-ui/public/locales/es/models.json +++ b/core/http/react-ui/public/locales/es/models.json @@ -1,6 +1,17 @@ { "title": "Instalar modelos", "subtitle": "Explora e instala modelos de IA desde la galería", + "recommended": { + "title": "Recomendado para tu hardware", + "cpuNote": "No se detectó GPU - modelos pequeños que responden bien en CPU.", + "gpuNote": "Ajustados a tu VRAM disponible, con margen para el contexto.", + "install": "Instalar", + "installing": "Instalando", + "installStarted": "Instalando {{model}}…", + "installFailed": "Error al instalar: {{message}}", + "dismiss": "Descartar recomendaciones", + "summary": "{{n}} modelos sugeridos" + }, "stats": { "available": "Disponibles", "installed": "Instalados" @@ -24,8 +35,12 @@ "embedding": "Embedding", "rerank": "Rerank", "fitsGpu": "Cabe en la GPU", + "collapseVariants": "Una fila por modelo", "allBackends": "Todos los backends", - "searchBackends": "Buscar backends..." + "searchBackends": "Buscar backends...", + "contextSize": "Contexto:", + "useCaseLabel": "Filtrar por caso de uso", + "unavailableForBackend": "No disponible para el backend seleccionado" }, "search": { "placeholder": "Buscar modelos...", @@ -71,6 +86,7 @@ "empty": { "title": "No se encontraron modelos", "withFilters": "Ningún modelo coincide con la búsqueda o filtros actuales.", + "collapsedVariantsHint": "Las compilaciones alternativas que otra entrada ya ofrece están ocultas. Desactiva \"Una fila por modelo\" para incluirlas.", "noFilters": "La galería de modelos está vacía." }, "deleteDialog": { @@ -83,5 +99,27 @@ "loadFailed": "Error al cargar modelos: {{message}}", "installFailed": "Error al instalar: {{message}}", "deleteFailed": "Error al eliminar: {{message}}" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "fits": "Fits", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "quantizationTitle": "Formato de pesos", + "unknownQuantization": "Formato desconocido", + "showDetails": "Mostrar todos los detalles de {{variant}}", + "hideDetails": "Ocultar todos los detalles de {{variant}}", + "detailsLoading": "Cargando detalles...", + "detailsUnavailable": "No se pudieron cargar los detalles de {{variant}}.", + "features": { + "dflash": "Más rápido: DFlash", + "mtp": "Más rápido: MTP" + } } } diff --git a/core/http/react-ui/public/locales/id/models.json b/core/http/react-ui/public/locales/id/models.json index a8c5404fa..87ef33bf8 100644 --- a/core/http/react-ui/public/locales/id/models.json +++ b/core/http/react-ui/public/locales/id/models.json @@ -2,6 +2,17 @@ "title": "Instal Model", "subtitle": "Telusuri dan instal model AI dari galeri", "models": "Model", + "recommended": { + "title": "Direkomendasikan untuk perangkat keras Anda", + "cpuNote": "GPU tidak terdeteksi - model kecil yang tetap responsif di CPU.", + "gpuNote": "Disesuaikan dengan VRAM yang tersedia, menyisakan ruang untuk konteks.", + "install": "Instal", + "installing": "Menginstal", + "installStarted": "Menginstal {{model}}…", + "installFailed": "Instalasi gagal: {{message}}", + "dismiss": "Tutup rekomendasi", + "summary": "{{n}} model disarankan" + }, "stats": { "available": "Tersedia", "installed": "Terinstal" @@ -31,8 +42,12 @@ "detection": "Deteksi", "vad": "VAD", "fitsGpu": "Muat di GPU", + "collapseVariants": "Satu baris per model", "allBackends": "Semua Backend", - "searchBackends": "Cari backends..." + "searchBackends": "Cari backends...", + "contextSize": "Konteks:", + "useCaseLabel": "Filter berdasarkan kasus penggunaan", + "unavailableForBackend": "Tidak tersedia untuk backend yang dipilih" }, "search": { "placeholder": "Cari model...", @@ -78,6 +93,7 @@ "empty": { "title": "Model tidak ditemukan", "withFilters": "Tidak ada model yang cocok dengan pencarian atau filter Anda.", + "collapsedVariantsHint": "Build alternatif yang sudah ditawarkan entri lain disembunyikan. Matikan \"Satu baris per model\" untuk menyertakannya.", "noFilters": "Galeri model kosong." }, "deleteDialog": { @@ -96,5 +112,27 @@ "selectModel": "Pilih model...", "searchPlaceholder": "Cari model...", "noModels": "Model tidak tersedia" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "fits": "Fits", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "quantizationTitle": "Format bobot", + "unknownQuantization": "Format tidak diketahui", + "showDetails": "Tampilkan detail lengkap {{variant}}", + "hideDetails": "Sembunyikan detail lengkap {{variant}}", + "detailsLoading": "Memuat detail...", + "detailsUnavailable": "Detail untuk {{variant}} tidak dapat dimuat.", + "features": { + "dflash": "Lebih cepat: DFlash", + "mtp": "Lebih cepat: MTP" + } } } diff --git a/core/http/react-ui/public/locales/it/models.json b/core/http/react-ui/public/locales/it/models.json index 25c371eb1..ee5ad7ff5 100644 --- a/core/http/react-ui/public/locales/it/models.json +++ b/core/http/react-ui/public/locales/it/models.json @@ -1,6 +1,17 @@ { "title": "Installa modelli", "subtitle": "Sfoglia e installa modelli AI dalla galleria", + "recommended": { + "title": "Consigliati per il tuo hardware", + "cpuNote": "Nessuna GPU rilevata - modelli piccoli che restano reattivi su CPU.", + "gpuNote": "Dimensionati per la VRAM disponibile, con spazio per il contesto.", + "install": "Installa", + "installing": "Installazione", + "installStarted": "Installazione di {{model}}…", + "installFailed": "Installazione non riuscita: {{message}}", + "dismiss": "Nascondi i consigli", + "summary": "{{n}} modelli suggeriti" + }, "stats": { "available": "Disponibili", "installed": "Installati" @@ -24,8 +35,12 @@ "embedding": "Embedding", "rerank": "Rerank", "fitsGpu": "Entra nella GPU", + "collapseVariants": "Una riga per modello", "allBackends": "Tutti i backend", - "searchBackends": "Cerca backend..." + "searchBackends": "Cerca backend...", + "contextSize": "Contesto:", + "useCaseLabel": "Filtra per caso d'uso", + "unavailableForBackend": "Non disponibile per il backend selezionato" }, "search": { "placeholder": "Cerca modelli...", @@ -71,6 +86,7 @@ "empty": { "title": "Nessun modello trovato", "withFilters": "Nessun modello corrisponde ai filtri attuali.", + "collapsedVariantsHint": "Le build alternative gia offerte da un'altra voce sono nascoste. Disattiva \"Una riga per modello\" per includerle.", "noFilters": "La galleria modelli è vuota." }, "deleteDialog": { @@ -83,5 +99,27 @@ "loadFailed": "Caricamento modelli fallito: {{message}}", "installFailed": "Installazione fallita: {{message}}", "deleteFailed": "Eliminazione fallita: {{message}}" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "fits": "Fits", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "quantizationTitle": "Formato dei pesi", + "unknownQuantization": "Formato sconosciuto", + "showDetails": "Mostra tutti i dettagli di {{variant}}", + "hideDetails": "Nascondi tutti i dettagli di {{variant}}", + "detailsLoading": "Caricamento dettagli...", + "detailsUnavailable": "Impossibile caricare i dettagli di {{variant}}.", + "features": { + "dflash": "Più veloce: DFlash", + "mtp": "Più veloce: MTP" + } } } diff --git a/core/http/react-ui/public/locales/ko/models.json b/core/http/react-ui/public/locales/ko/models.json index 2ca05d4a7..bb9af8813 100644 --- a/core/http/react-ui/public/locales/ko/models.json +++ b/core/http/react-ui/public/locales/ko/models.json @@ -1,6 +1,17 @@ { "title": "모델 설치", "subtitle": "갤러리에서 AI 모델을 둘러보고 설치합니다", + "recommended": { + "title": "하드웨어에 맞는 추천", + "cpuNote": "GPU가 감지되지 않았습니다 - CPU에서도 빠르게 동작하는 소형 모델입니다.", + "gpuNote": "사용 가능한 VRAM에 맞추고 컨텍스트 여유를 남겼습니다.", + "install": "설치", + "installing": "설치 중", + "installStarted": "{{model}} 설치 중…", + "installFailed": "설치 실패: {{message}}", + "dismiss": "추천 닫기", + "summary": "추천 모델 {{n}}개" + }, "stats": { "available": "사용 가능", "installed": "설치됨" @@ -30,8 +41,12 @@ "detection": "감지", "vad": "VAD", "fitsGpu": "GPU에 적합", + "collapseVariants": "모델당 한 행", "allBackends": "모든 백엔드", - "searchBackends": "백엔드 검색..." + "searchBackends": "백엔드 검색...", + "contextSize": "컨텍스트:", + "useCaseLabel": "사용 사례로 필터링", + "unavailableForBackend": "선택한 백엔드에서 사용할 수 없음" }, "search": { "placeholder": "모델 검색...", @@ -77,6 +92,7 @@ "empty": { "title": "모델을 찾을 수 없습니다", "withFilters": "현재 검색 또는 필터와 일치하는 모델이 없습니다.", + "collapsedVariantsHint": "다른 항목이 이미 제공하는 대체 빌드는 숨겨져 있습니다. 포함하려면 \"모델당 한 행\"을 끄세요.", "noFilters": "모델 갤러리가 비어 있습니다." }, "deleteDialog": { diff --git a/core/http/react-ui/public/locales/zh-CN/models.json b/core/http/react-ui/public/locales/zh-CN/models.json index 246153e68..f44284757 100644 --- a/core/http/react-ui/public/locales/zh-CN/models.json +++ b/core/http/react-ui/public/locales/zh-CN/models.json @@ -1,6 +1,17 @@ { "title": "安装模型", "subtitle": "从模型库浏览和安装 AI 模型", + "recommended": { + "title": "适合你硬件的推荐", + "cpuNote": "未检测到 GPU - 这些小模型在 CPU 上依然流畅。", + "gpuNote": "已按可用显存选择，并为上下文留出空间。", + "install": "安装", + "installing": "安装中", + "installStarted": "正在安装 {{model}}…", + "installFailed": "安装失败：{{message}}", + "dismiss": "关闭推荐", + "summary": "已推荐 {{n}} 个模型" + }, "stats": { "available": "可用", "installed": "已安装" @@ -24,8 +35,12 @@ "embedding": "嵌入", "rerank": "重排", "fitsGpu": "适合 GPU", + "collapseVariants": "每个模型一行", "allBackends": "所有后端", - "searchBackends": "搜索后端..." + "searchBackends": "搜索后端...", + "contextSize": "上下文：", + "useCaseLabel": "按用例筛选", + "unavailableForBackend": "所选后端不支持" }, "search": { "placeholder": "搜索模型...", @@ -71,6 +86,7 @@ "empty": { "title": "未找到模型", "withFilters": "没有模型与当前搜索或筛选条件匹配。", + "collapsedVariantsHint": "其他条目已经提供的替代构建已被隐藏。关闭“每个模型一行”即可将其包含在内。", "noFilters": "模型库为空。" }, "deleteDialog": { @@ -83,5 +99,27 @@ "loadFailed": "加载模型失败：{{message}}", "installFailed": "安装失败：{{message}}", "deleteFailed": "删除失败：{{message}}" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "fits": "Fits", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "quantizationTitle": "权重格式", + "unknownQuantization": "未知格式", + "showDetails": "显示 {{variant}} 的完整详情", + "hideDetails": "隐藏 {{variant}} 的完整详情", + "detailsLoading": "正在加载详情...", + "detailsUnavailable": "无法加载 {{variant}} 的详情。", + "features": { + "dflash": "更快：DFlash", + "mtp": "更快：MTP" + } } } diff --git a/core/http/react-ui/src/App.css b/core/http/react-ui/src/App.css index 130f33084..a973b1294 100644 --- a/core/http/react-ui/src/App.css +++ b/core/http/react-ui/src/App.css @@ -2191,6 +2191,68 @@ select.input { user-select: none; white-space: nowrap; } +/* Models gallery filter block. Three bands (query / taxonomy / refinements) + stacked by .filter-bar-group. The refinements live in their own band so the + chip row's wrap height can never displace them. */ +.models-filters__query { + flex-wrap: nowrap; + gap: var(--spacing-sm); +} +.models-filters__backend { + flex: 0 0 auto; + min-width: 0; +} +/* .filter-bar carries its own bottom margin for standalone use; inside the + group the parent's gap owns the rhythm. */ +.models-filters > .filter-bar { + margin-bottom: 0; +} +.models-filters__refine { + display: flex; + align-items: center; + flex-wrap: wrap; + gap: var(--spacing-sm) var(--spacing-lg); + padding-top: var(--spacing-sm); + border-top: 1px solid var(--color-border-subtle); +} +/* Both refinements read as one secondary group: same size, same colour and the + same icon-then-label rhythm as .filter-bar-group__toggle. */ +.models-filters__context { + display: flex; + align-items: center; + gap: var(--spacing-sm); + font-size: var(--text-xs); + color: var(--color-text-secondary); +} +.models-filters__context > label { + display: inline-flex; + align-items: center; + gap: var(--spacing-xs); + white-space: nowrap; + cursor: pointer; +} +.models-filters__context input[type='range'] { + width: 140px; + accent-color: var(--color-primary); + cursor: pointer; +} +.models-filters__context-value { + font-weight: var(--font-weight-medium); + min-width: 3em; + font-variant-numeric: tabular-nums; +} + +@media (max-width: 768px) { + /* Search and backend select each take a full row rather than squeezing the + input below a usable width. */ + .models-filters__query { + flex-wrap: wrap; + } + .models-filters__backend { + flex: 1 1 100%; + } +} + .filter-btn__count { display: inline-flex; align-items: center; @@ -2720,6 +2782,40 @@ button.collapsible-header:focus-visible { border-color: var(--color-primary-border); } +/* The global :focus-visible ring is wrapped in :where(), so it carries the + specificity of a bare :focus-visible - the same (0,1,0) as .filter-btn, which + is declared later and therefore won with its resting shadow. Chips were left + with no visible keyboard focus indicator at all; this restates the ring where + it outranks both the resting and the hover shadow. */ +.filter-btn:focus-visible { + box-shadow: 0 0 0 3px var(--color-focus-ring); + border-color: var(--color-primary); +} + +/* A chip is disabled when the selected backend cannot serve that use case. It + keeps its resting colours so the row still reads as a set, dimmed enough to + register as out of play. */ +.filter-btn:disabled { + opacity: 0.4; + cursor: not-allowed; + box-shadow: none; +} +.filter-btn:disabled:hover { + transform: none; + box-shadow: none; + color: var(--color-text-secondary); + border-color: var(--color-border-default); +} + +@media (prefers-reduced-motion: reduce) { + .filter-btn { + transition: none; + } + .filter-btn:hover { + transform: none; + } +} + /* Login page */ .login-page { min-height: 100vh; @@ -2890,6 +2986,16 @@ button.collapsible-header:focus-visible { margin: 0; } +/* Secondary to .empty-state-text: a hint about a default that may be narrowing + the result, so it must not compete with the reason stated above it. */ +.empty-state-hint { + color: var(--color-text-muted); + font-size: var(--text-sm); + line-height: var(--leading-normal); + max-width: 52ch; + margin: 0; +} + /* Opt-in editorial sub-parts — pages can adopt these class names over time */ .empty-state__eyebrow { display: inline-flex; @@ -6449,24 +6555,47 @@ button.collapsible-header:focus-visible { margin-bottom: var(--spacing-md); padding: var(--spacing-md) var(--spacing-lg); } +/* Collapsed keeps the strip and its heading, only dropping the cards, so the + panel is recoverable in place rather than gone. Tighter block padding is + what actually returns vertical space to the gallery. */ +.rec-models--collapsed { + padding-top: var(--spacing-sm); + padding-bottom: var(--spacing-sm); +} .rec-models-head { - display: flex; - align-items: flex-start; - justify-content: space-between; - gap: var(--spacing-md); -} -.rec-models-title { display: flex; align-items: center; - gap: var(--spacing-sm); + gap: var(--spacing-sm) var(--spacing-md); flex-wrap: wrap; } -.rec-models-title i { +.rec-models-toggle { + display: inline-flex; + align-items: center; + gap: var(--spacing-sm); + background: none; + border: none; + padding: 0; + margin: 0; + font: inherit; + color: var(--color-text-primary); + cursor: pointer; + text-align: left; +} +.rec-models-toggle i { color: var(--color-primary); } +.rec-models-chevron { + font-size: 0.75rem; + transition: transform 150ms ease; +} +.rec-models--collapsed .rec-models-chevron { + transform: rotate(-90deg); +} .rec-models-note { font-size: 0.8125rem; color: var(--color-text-secondary); + flex: 1 1 12rem; + min-width: 0; } .rec-models-dismiss { background: none; @@ -6481,9 +6610,36 @@ button.collapsible-header:focus-visible { } .rec-models-grid { display: grid; - grid-template-columns: repeat(auto-fill, minmax(220px, 1fr)); + /* min() keeps the 220px track from forcing horizontal overflow on narrow + phones, where one column is all that fits anyway. */ + grid-template-columns: repeat(auto-fill, minmax(min(220px, 100%), 1fr)); gap: var(--spacing-sm); margin-top: var(--spacing-md); + /* Reveal animates opacity and transform only; the collapsed state is the + [hidden] attribute, so no height is being animated. */ + animation: rec-models-reveal 180ms ease-out; +} +/* The class sets display:grid, which would otherwise beat the UA [hidden] rule. */ +.rec-models-grid[hidden] { + display: none; +} +@keyframes rec-models-reveal { + from { + opacity: 0; + transform: translateY(-4px); + } + to { + opacity: 1; + transform: none; + } +} +@media (prefers-reduced-motion: reduce) { + .rec-models-grid { + animation: none; + } + .rec-models-chevron { + transition: none; + } } .rec-models-item { display: flex; @@ -9164,3 +9320,328 @@ button.collapsible-header:focus-visible { .model-chip__state { opacity: 0.85; font-style: normal; } .node-filter { margin-bottom: var(--spacing-lg); } .node-detail__metrics { display: flex; gap: var(--spacing-xl); margin: var(--spacing-md) 0 var(--spacing-lg); flex-wrap: wrap; } + +/* Rendered Markdown --------------------------------------------------------- + Gallery descriptions, backend descriptions and voice notes are all + author-supplied Markdown pasted from model cards. Left unscoped they inherit + browser defaults, so a description that opens with `#` renders a 2em heading + inside a 13px surface while a `##` further down is indistinguishable from + body text. This maps the whole element set onto the UI type scale so any + description reads as part of the app rather than as a document dropped into + it. Apply it to every element fed by renderMarkdown(). */ +.markdown-body { + color: var(--color-text-secondary); + font-size: var(--text-sm); + line-height: var(--leading-normal); +} +.markdown-body > :first-child { margin-top: 0; } +.markdown-body > :last-child { margin-bottom: 0; } + +.markdown-body h1, +.markdown-body h2, +.markdown-body h3, +.markdown-body h4, +.markdown-body h5, +.markdown-body h6 { + color: var(--color-text-primary); + font-weight: var(--font-weight-semibold); + line-height: var(--leading-tight); + margin: var(--spacing-md) 0 var(--spacing-xs); +} +.markdown-body h1 { font-size: var(--text-lg); } +.markdown-body h2 { font-size: var(--text-base); } +.markdown-body h3 { font-size: var(--text-sm); } +/* Below h3 the scale has no room left to separate levels by size, so the + remaining depth is carried by case and weight instead. */ +.markdown-body h4, +.markdown-body h5, +.markdown-body h6 { + font-size: var(--text-xs); + color: var(--color-text-secondary); + text-transform: uppercase; + letter-spacing: 0.06em; +} + +.markdown-body p { margin: 0 0 var(--spacing-sm); } +.markdown-body ul, +.markdown-body ol { + margin: 0 0 var(--spacing-sm); + padding-left: var(--spacing-lg); +} +.markdown-body li { margin-bottom: var(--spacing-xs); } +.markdown-body li:last-child { margin-bottom: 0; } +.markdown-body li > ul, +.markdown-body li > ol { margin-top: var(--spacing-xs); } + +.markdown-body a { + color: var(--color-primary); + text-decoration: underline; + text-underline-offset: 2px; +} +.markdown-body a:hover { color: var(--color-primary-hover); } + +.markdown-body code { + font-family: var(--font-mono); + font-size: 0.9em; + background: var(--color-bg-tertiary); + border-radius: var(--radius-sm); + padding: 1px 4px; +} +.markdown-body pre { + margin: 0 0 var(--spacing-sm); + padding: var(--spacing-sm); + background: var(--color-surface-sunken); + border: 1px solid var(--color-border-subtle); + border-radius: var(--radius-md); + overflow-x: auto; +} +.markdown-body pre code { + background: none; + padding: 0; + font-size: var(--text-xs); +} + +.markdown-body blockquote { + margin: 0 0 var(--spacing-sm); + padding-left: var(--spacing-sm); + border-left: 2px solid var(--color-border-strong); + color: var(--color-text-muted); +} +.markdown-body img { + max-width: 100%; + height: auto; + border-radius: var(--radius-sm); +} +.markdown-body hr { + border: 0; + border-top: 1px solid var(--color-border-divider); + margin: var(--spacing-md) 0; +} +/* Tables are the one element that can exceed the measure, so they scroll in + their own box rather than widening the surface around them. */ +.markdown-body table { + display: block; + width: max-content; + max-width: 100%; + overflow-x: auto; + border-collapse: collapse; + margin: 0 0 var(--spacing-sm); + font-size: var(--text-xs); +} +.markdown-body th, +.markdown-body td { + border: 1px solid var(--color-border-subtle); + padding: var(--spacing-xs) var(--spacing-sm); + text-align: left; +} +.markdown-body th { + background: var(--color-bg-tertiary); + font-weight: var(--font-weight-medium); +} + +/* Prose block in a detail pane. The label/value table beside it is right for + scalars, but multi-paragraph prose in a value cell runs the full pane width + and swamps the rows it sits above, so the description gets its own + full-width block with a reading measure. */ +.detail-prose { + margin-bottom: var(--spacing-md); +} +.detail-prose__label { + font-size: var(--text-sm); + font-weight: var(--font-weight-medium); + color: var(--color-text-secondary); + margin-bottom: var(--spacing-xs); +} +.detail-prose__body { + max-width: 68ch; +} + +/* Variant list -------------------------------------------------------------- + One row per alternative build of the same model. Names vary in length, so + the rows share the parent's tracks and line up as columns; the name leads + because it is what the reader is choosing between. Only the informative + status is badged: "fits" is true of nearly every row and says nothing, while + a build that does not fit, and the one that plain Install would pick, do. */ +.variant-list { + display: grid; + width: 100%; + /* info, name, backend, quantization, size, status, action, filler. + Every informative track is content-sized and the filler takes the slack, + so the rows stay packed together rather than stranding the install + affordance an inch of empty space away from the name it acts on. The + grid still spans the pane, which is what lets a revealed detail panel + have the full width its file table needs. */ + grid-template-columns: max-content minmax(0, auto) max-content max-content max-content max-content max-content 1fr; + column-gap: var(--spacing-md); + row-gap: 2px; +} +/* One variant: the info control, the install row, and the details the control + reveals. It spans the list's tracks and passes them down, so the install row + still shares its columns with every other row despite the extra nesting. */ +.variant-entry { + display: grid; + grid-column: 1 / -1; + grid-template-columns: subgrid; + align-items: center; +} +.variant-row { + display: grid; + /* Stops short of the filler track so the hover and focus box hugs the row's + own content rather than trailing across the empty right-hand side. */ + grid-column: 2 / -2; + grid-template-columns: subgrid; + align-items: center; + width: 100%; + margin: 0; + padding: var(--spacing-xs) var(--spacing-sm); + font: inherit; + text-align: left; + color: inherit; + background: none; + border: 1px solid transparent; + border-radius: var(--radius-sm); + cursor: pointer; + transition: background-color var(--duration-fast) var(--ease-default), + border-color var(--duration-fast) var(--ease-default); +} +/* Subgrid keeps the columns shared across rows; without it each row falls back + to sizing its own tracks, which is ragged but still legible. */ +@supports not (grid-template-columns: subgrid) { + .variant-entry { grid-template-columns: max-content minmax(22ch, auto) max-content max-content max-content max-content max-content 1fr; } + .variant-row { grid-template-columns: minmax(22ch, auto) max-content max-content max-content max-content max-content; } +} +.variant-row:hover:not(:disabled) { + background: var(--color-bg-hover); + border-color: var(--color-border-default); +} +.variant-row:focus-visible { + outline: 2px solid var(--color-focus-ring); + outline-offset: 1px; +} +.variant-row:disabled { + cursor: not-allowed; + opacity: 0.55; +} +.variant-row__name { + font-family: var(--font-mono); + font-size: var(--text-sm); + font-weight: var(--font-weight-medium); + color: var(--color-text-primary); + overflow: hidden; + text-overflow: ellipsis; +} +.variant-row__backend, +.variant-row__size { + font-size: var(--text-xs); + color: var(--color-text-muted); +} +/* Monospaced like the name above it, because a weight format is a literal + token and Q4_K_M vs Q4_K_S has to be told apart at a glance in a column. */ +.variant-row__quant { + font-family: var(--font-mono); + font-size: var(--text-xs); + color: var(--color-text-secondary); + white-space: nowrap; +} +/* An entry naming no weight format is a real state, not a gap: it recedes to + the muted colour and drops the mono face, so it reads as prose saying + "unknown" rather than as a token. */ +.variant-row__quant--unknown { + font-family: inherit; + font-style: italic; + color: var(--color-text-disabled); +} +.variant-row__size { + text-align: right; + font-variant-numeric: tabular-nums; +} +.variant-row__status { + display: inline-flex; + align-items: center; + gap: var(--spacing-xs); + flex-wrap: wrap; +} +.variant-row__action { + color: var(--color-text-disabled); + font-size: var(--text-xs); + transition: color var(--duration-fast) var(--ease-default); +} +.variant-row:hover:not(:disabled) .variant-row__action, +.variant-row:focus-visible .variant-row__action { + color: var(--color-primary); +} +/* A build that does not fit stays installable, an explicit choice being an + override the server honours, but it should not read as a peer of the ones + that do. */ +.variant-row--unfit .variant-row__name { + color: var(--color-text-muted); +} + +/* Quiet until asked for: the row's job is to be compared and installed, so the + way into "everything about this one" recedes until the eye or the keyboard + lands on it. */ +.variant-row__info { + grid-column: 1; + display: inline-flex; + align-items: center; + justify-content: center; + width: 1.75rem; + height: 1.75rem; + margin: 0; + padding: 0; + font-size: var(--text-xs); + color: var(--color-text-disabled); + background: none; + border: 1px solid transparent; + border-radius: var(--radius-sm); + cursor: pointer; + transition: color var(--duration-fast) var(--ease-default), + background-color var(--duration-fast) var(--ease-default); +} +.variant-row__info:hover, +.variant-entry:hover .variant-row__info { + color: var(--color-primary); +} +.variant-row__info:hover { + background: var(--color-bg-hover); +} +.variant-row__info:focus-visible { + outline: 2px solid var(--color-focus-ring); + outline-offset: 1px; + color: var(--color-primary); +} +.variant-row__info[aria-expanded="true"] { + color: var(--color-primary); + background: var(--color-bg-hover); + border-color: var(--color-border-default); +} + +/* The third level of disclosure on this page, so it leans on an inset and a + rule rather than another card: a nested panel with its own border inside an + expanded row inside an expanded table reads as debris. */ +.variant-detail { + grid-column: 1 / -1; + min-width: 0; + margin: var(--spacing-xs) 0 var(--spacing-sm) var(--spacing-lg); + border-left: 2px solid var(--color-border-default); + background: var(--color-surface-sunken); + border-radius: 0 var(--radius-md) var(--radius-md) 0; + overflow: hidden; +} +.variant-detail__state { + display: flex; + align-items: center; + gap: var(--spacing-sm); + padding: var(--spacing-sm) var(--spacing-md); + font-size: var(--text-sm); + color: var(--color-text-muted); +} +.variant-detail__state--error { + color: var(--color-error); +} + +@media (prefers-reduced-motion: reduce) { + .variant-row, + .variant-row__info, + .variant-row__action { transition: none; } +} diff --git a/core/http/react-ui/src/components/RecommendedModels.jsx b/core/http/react-ui/src/components/RecommendedModels.jsx index 7620406c8..55ec4e944 100644 --- a/core/http/react-ui/src/components/RecommendedModels.jsx +++ b/core/http/react-ui/src/components/RecommendedModels.jsx @@ -3,28 +3,68 @@ import { useTranslation } from 'react-i18next' import { modelsApi } from '../utils/api' import { useRecommendedModels, isNvfp4Name } from '../hooks/useRecommendedModels' -const DISMISS_KEY = 'localai_rec_models_dismissed' +// Page-scoped storage keys, matching the Models page convention +// (localai-models-*). The underscore key is the pre-rename one: reading it +// keeps an existing dismissal honoured instead of resurrecting the panel for +// everyone who already closed it. +const DISMISS_KEY = 'localai-models-recommended-dismissed' +const LEGACY_DISMISS_KEY = 'localai_rec_models_dismissed' +const COLLAPSE_KEY = 'localai-models-recommended-collapsed' +const CONTENT_ID = 'rec-models-content' + +function readDismissed() { + try { + return localStorage.getItem(DISMISS_KEY) === '1' || localStorage.getItem(LEGACY_DISMISS_KEY) === '1' + } catch { + return false + } +} + +// null means "the user has never chosen", which is what lets the installed +// count pick the default instead of overriding an explicit preference. +function readCollapsePref() { + try { + const raw = localStorage.getItem(COLLAPSE_KEY) + if (raw === '1') return true + if (raw === '0') return false + } catch { /* ignore */ } + return null +} // "Recommended for your hardware" strip at the top of the Models gallery. Shares // the hardware-fit ranking with the empty-state starter widget via -// useRecommendedModels, but styled for the gallery page and dismissible (the -// gallery is a repeat-visit surface, so it shouldn't nag). -export default function RecommendedModels({ addToast }) { +// useRecommendedModels, but styled for the gallery page. +// +// Prominence tracks need: someone with nothing installed is exactly who this is +// for and gets it expanded, while a stocked instance gets a one-line summary +// that still expands on demand. Both the collapse choice and the dismissal +// persist, so the gallery stops re-litigating the decision on every visit. +export default function RecommendedModels({ addToast, installedCount = null }) { const { t } = useTranslation('models') const { recommended, tier, loading } = useRecommendedModels({ count: 4 }) const [installing, setInstalling] = useState(() => new Set()) - const [dismissed, setDismissed] = useState(() => { - try { return localStorage.getItem(DISMISS_KEY) === '1' } catch { return false } - }) + const [dismissed, setDismissed] = useState(readDismissed) + const [collapsePref, setCollapsePref] = useState(readCollapsePref) if (loading || dismissed) return null if (!recommended || recommended.length === 0) return null + // Wait for the installed count before committing to a default: rendering + // expanded and collapsing a frame later would shove the gallery around. + if (installedCount === null || installedCount === undefined) return null + + const collapsed = collapsePref === null ? installedCount > 0 : collapsePref const dismiss = () => { try { localStorage.setItem(DISMISS_KEY, '1') } catch { /* ignore */ } setDismissed(true) } + const toggle = () => { + const next = !collapsed + try { localStorage.setItem(COLLAPSE_KEY, next ? '1' : '0') } catch { /* ignore */ } + setCollapsePref(next) + } + const install = async (name) => { setInstalling(prev => new Set(prev).add(name)) try { @@ -43,18 +83,33 @@ export default function RecommendedModels({ addToast }) { const isGpu = tier.id !== 'cpu' return ( -
+
-
+ {/* The accessible name is the visible title alone; the note sits + outside the control so the name stays short and matches the label a + voice-control user would speak. State comes from aria-expanded. */} +