mirror of
https://github.com/mudler/LocalAI.git
synced 2026-11-08 13:33:26 -05:00
44c6046472a40a2f0a33ed2e3e2eeaafa5fac348
982
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ettore Di Giacinto
|
44c6046472 |
fix(gallery): install entries with no url or config_file on an empty base
applyModel had three branches: fetch a base config from url:, build one from
an inline config_file:, or fail with "invalid gallery model". An entry
declaring neither is now installed on an empty base config, with overrides:
and files: supplying everything.
This is what the ~345 entries pointing at gallery/virtual.yaml were already
getting. That stub is five lines carrying name, description and license.
description and license are overwritten from the gallery entry immediately
after the fetch, and the name never reaches disk because InstallModel prefers
the install name. Crucially applyModel passes model.Overrides to InstallModel
as a separate argument rather than merging it into the fetched config, so
nothing an author writes depends on that base existing. The fetch bought a
round trip to GitHub and nothing else.
That makes
|
||
|
Ettore Di Giacinto
|
f4ef801733 |
fix(gallery): make deepseek-v4-flash variant targets installable
Clicking install on deepseek-v4-flash failed with "invalid gallery model". The parent entry is fine, but all four entries it was grouped with declared neither url: nor config_file:, and applyModel needs one of the two to have anything to build a config from. They carry urls: (plural), the informational HuggingFace link list, which is a different field. None of the four was ever independently installable, so grouping them routed a previously-working install into a broken entry. Give each the url: the parent already resolves through. virtual.yaml is a no-op base, and applyModel passes overrides to InstallModel separately from the fetched config, so backend: ds4, the parameters and the ssd/mtp options all still land exactly as authored. This is the same pattern the parent and many other GGUF entries in the index already use. Add the lint rule that should have caught this. checkVariantReferences only proved a target exists and is not itself a parent, which is structural validity: an entry can exist, declare no variants, and still be uninstallable. checkVariantTargetsInstallable mirrors applyModel's precondition instead, and names the parent, the target and the missing fields, because whoever hits it is reading a gallery entry and has no reason to know applyModel exists. The two index-driven resolution specs live in their own Ordered container: an Ordered container stops at its first failure, so sharing one with the lint rules let a lint breach skip them silently. Nine further entries gallery-wide have the same defect and are unrelated to variants, so they are broken installs that predate this branch. They are left alone here rather than buried in a regression fix, and widening the rule to cover every entry is deferred with them so the gate can ratchet up in one step instead of needing a skip list. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
72260d8b03 |
feat(gallery): make a declared tag the sole serving-feature signal
Variant auto-selection ranks survivors by fit, then engine, then serving feature, then size. The serving-feature lookup recognised a speculative build by either a declared tag or a whole segment of its entry name. Drop the name half: a tag is now the only signal. A name is author-supplied free text and a naming convention is not a contract, so reading a marker out of one infers a capability nobody declared. The gallery already had the failure in it: the four NVFP4 entries name MTP-bearing weights while setting no option that enables speculative decoding, and being live variants they were winning the feature axis without answering any faster. overrides.options was considered as the replacement and rejected. It carries spec_type:draft-mtp / spec_type:draft-dflash, which is what actually turns the feature on, but that spelling is llama.cpp's config vocabulary: ds4 spells the same feature mtp_path and sglang spells it speculative_algorithm in a referenced config. Keying a cross-backend ranking decision on one backend's option syntax would rank the other backends' builds as plain. Options are the curation-time check instead, and never reach the selection logic. With no fallback left, tag correctness is load bearing, so audit every entry against the rule "tagged when the entry configures that feature, in whatever vocabulary its backend uses". Three entries configure MTP untagged and gain the tag (hy3, glm-5.2, qwythos-9b-claude-mythos-5-1m, all spec_type:draft-mtp with no marker in their names). Four carry the tag while configuring nothing and lose it: qwen3.6-27b-nvfp4-mtp, qwen3.6-35b-a3b-nvfp4-mtp, qwopus3.6-27b-coder-mtp-nvfp4 and qwopus3.6-27b-v2-mtp-nvfp4, whose only option is use_jinja:true. The dflash side was checked independently rather than assumed consistent: all five dflash entries declare spec_type:draft-dflash and all five are tagged, so it needed no edits. Four entries keep a tag that a literal spec_type-only reading would strip, because they configure MTP through a different backend: deepseek-v4-flash-q2-mtp via ds4's mtp_path/mtp_draft, and the three sglang entries via speculative_algorithm in their referenced configs. Stripping those would contradict the reason spec_type was rejected as the signal and would demote four genuinely faster builds to plain. The index was edited by line insertion and deletion only, never round-tripped through a serializer. A resolved-tag diff across all 1272 named entries, taken after merge keys are applied, shows exactly these 7 changing and no entry gaining or losing a tag through an anchor. The two specs that pinned the name fallback are inverted rather than deleted, since a name silently promoting a build is the regression worth guarding. The whole-token guard survives on the tag path, where smtp must still not match mtp. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
20ea4feba3 |
feat(gallery): make the mtp tag authoritative for serving-feature ranking
Variant auto-selection ranks survivors by fit, then engine, then serving feature, then size. The serving-feature lookup read only whole alphanumeric segments of a variant's entry name, because tags were inconsistent: every dflash entry carried a dflash tag, but only 7 of 20 MTP entries carried an mtp tag. Tag the 13 untagged MTP entries, then teach the lookup to read tags as well as names. A tag is now the authoritative signal and is compared whole and case-insensitively, which is safe precisely because a tag is a deliberate declaration rather than free text: there is no word-inside-a-word failure mode, so the segment splitting the name half needs is unnecessary there. The name check stays as a fallback rather than being replaced. Switching to tags only would have regressed the six already-grouped entries on the day it shipped, and would depend on tagging discipline that does not exist yet. The lookup still names no feature, so adding one remains a one-line edit to servingFeaturePreferenceTokens. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3292f10da7 |
test: check env restore errors in capability and variant specs
errcheck flagged ten unchecked os.Setenv and os.Unsetenv returns in the specs added while the pre-commit hook was being skipped. Restoring an env var is exactly the place a silent failure leaks state into the next spec, so assert on it rather than suppressing the linter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
460ee79f3c |
gallery: prefer speculative-decoding builds when they fit
Rank serving features between engine preference and size, so a host that can
hold a DFlash or MTP build of a model's weights installs it instead of the
plain build. Both answer faster for the same output, so whenever one survives
the filters there is no reason to take the plain build.
Precedence is now fit, then engine, then serving feature, then size. Engine
outranks the feature deliberately: a serving feature makes the right engine
faster, it does not make a wrong engine right, so a plain vLLM build still
beats a DFlash llama.cpp build on NVIDIA. Fit outranks both, and a drafter
pairing is strictly larger than the plain build, so the existing size filter
drops it on a host too small for it before this axis is consulted.
The order lives in a third preference table in pkg/system, alongside the build
tag and engine name tables. It is the odd one of the three: not keyed by
capability, because no hardware prefers a plain build over an equivalent
faster one, and matched against whole segments of a gallery ENTRY NAME rather
than as a substring of a backend value. Nothing on a gallery entry declares a
serving feature, and tags are not a usable substitute: gemma-4-e2b-it:sglang-mtp
carries an mtp tag while ornith-1.0-9b-mtp and qwen3.6-27b-nvfp4-mtp carry
none. Entry names are author-supplied free text, unlike the closed engine
vocabulary, so a short marker can turn up inside an unrelated word and whole
segment matching is what keeps smtp-assistant from ranking as an MTP build.
The block comment over the tables now documents all three together and states
what each is matched against; the ranking code names no feature, so adding one
stays a one-line edit to the table.
|
||
|
Ettore Di Giacinto
|
2070b002be |
fix(gallery): prefer llama.cpp over GPU serving engines on hosts with no GPU
engineNamePreferenceRules had no row for the "default" capability, which getSystemCapabilities() returns both when no GPU is detected and when a GPU is present but under the 4 GiB VRAM floor. A missing row yields an empty preference list, which preferenceRank reads as "score everything equally", collapsing variant selection to size alone. That would be harmless if the hardware filter dropped GPU serving engines on such a host, but it does not. IsBackendCompatible derives support from the engine NAME, and "vllm" and "sglang" contain none of the darwin, cuda, rocm or sycl tokens it keys on, so they fall through to its closing "return true". A vLLM variant therefore survives on a CPU-only box and wins whenever its build is the larger of the two on offer: the machine installs vLLM in preference to llama.cpp. darwin-x86 had the identical hole. It was documented as a deliberate omission because nothing accelerates on an Intel Mac, which is true about acceleration and wrong about consequence: with every engine tied, download size decides. Add rows for both putting llama-cpp first. The GPU engines are enumerated behind it rather than left unmatched: an unmatched engine already ranks below every listed one, so llama.cpp would win either way, but unmatched engines also tie with each other and let size decide among them. Naming them fixes that order. MLX is left off the darwin-x86 row on purpose so it ranks last, since IsBackendCompatible admits darwin-tokened engines on that capability even though MLX needs Apple silicon. Preference orders survivors and never filters, so a model published only as a vLLM build is still installed on a host with no GPU; there is a spec for it. Surveyed every other value getSystemCapabilities() can return. nvidia, amd, intel and vulkan have rows; the l4t and cuda-refined values reach the nvidia row by prefix; "apple" and "" cannot reach the vendor fallthrough because the darwin and no-GPU branches return earlier. default and darwin-x86 were the only live holes. BackendPreferenceTokens and its build-tag table are untouched, and preferenceRank stays generic, naming no engine and no capability. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
c1d2947616 |
fix(cli,mcp): describe variant auto-selection as preference before size
The CLI flag help and the install_model tool schema both still said auto-selection takes the largest build that runs. Ranking now puts engine preference ahead of size, so on NVIDIA a vLLM build wins over a larger llama.cpp one. An assistant reading the old schema would tell users the wrong thing. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d32f45beca |
fix(gallery): rank variants by engine name, not backend build tag
Variant auto-selection ranked candidates with
SystemState.BackendPreferenceTokens, but that function and the variant
ranker speak different vocabularies.
BackendPreferenceTokens returns BUILD TAGS ("cuda", "rocm", "sycl",
"vulkan", "metal", "cpu"). It exists to match installed backend build
directory names like "llama-cpp-cuda-12" during alias resolution in
ListSystemBackends. Variant ranking instead matches a gallery entry's
`backend:` value, which is an ENGINE NAME: "llama-cpp", "vllm",
"vllm-omni", "sglang", "mlx" and the rest. No engine name in
gallery/index.yaml contains "cuda", "rocm", "sycl" or "vulkan".
preferenceRank matches by substring, so on an NVIDIA host the tokens
[cuda, vulkan, cpu] matched neither "llama-cpp" nor "vllm", every
candidate scored identically and size alone decided. The NVIDIA, AMD,
Intel, darwin-x86 and vulkan rules were all inert. Only metal appeared
to work, and only because the token "mlx" happens to equal an engine
name. The mismatch does not error, it silently deletes the feature.
Separate the two vocabularies. backendBuildTagPreferenceRules keeps the
build tags and its original output for every capability, including
metal, whose "mlx" token is removed again; its alias-resolution consumer
is byte-identical to before. engineNamePreferenceRules is new, holds
engine names, and is read by the new EnginePreferenceTokens, which
HostResolveEnv wires into the renamed ResolveEnv.EnginePreference. Both
tables sit adjacent under one block comment naming each vocabulary and
each consumer, and share one lookup helper so their semantics cannot
drift.
On NVIDIA the order is vLLM, then SGLang, then llama-cpp: vLLM is the
throughput engine and a model published with a vLLM build is published
that way because that build is the one worth running. AMD and Intel get
the same order, since rocm and intel builds of both serving engines
ship. Metal prefers mlx over llama-cpp. Vulkan prefers llama-cpp, the
only LLM engine with a Vulkan build. darwin-x86 and unknown
capabilities are deliberately absent rather than guessed at, degrading
to the size-only ordering that predates preference.
preferenceRank stays generic and names no engine and no capability, so
adding a runtime remains a one-line table edit.
Specs pin the NVIDIA and metal rules through the live table and the real
HostResolveEnv wiring, so emptying the engine table or wiring the build
tag source back in both go red. A regression table asserts
BackendPreferenceTokens' original output per capability, and mirrored
locks assert neither table carries the other's vocabulary.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
97b4d4e3eb |
feat(gallery): rank model variants by host backend preference
Variant auto-selection filtered candidates by whether their backend can run on the host, then ranked the survivors by size alone. The backend never influenced the choice beyond that gate, so a Mac offered both an MLX build and a llama.cpp build kept neither filtered and installed whichever was larger, leaving the native accelerated runtime unused. The same held for CUDA against CPU on NVIDIA and ROCm against Vulkan on AMD. Rank by the host's backend preference between the fit tier and size: fit stays a filter, preference decides among the builds the host can equally hold, and size still separates builds on equally preferred runtimes. The preference data stays in one declarative table in pkg/system, now read by a prefix lookup instead of a switch, so adding a capability or reordering one host's runtimes is a one-line edit and the gallery's ranking code carries no per-backend branching. MLX joins the metal rule ahead of metal itself, which is inert for the existing alias-resolution consumer because no alias group holds a candidate named for mlx. An unrecognised backend, an unrecognised capability and an absent preference list all collapse to the previous size-only ordering rather than erroring or dropping candidates. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
dd20e8f486 |
feat(ui): show the collapsed model listing by default
The gallery listing is what a user reaches for to answer "what can I install". Answering that with several rows for the same model, one per build, makes the reader do the deduplication the collapsed view already does, so the collapsed view is the one to land on. The UI now asks for collapse_variants=true unless the toggle says otherwise. The server default is deliberately untouched: a request with the parameter absent still returns the full listing, because other API clients depend on that response and collapsing it under them would be a breaking change. Opting out omits the parameter rather than sending false, so it asks for exactly the listing everyone else gets. The stored preference changes vocabulary from '1'/'0' to 'on'/'off'. The previous build wrote it from an effect that runs on mount, so a stored '0' recorded that the page had been opened rather than that anyone chose the expanded view, and honouring it would pin every earlier visitor to a default they never picked. Only the new vocabulary counts as a choice; a legacy '1' meant the collapsed view and is what the new default gives anyway, so no earlier deliberate choice is lost. Collapsing being the default also changes what the empty state may say about it. An opted-into filter can be named as the cause of an empty result; a default cannot, so the filters keep the top line and the collapsed view drops to a hint below it, shown only once filters are narrowing the set. For the same reason "Clear filters" now restores the collapsed default instead of switching it off, and the toggle alone no longer counts as a filter worth offering to clear. The label stays "One row per model": it describes the view the user is looking at rather than an action, so it reads the same whether it is opted into or out of. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
f7ff7b0ed3 |
feat(gallery): collapse the listing to one row per model
The listing supported has_variants=true, which narrowed to entries that DECLARE variants. With adoption at three entries that showed three rows, which is useless; it was always a placeholder. Replace it with the view that is actually useful: the deduplicated gallery. Show every entry installable in its own right and nothing twice, which means the parents plus every entry nobody references, and hide only the builds another entry already offers as a variant, since those are reachable through their parent. The parameter is renamed to collapse_variants accordingly: the filter is no longer a predicate on a row's own metadata but a view over the whole gallery. Default stays off, so the response with the parameter absent is unchanged. VariantReferencedIDs never reports an entry that declares variants of its own, so parents are always visible. That guarantees every hidden entry has a visible entry offering it, and no chain can strand a row. Variant resolution already refuses to install such a reference, but the listing has to stay coherent in the presence of a gallery that has one rather than silently swallowing entries. Self-references and dangling references hide nothing. The referenced set is computed over the whole gallery rather than over what the other filters left, so an entry is hidden because a parent offers it and never because of what the user searched for. The pass is over metadata already in memory: it resolves nothing over the network and triggers no variant description or size probe, so the listing's zero-probe contract still holds. The UI toggle keeps its behaviour (persistence, page reset, clear filters) and becomes "One row per model", which says what the user gets. Its localStorage key moves too, since the stored value meant a different filter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
8130d7ce9d |
ui(models): polish the variant detail view and scope rendered Markdown
The gallery detail pane rendered every field through the same two-column label/value row, including the description. Multi-paragraph prose in a value cell ran eight rows tall at the top of the pane on a ~1200px measure, breaking the grid's rhythm exactly where the eye enters. Move it into its own full-width block above the table, capped at a 68ch measure, keeping the label. Rendered Markdown had no scoped typography anywhere in the app, so a description opening with `#` inherited the browser default 2em inside a 13px surface while a `##` further down was indistinguishable from body text. Add a reusable .markdown-body block mapping h1-h6, paragraphs, lists, links, code, blockquotes, images and tables onto the existing type scale, and apply it to every renderMarkdown() consumer: the models detail, the backends detail, both Manage details and the voice library detail. Rebalance the variants list so the name leads. Backend and size drop from badge/secondary weight to muted metadata; the FITS badge goes entirely, since it was true of nearly every row and so said nothing, while the variant that does not fit keeps a warning badge and a dimmed name. AUTO-SELECTED stays marked because it answers what a plain Install produces. Rows share the parent's grid tracks via subgrid so name, backend, size and status line up down the list instead of raggedly following name length. Finally, make each variant row actionable. It looked like a list of choices but was inert text, with per-variant install hidden behind the split-button chevron elsewhere; each row is now a button onto the existing handleInstall(modelId, variant) path, with hover, keyboard focus and a disabled state while an install is in flight. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
08a434da4f |
fix(ui): strip Markdown from the backends table description cell
Commit
|
||
|
Ettore Di Giacinto
|
b35d630cf6 |
fix(ui): render gallery model descriptions as Markdown
Gallery descriptions are Markdown, but the React UI dumped them raw, so a model whose description opens with an ATX heading showed a literal "# Qwen3.6-27B [](https://chat.qwen.ai)" in the list. Full-description areas now render through renderMarkdown (marked + DOMPurify), matching how Backends.jsx and the Manage detail panels already handle the same content: - Models.jsx expanded detail row - VoiceLibrary.jsx voice detail header The truncated one-line previews must not render block Markdown: a leading "#" would become an <h1> and wreck the row height and rhythm. They get a new stripMarkdown() helper instead, which reduces Markdown to a single line of readable plain text. It is used for the cell text and for the title tooltip, since a tooltip full of "[](url)" is no better than a cell full of it: - Models.jsx gallery table description cell - Manage.jsx model and backend resource-row descriptions stripMarkdown walks marked's lexer output rather than running regexes over the source, so what it strips is by construction what renderMarkdown would have rendered, and it needs no new dependency. Output lands in JSX text nodes, so React escapes it; no new dangerouslySetInnerHTML beyond the two full-description sites, both of which run DOMPurify. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
831b127ded |
feat(ui): filter the model gallery to entries that declare variants
The gallery is heading towards showing parent entries and hiding the individual builds they reference, so a user sees one row per model rather than six quantizations of it. Adoption is a single entry today, so defaulting to that would leave a one-row gallery. This ships the migration-phase inverse instead: the default is untouched, and a toggle narrows the list to only the entries that declare variants. It previews the end state and changes nothing until someone asks for it. The filter is server-side, next to term/tag/backend/capability and above the pagination arithmetic. The listing paginates at 9 items, so narrowing on the client would leave totalPages and availableModels describing the unfiltered set and hand the user empty pages. It selects on HasVariants(), which reads already-loaded metadata, so it issues no variant probes. The parameter is named has_variants after the listing field it selects on, and is compared against "true" like the other boolean query params (all_users, save_checkpoint), so has_variants=false reads as absent. With it omitted the response is byte-for-byte what it was before. The control is the shared Toggle component, matching the fitsFilter toggle already on this page: same wrapper class, same icon and label shape, same localStorage persistence. Unlike fitsFilter it resets to page 1 on change, which a server-side filter has to do. Stacking the toggle with a tag or backend filter easily yields nothing while one entry declares variants, so the empty state now names the variants filter as the cause rather than leaving a user to conclude the gallery is broken. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
4f52c737ad |
fix(gallery): describe model variants from a companion endpoint
Variant description probes each referenced entry's weight files over the network: an HTTP HEAD plus a ranged GET, serial, five seconds per probe with no aggregate deadline. Running it inline in GET /api/models made one listing cost (entries x variants) round trips. The Manage page fetches with items=9999, so at 200 declaring entries that is ~1000 serial probes, minutes of a blocked handler and gigabytes of range traffic for a single page load. Only one entry declares variants today, but the feature exists so that many will. Follow the precedent already set for VRAM estimates. The listing now reports only has_variants, a length check on loaded metadata that touches nothing, and GET /api/models/variants/:id returns the description for one entry, mirroring estimate/:id in route shape, auth and error handling. DescribeVariants itself is unchanged; only its caller moved. The picker fetches lazily at the two points where a user asks to see variants, opening the split-button menu and expanding the detail row, and caches per entry for the page session. An entry declaring no variants issues no request at all. A spec counts real HTTP hits on the weight files, so it goes red if description becomes reachable from the listing path again through any caller. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
280c1d17b0 |
feat(ui): add a model variant picker to the models gallery
PR #10943 shipped the server side: a gallery entry may declare `variants:`, `GET /api/models` attaches `variants` and `auto_variant` to declaring entries, and `POST /api/models/install/:id` accepts a `variant` query parameter. Nothing in the UI consumed any of it, so the feature was not reachable from the browser. This wires it up. modelsApi.install takes an optional second argument and appends an encoded `?variant=` only when one is given, so every existing call site keeps sending exactly the request it sent before. On the models table, an entry that declares variants gets a split button. The primary Install still installs the auto-selected build, because auto is the default and the point of the feature; the chevron opens a menu for a deliberate override. It follows the Backends.jsx precedent: one shared Popover re-anchored per row, rendering .action-menu items, which brings Escape, outside-click and focus return along with it. An entry that declares no variants renders exactly as it did before. A variant that does not fit is dimmed but stays selectable, since the server honors an explicit choice with a warning rather than refusing it. memory_bytes is omitempty on the wire, so an absent key means the size is unknown and never zero. A single helper guards both the menu and the detail row, because formatBytes would otherwise render a falsy value as "0 B", which reads as "needs nothing". The expanded detail row gains a Variants section listing each build's backend, size, whether it fits, which is the entry's own build, and which one auto-selection would pick, built from the existing DetailRow helper and .badge classes. Eight Playwright specs cover the picker, including that plain Install sends no variant parameter and that choosing one sends it. One pre-existing assertion was scoped with .first(): the Variants section legitimately adds more llama-cpp badges to the detail row, which tripped strict mode. UI line coverage 49.42% -> 49.36% against a 40.0 baseline and 0.8pp tolerance; branch coverage rose 72.04% -> 72.66%. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
7fd90419f5 |
fix(gallery): budget variant memory from RAM when a GPU reports no VRAM
Variant selection read its memory budget from VRAM whenever a GPU capability was detected, and from system RAM only when none was. Apple Silicon satisfies the first branch and fails the premise: arm64 macs report the metal capability unconditionally, without probing anything, while TotalAvailableVRAM has no discrete VRAM pool to find and returns zero. The budget therefore came out as zero on every Mac. Zero drops every variant carrying a known size, so the base build was installed on all of them however much memory the machine had. The feature was inert on the platform, and silently: falling back to the base is a legitimate outcome, so nothing looked wrong. Take VRAM only when it is actually a number, and fall back to RAM otherwise. On a unified-memory host RAM is not an approximation of the budget, it is the budget, since the GPU shares it. A discrete GPU whose VRAM could not be read also lands on RAM, which overstates what the card holds but understates nothing the host has; the previous zero understated both. An unreadable RAM figure still yields zero and still installs the base, so a genuinely unknown host is not talked into a larger download. This is what turned tests-apple red: "installs a fitting variant's payload under the entry's own name" asserts on selection, and the runner resolved to the base because its budget was zero. The added specs pin the branch directly rather than relying on a macOS runner to notice again. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
9a43a27c2a |
fix(gallery): rank the entry's own build against its variants
Variant selection pulled the declaring entry's own payload, the base, out of the candidate set and consulted it only once every declared variant had been rejected. Two real failures followed. A variant whose size the probe cannot determine deliberately survives the memory filter, because nothing proves it does not fit. As the only survivor it then won outright on any host, however small: a 2GiB machine installed an unmeasured variant in preference to the 4GiB build the entry itself ships, with no warning. 241 of the 1280 current index entries carry no files and no size, which is exactly that shape. "Largest wins" also broke whenever the base was the largest. An author writing a Q8 entry that offers a Q4 downgrade for small hosts, a natural shape that nothing in the lint, schema or docs discourages, had the Q4 installed on every large host instead. Make the base an ordinary participant. It is still exempt from both filters, so selection always terminates on something installable, but it is now ranked against the variants: a proven fit first and largest, then the base, then any variant whose size nothing could measure. Both failures disappear together. The base is probed for its size accordingly, which it was not before, because an unsized base would lose every contest to an unmeasurable variant. FellBackToBase is kept but narrowed to "no declared variant survived", rather than "the base was chosen", since the base now also wins on merit and that is not worth warning about. A recalled variant pin also became a permanent install failure. A pin the caller supplies on this request must stay fatal, but one recalled from ._gallery_<name>.yaml can be invalidated by any later gallery edit, and failing on it turned one rename into a model that could never be reinstalled or upgraded again short of deleting a dotfile the user has never heard of. A stale recalled pin is now dropped with a warning naming it, and selection runs as if it had never been recorded. Also drop the last textual reference to two abandoned designs from the DetectedCapability comment, correct the documented variants JSON example, which showed a memory_bytes of 0 that omitempty makes impossible, and remove an em dash from the install skill. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
84de7a68a8 |
feat(gallery): drop the redundant variant min_memory field
Variant.MinMemory was an authored override for when the live probe misreads a variant's footprint. It duplicated an existing field: probeEntryMemory already passes the entry's declared size: into EstimateModelMultiContext, whose cascade prefers that declared size over its own guesswork. Correcting size: on the referenced entry fixes the figure for every consumer rather than only for variant selection, so min_memory shadowed the right answer. A variant is now nothing but a name. Its effective size is exactly the probe result, and an unknown stays unknown: it survives the filter and ranks last. EffectiveMemory loses its error return along with the field. The authored string was the only thing that could fail to parse, so the error had no remaining source and was propagating dead nil-checks through SelectVariant, DescribeVariants and the pin warning. Selection behaviour is unchanged. The specs covering probe-derived sizing, ranking, filtering, the unknown-size path, pin recall, entry/variant metadata split and deep-copy isolation all survive; the three install specs that needed a definite size now declare it through the referenced entry's own size:, which exercises the documented escape hatch directly. gallery/index.yaml is untouched: no entry ever carried the key. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
77d01de6ff |
feat(gallery): expose model variants for selection over API, CLI and MCP
A gallery entry may carry `variants:`, alternative builds of the same model. Selection already worked at install time, but nothing could see what an entry offered or ask for a specific build, so the feature was undrivable. Listing: `GET /api/models` now reports `variants` and `auto_variant` for the entries that declare variants. Each variant carries its resolved backend, its measured size and whether it fits this host. `auto_variant` is what installing without a choice would pick right now. The new gallery.DescribeVariants runs the same variantOptions + SelectVariant pass the installer runs, so the reported default cannot drift from what installing actually does, and HostResolveEnv is extracted so both derive the host and share pkg/vram's probe cache from one place. Performance: an entry that declares no variants returns early without touching the probe, so the ~1280 ordinary entries cost exactly what they cost before. Selection: `variant` is accepted on POST /models/apply, as a query param on POST /api/models/install/:id, on the gallery apply file/string request, as `local-ai models install --variant`, and as a parameter on the install_model MCP tool (both the httpapi and inproc clients). Empty means auto-select. An unknown variant name now fails the install naming what was requested. This closes a real hole: an entry declaring no variants short-circuits before selection runs, so a requested variant was previously dropped silently and the install reported success. startup.InstallModels ends in a variadic model list, so install options could not be appended to it; InstallModelsWithOptions is added alongside and InstallModels delegates to it. No caller signature changed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
7159c381ea |
gallery: size model variants with a live probe, drop the nightly denormalizer
Selection needs each variant's size to decide whether it fits and to rank largest-first. That figure was written into the index by a nightly job, which made the gallery carry a derived value that could drift from the entry it was derived from. Derive it at install time instead. pkg/vram already sizes a model without downloading it, and the gallery UI already uses it: a remote GGUF header range-fetch, then an HTTP HEAD for the content length, then any declared size:. It caches its results, so reuse it rather than writing a second probing path. A probe failure must never fail an install, so an unprobeable variant is treated as unknown: it survives the memory filter, because nothing proves it does not fit, and it ranks last, so a known-good fit always beats a guess. If every probe fails, selection still terminates on the base entry. The probe is injected through ResolveEnv rather than called directly, for the same reason the backend compatibility check is: specs pin an exact size, or an exact failure, without reaching the network. With that in place three things are dead weight and go: - The nightly job and the fields it populated. Variant.Backend was redundant because the backend is resolved live from the referenced entry during selection, and Quantization was display-only that nothing read. - min_memory on the base entry. The base always installs and its floor could only warn, so it could not change any outcome. - The lint rules and schema entries for both. min_memory on individual variants stays, as the override for when the probed size is wrong. An authored figure now suppresses the probe entirely rather than merely outranking it, so it costs no round trip. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d7eb059f44 |
feat(gallery): select model variants by hardware fit, not authored order
Gallery entries could already carry a list of alternatives, but selection was
an authored, ordered, first-match policy: every candidate declared a
`capability` string and the VRAM floors had to descend in a hand-tuned order.
That pushed hardware knowledge onto whoever edits the gallery and made ordering
load-bearing, so a reordered list silently changed what users installed.
None of it was necessary. SystemState.IsBackendCompatible already derives
hardware support from a backend name alone: it knows MLX and metal are
Darwin-only, CUDA is NVIDIA-only, ROCm AMD-only, SYCL Intel-only. Selection can
read that instead of asking authors to restate it.
Authoring is now just a list of names:
- name: qwen3.6-27b
min_memory: 4GiB
variants:
- model: qwen3.6-27b-mlx-8bit
- model: qwen3.6-27b-gguf-q8
min_memory: 28GiB
and all the intelligence moved into the selector. Given a host it drops the
variants whose backend cannot run here, drops those whose known memory
requirement exceeds what the host has, and takes the LARGEST of what is left,
because a bigger footprint is a higher quality quantization of the same model.
A variant of unknown size is kept, since nothing proves it does not fit, but it
ranks last so a proven fit always beats a guess. An explicit pin still wins
outright, and if nothing survives the entry installs its own payload: the base
always installs, this never refuses.
Available memory is VRAM when a GPU was detected and system RAM otherwise, read
through xsysinfo so a cgroup limit is honored and a container gets its own
limit rather than the node's RAM.
Capability disappears entirely, from the types, the schema and the lint. VRAM
and RAM collapse into one `min_memory`, because a model's footprint is roughly
the same wherever it lives and one figure is compared against whichever applies.
The lint rules about ordering, the capability vocabulary and floor
relationships are deleted with the hazards they described; what remains is that
every variant names an entry that exists and does not itself declare variants,
plus that any memory figure actually parses.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
30b307293a |
feat(gallery): make candidate entries complete, installable entries
Reworks hardware-resolved gallery variants after a design pivot. There is no
longer a separate "meta" entry kind. A gallery entry is a normal, complete
entry that may additionally carry candidates:, a list of hardware-gated
upgrades over itself, and the entry is itself the last-resort candidate.
The previous design relied on a bare url: as the fallback for LocalAI releases
that predate candidates support. That fallback is empty in practice: none of
the 80 gallery/*.yaml files carry a top-level files:, and 1216 of 1281 index
entries carry their payload in the index entry itself, so a url alone yields a
config template with nothing to download. Since every released LocalAI reads
gallery/index.yaml live from master, merging a payload-less entry would have
shown every existing user a model that installs to a broken state. Making the
entry its own base candidate removes the problem at the root: old clients drop
the candidates key and install the entry exactly as they do today.
Resolution order is now explicit pin, then capability plus VRAM over the
declared upgrades, then the entry itself. The entry ALWAYS installs: when its
own min_vram or capability is unmet the installer warns and installs it
anyway, because there is nothing below it and refusing would make the gallery
behave worse the newer the client is. A pin naming the entry's own name is
valid and is how an operator declines an upgrade.
IsMeta() becomes HasCandidates(), ResolveMetaModel becomes ResolveVariant, and
the persisted meta_name record key becomes entry_name. GalleryBackend.IsMeta()
is a separate concept and is untouched.
The lint drops the three rules the pivot makes wrong (url equality with the
final candidate, no inline payload, unconstrained final candidate) and gains
one: the entry's own floor must sit strictly below every candidate's, since a
base that outranks a candidate makes that candidate unreachable.
The pilot entry is now the existing nanbeige4.1-3b-q4, which gains a 2GiB
floor of its own and a single 6GiB upgrade to nanbeige4.1-3b-q8, replacing the
separate nanbeige4.1-3b entry added in
|
||
|
Ettore Di Giacinto
|
d0d441bb43 |
feat(gallery): add nanbeige4.1-3b meta entry with hardware-resolved variants
Adds the first real meta entry to the gallery index. It resolves to the Q8_0 build on hosts with at least 6GiB of VRAM and to the Q4_K_M build everywhere else, installing either payload under the stable name nanbeige4.1-3b. The entry carries a url equal to its final candidate's url. LocalAI releases that predate candidates support parse the index non-strictly and drop the key silently, so without that url they would list the entry and install nothing. A regression spec parses the index the way those releases do and asserts every meta entry stays installable for them. Also teaches core/schema/gallery-model.schema.json about candidates. The schema sets additionalProperties: false at the top level, so an author following CONTRIBUTING.md and adding the yaml-language-server comment would otherwise get a validation error on this entry. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
2794b31865 |
test(gallery): close coverage gaps in the meta entry lint
The ordering invariant grouped candidates by capability and asserted floors descend within a group. A candidate with an EMPTY capability matches every host, so it does not belong in its own group: it dominates every later candidate whose floor is at or above its own, across capability groups. Track a running minimum floor over the unconditional candidates instead, which subsumes the old same-group check for the empty capability. Every spec skipped non-meta entries, so with zero meta entries in the index all five bodies were no-ops. Aligning GalleryModel.IsMeta() with GalleryBackend.IsMeta(), whose semantics are deliberately opposite, would have made all of them pass while checking nothing. Extract each invariant into a helper over a slice of entries returning the violations it finds, and cover those helpers with synthetic fixtures so the logic stays tested at zero meta entries. The index-driven specs are now a thin application of already proven logic. Also assert the index parses non-empty, report every violation in one run rather than aborting on the first, and parse the index once for the suite. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
8ed8cf60e9 |
test(gallery): lint meta model entry invariants in index.yaml
Adds Ginkgo specs that parse the shipped gallery/index.yaml and enforce the invariants that keep meta entries safe: a legacy url fallback equal to the final candidate's url, references only to existing non-meta entries, a min_vram floor on every candidate but the last-resort one, a capability drawn only from the vocabulary the system can report, and descending VRAM floors within a capability group. The capability check is the only compensating control for a typo there. Candidate matching is a case-sensitive exact comparison against SystemState.DetectedCapability(), so an unknown value never matches and falls through silently instead of erroring. The vocabulary therefore mirrors the raw return set of getSystemCapabilities(), which notably excludes "cpu": that is a fallback key inside Capability(capMap) on the meta backend path, never a reported capability. A CPU-only host reports "default". These pass vacuously until the pilot meta entry lands; the guard is intentionally in place before the thing it guards. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
4835c4373f |
fix(gallery): deep-copy meta overrides and make two specs functional
ResolveMetaModel detached the resolved entry's Overrides and ConfigFile with maps.Clone, which only copies the top level. Gallery overrides are nested in practice (parameters.model is near-universal) and the install path merges the caller's request with mergo.WithOverride, which recurses into nested maps and overwrites them in place, so the gallery entry's own inner maps were still reachable and still got rewritten by the last caller to install. Copy both maps all the way down instead, recursing through the container shapes a YAML decoder produces. ConfigFile is not mutated on the install path today, but it carries the same kind of nested payload and leaving it shallowly cloned would invite the bug back. Also fix two specs that passed whether or not their target fix was present: - "does not write the caller's overrides back into the gallery entry" re-read the catalog from disk, which re-unmarshals fresh structs and so cannot observe in-memory aliasing. It now asserts against the in-memory gallery entry and drives the real mergo merge. - "round-trips the resolution record to disk under the meta's name" asserted a name that is already correct in the config_file branch. It now drives the url branch via a file:// fixture, where the meta-name overlay actually applies. Both were verified red by reverting their fix. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
47f37fa80d |
fix(gallery): key meta pin recall on the installed name and detach resolved entries
Six review findings on the meta-entry install path. Pin recall was keyed on the gallery entry name while applyModel writes the record under the install name (req.Name when supplied), so a meta installed under a custom name with a pin lost that pin on reinstall and was silently re-resolved onto a different variant, possibly swapping its backend. Compute the install name with applyModel's own precedence before the recall. ResolveMetaModel returned a shallow struct copy, so the resolved entry's Overrides aliased the gallery entry's map and the install path's in-place mergo merge wrote the caller's request into the shared catalog. Detach Overrides, ConfigFile, AdditionalFiles, URLs and Tags. Not exploitable today only because this path re-unmarshals the gallery per call, which is a property nobody should have to rely on. Also: overlay the meta's name onto the persisted config for meta installs so the gallery file no longer records the variant's name; move the pinned-VRAM warning below the variant validation so a pin naming a nonexistent entry does not warn about VRAM before failing for an unrelated reason; and stop seeding config.URLs in the config_file branch, which duplicated every declared URL. Add seven network-free specs driving InstallModelFromGallery with a meta entry: variant payload wins over the meta's legacy url fallback, the resolution record round-trips to disk, a pin is recorded and honored on reinstall including under a custom install name, and the resolved entry does not alias the gallery's maps. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
49fa504160 |
feat(gallery): resolve meta model entries to hardware-appropriate variants at install
Meta gallery entries carry an ordered candidate list; at install time the first candidate the host satisfies is resolved and its payload installed under the meta's name, so the model keeps a stable name regardless of which variant backs it. The resolution is recorded in the installed gallery config so a reinstall honors a prior pin and operators can see the backing variant. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
ba33c21567 |
feat(gallery): allow gallery model entries to declare variant candidates
A gallery entry with a non-empty candidates list is a meta entry: it names an ordered list of concrete entries and resolves to the first one the host can satisfy, instead of describing model files directly. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
48821a00c7 |
feat(gallery): add hardware-aware model variant resolver
Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
6caf0b40f9 |
feat(gallery): add Candidate type for meta model entries
Candidate is one option in a meta entry's ordered variant list. It names a concrete gallery entry and declares when that entry suits the host. EffectiveMinVRAM resolves the VRAM floor, letting an authored min_vram win over a nightly-inferred one. An unparseable floor errors instead of being treated as absent: swallowing a typo would turn a constrained candidate into an unconstrained one and select a too-large variant rather than fail loudly. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
0389495388 |
fix(webui): use relative asset base so fonts and lazy chunks honor X-Forwarded-Prefix (#10889) (#10904)
The Vite build emitted path-absolute asset URLs (base: '/'). index.html
entry scripts and the favicon were rewritten to include the reverse-proxy
prefix in serveIndex, but two reference kinds are not in index.html and so
bypassed that rewrite:
- CSS `url()` font references (e.g. Font Awesome .woff2), which the browser
resolves relative to the stylesheet and which `<base href>` never affects
- lazily-imported route chunks, whose preload base came from the absolute
Vite base
Under a subpath mount (X-Forwarded-Prefix: /llm/) both were fetched from the
origin root, 404ing — missing-glyph "tofu" icons and broken lazy-loaded pages.
Switch Vite to a relative base ('./') so every generated URL resolves against
the file that references it: CSS fonts and route chunks now load from
`/llm/assets/...`, and index.html's now-relative entry refs resolve via the
`<base href>` serveIndex already injects on every response. Root deployments
are unaffected. The existing path-absolute rewrite in app.go still covers the
public `/favicon.svg`.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
2dade4a9f9 |
fix(model-artifacts): gate inferred artifact materialization by backend (#10910)
The managed-artifact materializer stages a HuggingFace snapshot into a directory (.artifacts/huggingface/<key>/snapshot/). That is the right load target for directory-consuming backends (transformers, vLLM, diffusers, ...), but PrimaryArtifactSpec inferred a managed artifact from ANY HuggingFace-shaped model reference regardless of backend. A single-file backend such as llama.cpp or whisper was therefore handed the snapshot directory instead of the weight file and failed to load it. The /import-model importer already guards this with a backend allow-list (managedArtifactBackends), but the loader-side inference did not. Move the allow-list into core/config as IsManagedArtifactBackend and apply it in PrimaryArtifactSpec: only directory-consuming backends may have an artifact inferred from a bare reference; every other backend stays on the legacy download-to-file path. An explicit artifacts: block still bypasses the gate, where single-file snapshot resolution handles the load path. The importer now shares the same predicate, so both paths agree on which backends auto-materialize. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
279f5b8a93 |
fix(model-artifacts): load single-file HF snapshots from the file, not the directory (#10909)
fix(model-artifacts): load single-file HF snapshots from the file, not the dir The managed Hugging Face artifact materializer (#10825) always pointed backends at the snapshot *directory* (.artifacts/huggingface/<key>/snapshot). For a single-file model reference such as huggingface://nomic-ai/nomic-embed-text-v1.5-GGUF/nomic-embed-text-v1.5.f16.gguf, the GGUF lives *inside* that directory, so llama.cpp was handed a directory and failed with "gguf_init_from_reader: failed to read magic". This has kept the tests-aio job red on master since the feature merged (the embeddings e2e tests could not load text-embedding-ada-002). Record the single file of a one-file snapshot as Resolved.PrimaryFile and have ModelFileName() resolve to snapshot/<PrimaryFile> when it is set. Multi-file snapshots (e.g. transformers repos consumed as a directory) keep pointing at the snapshot directory. PrimaryFile is derived from the resolved contents and is deliberately excluded from the artifact cache key. estimateModelSizeBytes now derives the snapshot directory from the cache key instead of ModelFileName(), so its manifest lookup is unaffected by the file-vs-directory resolution. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a89d780707 |
fix(gallery): keep multi-file HF install progress proportional during verify (#10908)
The artifact progress bridge mapped every PhaseVerifying event to a flat 95%. The materializer emits PhaseVerifying once per file (from each file's AfterDownload hook) and downloads run sequentially, so the first small file to finish pinned the bar at 95% - and, because progress is monotonic, it stayed at 95% for the entire remaining download (e.g. a 70GB checkpoint reporting 95% at 410MB / 69.7GB). Track per-file verify proportionally to the running aggregate bytes, the same way downloading does. CurrentBytes already reflects "completed files + this file", so the percentage advances honestly. The flat 95%/99% is now reserved for the genuinely once-per-install Committing/Persisting phases. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
4be6e22b5f |
feat(webui): surface user, client IP and user agent in API traces (#10886, #10887) (#10907)
The Operate → Traces "API Traces" panel already recorded who made each request (user_id/user_name) but never showed it, and did not capture the caller's network identity at all. Operators asked to see the requesting user (#10886) and the client IP + user agent (#10887) so a trace can be attributed to who/what issued it. Backend: add ClientIP and UserAgent to APIExchange and populate them from echo's c.RealIP() (honours X-Forwarded-For / X-Real-IP behind a trusted proxy) and the request's User-Agent header. Both are omitempty and the /api/traces swagger response is map[string]any, so this is additive. UI: add a sortable "User" column to the API traces table and a metadata block (User / Client IP / User Agent) at the top of the expanded row detail. Fields render only when present, so older buffered traces and unauthenticated/local requests degrade cleanly. Adds an e2e spec covering the new column value and the expanded metadata. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
bf484c5181 |
feat(webui): show date alongside time in the Traces view (#10888) (#10905)
The Operate -> Traces table rendered the request time with the time of day only, so entries that span more than one day were ambiguous. Add a formatDateTime helper (localized date + existing time-with-millis) and use it for the Traces "Time" column, keeping the cell on a single line. The shared formatTimestamp used by the log views is unchanged. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
futurehua
|
d3ea65a112 |
refactor: replace Split in loops with more efficient SplitSeq (#10879)
Signed-off-by: futurehua <futurehua@outlook.com> |
||
|
walcz-de
|
6ccb1130d8 |
fix(agent-ui): reset streamed text at generation boundaries in agent chat (#10664)
One agent turn runs several internal LLM generations (tool selection, reasoning, final answer) that all emit stream_event deltas over the same per-agent SSE channel. The chat page accumulated every 'content' delta into a single live bubble and ignored the 'done' boundary events, so the internal generations' text (e.g. the English tool-selection rationale) merged with — and visually corrupted — the streamed final answer. Reset the accumulated content/reasoning on 'done': each generation gets a clean live bubble, and the authoritative full answer still arrives via the final json_message event as before. Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> |
||
|
Tai AnandAnai-Guo
|
45b8047736 |
fix(p2p): serialize access to p2pCtx/p2pCancel (#10839) (#10861)
StopP2P() read and wrote a.p2pCtx/a.p2pCancel without holding a.p2pMutex, and StartP2P() reassigned both fields with no lock at all -- including when RestartP2P() calls it from a background goroutine after releasing the mutex. Both paths are reachable from POST /api/settings (empty p2p_token -> StopP2P, non-empty -> RestartP2P), so concurrent requests race on the same fields. Take a.p2pMutex in StopP2P and around the field publication in StartP2P, factor the shared teardown into stopP2PLocked() so RestartP2P reuses it, and route the goroutine error path through StopP2P instead of touching a.p2pCancel unlocked. Signed-off-by: Anai-Guo <antai12232931@anaiguo.com> Co-authored-by: Anai-Guo <antai12232931@anaiguo.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
c1a891662c |
refactor(settings): single declarative registry for runtime settings (fixes the #10845 bug class) (#10864)
* feat(settings): add declarative runtime-settings field registry One fieldSpec row per RuntimeSettings field, with a reflection completeness spec so a field added without a registry row is a red test instead of a silently-dropped setting (the #10845 bug class). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * refactor(settings): drive ToRuntimeSettings/ApplyRuntimeSettings from the field registry Behavior-preserving: ~350 hand-written per-field lines become two loops over runtimeSettingsFields, gated by a To->Apply->To round-trip spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * feat(settings): baseline-driven startup merge for persisted runtime settings ApplyRuntimeSettingsAtStartup compares the live config against DefaultRuntimeBaseline (option-less-run defaults incl. kong-injected flag defaults) instead of per-field == 0 guards. Fixes persisted lru_eviction_max_retries, tracing_max_items, agent_job_retention_days, memory_reclaimer_threshold, galleries and autoload flags being silently ignored at boot. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * fix(settings): registry-driven startup merge, applied before consumers loadRuntimeSettingsFromFile becomes a thin wrapper over ApplyRuntimeSettingsAtStartup and runs at the top of New(), before model configs capture app-level defaults. WithThreads stops eagerly resolving 0 so a persisted thread count survives restart while LOCALAI_THREADS still wins (#10845); the physical-core fallback moves after the merge. Also: run.go now injects the memory-reclaimer threshold unconditionally so the option-less boot matches DefaultRuntimeBaseline and a UI-saved threshold survives restart. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * refactor(settings): file watcher delegates to the registry merge; shared API-key merge Manual edits to runtime_settings.json now behave like a boot-time load (env still wins) instead of the inverted diverged-from-startup guard that ignored most manual edits. MergeAPIKeys dedups env keys in one place for the endpoint and the watcher. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * docs(settings): document unified runtime-settings precedence Document the single env/CLI > runtime_settings.json > defaults rule, applied identically at boot, on POST /api/settings, and on manual file edits, plus the two known limitations (default-valued env vars are indistinguishable from unset; API-changed fields hot-apply on the next restart only). Also add a completion debug log when the watcher applies runtime_settings.json. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * test(settings): reset the global VRAM cap leaked by the round-trip spec The round-trip spec applies vram_budget=12GiB, whose post-loop hook installs a process-global default cap; without a reset every spec ordered after it runs under that phantom budget. Also drop a stale enumeration in the ApplyRuntimeSettings doc comment. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Tai AnandAnai-Guo
|
dc2cc4da43 |
fix(audio-transform): serialize WebSocket writes to avoid concurrent-write panic (#10857)
* fix(audio-transform): serialize WebSocket writes to avoid concurrent-write panic AudioTransformStreamEndpoint writes to the same Gorilla WebSocket connection from two goroutines: the backend-forwarding goroutine emits binary PCM frames (and can call sendWSError on a backend recv error), while the read loop calls sendWSError for malformed mid-stream JSON or a backend send failure. Gorilla WebSocket permits only one concurrent writer, so these writers race and can panic with "concurrent write to websocket connection", resetting the client session; a -race build reports the data race directly. Wrap the connection in a lockedConn that serializes WriteMessage behind a mutex, mirroring the existing lockedConn used by the openresponses WebSocket endpoint. Reads stay on the single read loop, so only writes need the lock. Fixes #10844 Signed-off-by: Tai An <antai12232931@outlook.com> * chore: empty commit to re-trigger checks Signed-off-by: Anai-Guo <antai12232931@anaiguo.com> --------- Signed-off-by: Tai An <antai12232931@outlook.com> Signed-off-by: Anai-Guo <antai12232931@anaiguo.com> Co-authored-by: Anai-Guo <antai12232931@anaiguo.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
bbe018c1a0 |
feat(bonsai): PrismML llama.cpp fork backend + Bonsai/Ternary-Bonsai gallery models (#10834)
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp (github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0 (1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants. Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models are served with the same OpenAI-compatible API. No grpc-server allow-list patch is needed (bonsai adds weight quants, transparent to the server, not KV-cache types), and the reused server compiles cleanly against the fork with no skew patches (validated locally via a CPU docker build; patches/ is present but empty for any future re-pin skew). Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh, backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml meta-backend + per-platform images, and a nightly bump_deps entry tracking the `prism` branch. Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64, +pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Tai AnandAnai Guo
|
808312b4b9 |
fix(watchdog): guard StopWatchdog with watchdogMutex to prevent double close (#10841) (#10859)
fix(watchdog): guard StopWatchdog with watchdogMutex to prevent double close StopWatchdog checked, closed and cleared a.watchdogStop without holding a.watchdogMutex, while startWatchdog and RestartWatchdog reassign and close the same channel under that lock. POST /api/settings dispatches to StopWatchdog or RestartWatchdog depending on ApplicationConfig.WatchdogShouldRun(), so both are reachable concurrently. Two callers can observe a non-nil watchdogStop and both close it, which panics with 'close of closed channel' and takes the server down. Take the mutex, matching the other two writers. StopWatchdog is only called from the settings handler, which holds no lock, so this cannot deadlock. Fixes #10841 Co-authored-by: Anai Guo <antai12232931@anaiguo.com> |
||
|
Tai An
|
5fe48e4910 |
fix(backend): don't crash the whole process on an invalid cutstrings/extract_regex (#10855)
Finetune() compiled every model cutstrings/extract_regex entry via regexp.Compile
and called xlog.Fatal on failure, which terminates the entire local-ai process.
A single model config with an invalid regex (e.g. cutstrings: ["("]) turns one
/v1/chat/completions request into a process-level denial of service.
Log the compile error and skip the offending pattern instead. The mutex is
released before continuing, and skipping avoids dereferencing the nil regexp
that removing the fatal would otherwise leave behind.
Fixes #10843
Signed-off-by: Tai An <antai12232931@outlook.com>
|
||
|
Tai AnandAnai-Guo
|
688f904a10 |
fix(runtime-settings): apply persisted threads/context_size/f16 at startup (#10853)
ApplyRuntimeSettings persists the performance settings (threads, context_size, f16) on the live /api/settings path, but the startup loader loadRuntimeSettingsFromFile never read them back, so a value saved via the Middleware UI was silently ignored on the next restart: the model booted with the CLI/physical-core default and GET /api/settings echoed that default instead of the saved value (#10845). Threads needs special handling: unlike context_size/f16, WithThreads eagerly resolves an unset (0) value to xsysinfo.CPUPhysicalCores() at option-apply time, so options.Threads is never 0 in the loader and the usual "== default" heuristic cannot tell an env/CLI value from the physical-core fallback. Detect LOCALAI_THREADS/THREADS explicitly so the env still wins over the persisted file value. Signed-off-by: Anai-Guo <Anai-Guo@users.noreply.github.com> Co-authored-by: Anai-Guo <Anai-Guo@users.noreply.github.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
8cec22c3b7 |
feat(vram): per-node VRAM allocation budget (LOCALAI_VRAM_BUDGET) (#10833)
* feat(vram): add vrambudget primitive for per-node VRAM caps Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): apply default VRAM budget in xsysinfo aggregate getters Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): wire LOCALAI_VRAM_BUDGET flag to xsysinfo default budget Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): persist VRAM budget via runtime settings with live apply Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(vram): reset process-global VRAM budget after runtime-settings spec Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): add VRAM budget field to Settings page Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): store and enforce per-node VRAM budget in the node registry Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): apply per-node VRAM budget in router hardware defaults Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): report worker VRAM budget in node registration The distributed worker now reports its operator-set VRAM budget string (LOCALAI_VRAM_BUDGET) to the server on registration. The worker keeps reporting RAW total/available VRAM and never sets the xsysinfo process-global budget (that stays standalone-only); the server resolves and enforces the budget uniformly (Task 6). Also closes a Task 6 gap: on re-registration, a struct Updates zero-skips an empty budget, so a worker that dropped LOCALAI_VRAM_BUDGET left the stale cap in place. For non-admin-override nodes the budget columns are now force-written (map Updates) even when empty, so removing the env var clears the cap; admin overrides are preserved unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(vram): drop em dash from worker-clear comment Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): add node VRAM budget admin endpoints Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): add node VRAM budget control to the node UI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): expose set_node_vram_budget MCP admin tool Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(vram): document LOCALAI_VRAM_BUDGET and node VRAM budget UI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vram): avoid double-applying VRAM budget in GetResourceAggregateInfo The GPU-branch aggregate returned by GetResourceInfo is sourced from GetGPUAggregateInfo, which already caps total/free/used against the process-wide VRAM budget. GetResourceAggregateInfo then applied the budget a second time. For an absolute budget this is idempotent, but for a percentage budget b.Apply resolves the ceiling as a fraction of its input total, so a second pass yields P*(P*T) instead of P*T and distorts UsagePercent (read by the memory reclaimer in pkg/model/watchdog.go). Remove the redundant second application so the budget is applied exactly once, against the raw physical totals, upstream in GetGPUAggregateInfo. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vram): implement SetNodeVRAMBudget on mcp assistant test stub The LocalAIClient interface gained SetNodeVRAMBudget; the stubClient in core/http/endpoints/mcp used by the assistant tests is a separate implementer and needs the method too (broke golangci-lint typecheck and both test jobs). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |