From 9a43a27c2ac72567bcd32c2bcd4375723cc33587 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 18 Jul 2026 23:49:14 +0000 Subject: [PATCH] fix(gallery): rank the entry's own build against its variants Variant selection pulled the declaring entry's own payload, the base, out of the candidate set and consulted it only once every declared variant had been rejected. Two real failures followed. A variant whose size the probe cannot determine deliberately survives the memory filter, because nothing proves it does not fit. As the only survivor it then won outright on any host, however small: a 2GiB machine installed an unmeasured variant in preference to the 4GiB build the entry itself ships, with no warning. 241 of the 1280 current index entries carry no files and no size, which is exactly that shape. "Largest wins" also broke whenever the base was the largest. An author writing a Q8 entry that offers a Q4 downgrade for small hosts, a natural shape that nothing in the lint, schema or docs discourages, had the Q4 installed on every large host instead. Make the base an ordinary participant. It is still exempt from both filters, so selection always terminates on something installable, but it is now ranked against the variants: a proven fit first and largest, then the base, then any variant whose size nothing could measure. Both failures disappear together. The base is probed for its size accordingly, which it was not before, because an unsized base would lose every contest to an unmeasurable variant. FellBackToBase is kept but narrowed to "no declared variant survived", rather than "the base was chosen", since the base now also wins on merit and that is not worth warning about. A recalled variant pin also became a permanent install failure. A pin the caller supplies on this request must stay fatal, but one recalled from ._gallery_.yaml can be invalidated by any later gallery edit, and failing on it turned one rename into a model that could never be reinstalled or upgraded again short of deleting a dotfile the user has never heard of. A stale recalled pin is now dropped with a warning naming it, and selection runs as if it had never been recorded. Also drop the last textual reference to two abandoned designs from the DetectedCapability comment, correct the documented variants JSON example, which showed a memory_bytes of 0 that omitempty makes impossible, and remove an em dash from the install skill. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto --- .agents/adding-gallery-models.md | 10 +- core/gallery/describe_variants.go | 7 +- core/gallery/models.go | 43 ++++-- core/gallery/resolve_variant.go | 129 ++++++++++++------ core/gallery/resolve_variant_test.go | 74 +++++++++- core/gallery/variants_install_test.go | 101 ++++++++++++-- core/services/galleryop/model_variant_test.go | 13 +- docs/content/features/model-gallery.md | 24 +++- .../prompts/skills/install_chat_model.md | 2 +- pkg/system/capabilities.go | 6 +- 10 files changed, 322 insertions(+), 87 deletions(-) diff --git a/.agents/adding-gallery-models.md b/.agents/adding-gallery-models.md index 93563c762..aef70e060 100644 --- a/.agents/adding-gallery-models.md +++ b/.agents/adding-gallery-models.md @@ -114,11 +114,15 @@ Rules: and must not declare `variants` of its own. - **Order carries no meaning.** Do not try to encode a preference; write the list in whatever order reads best. +- **A variant may be smaller than the declaring entry.** Offering a downgrade + for small hosts is a normal shape: the declaring entry's own build competes on + size like every other candidate, so a large host keeps the large build. - **Do not describe hardware.** At install time LocalAI drops variants whose backend cannot run on the host, drops those that do not fit available memory, - and installs the largest survivor, falling back to the declaring entry's own - build. Sizes are measured live from the weights and cached, so nothing has to - be written down. + and installs the largest of what is left, the declaring entry's own build + included. That build is never dropped, so selection always terminates on + something installable. Sizes are measured live from the weights and cached, so + nothing has to be written down. - A variant is nothing but a name; there is no per-variant memory field. When the measured size for a build is wrong, correct it on the referenced entry by setting that entry's own `size:` (e.g. `size: "20GiB"`). The estimator prefers diff --git a/core/gallery/describe_variants.go b/core/gallery/describe_variants.go index 726308909..b7bf52b88 100644 --- a/core/gallery/describe_variants.go +++ b/core/gallery/describe_variants.go @@ -12,9 +12,10 @@ type VariantView struct { // it, and it is also the reason Fits may be false on a host whose memory // would be ample. Backend string `json:"backend"` - // MemoryBytes is the measured footprint, or 0 when it could not be - // determined. Zero is unknown, never "needs nothing", so a client must - // render it as unknown rather than as a free option. + // MemoryBytes is the measured footprint. It is omitted from the JSON + // entirely when the size could not be determined, rather than serialized as + // a zero that a client would have to know to read as unknown. An absent key + // never means "needs nothing". MemoryBytes uint64 `json:"memory_bytes,omitempty"` // Fits reports whether auto-selection would consider this variant on this // host: its backend can run here and its known footprint is within budget. diff --git a/core/gallery/models.go b/core/gallery/models.go index d1fbae9b6..18a314dfc 100644 --- a/core/gallery/models.go +++ b/core/gallery/models.go @@ -111,13 +111,18 @@ func variantOptions(models []*GalleryModel, entry *GalleryModel, env ResolveEnv) options = append(options, option) } - // The base is never filtered nor ranked, so its size would change nothing - // and probing it would spend a round trip on a discarded answer. - return append(options, VariantOption{ + // The base is probed like any other candidate. It is never filtered out, but + // it IS ranked against the variants, and an unsized base would lose every + // contest to a variant whose size nothing could measure. + base := VariantOption{ Variant: Variant{Model: entry.Name}, Backend: entry.Backend, IsBase: true, - }), nil + } + if env.ProbeMemory != nil { + base.ProbedMemory = env.ProbeMemory(entry) + } + return append(options, base), nil } // probeContextLength is the context size the footprint estimate is taken at. @@ -180,9 +185,10 @@ func probeEntryMemory(ctx context.Context, entry *GalleryModel) uint64 { // // The base entry always resolves, whatever the host has. It is a complete entry // that every older LocalAI release installs unconditionally, so refusing it here -// would make the gallery behave worse the newer the client is. That is also why -// the base declares no memory requirement of its own: a floor that can only -// warn cannot change any outcome. +// would make the gallery behave worse the newer the client is. Being exempt from +// the filters does not make it exempt from ranking: it is measured and compared +// like every declared variant, and wins by default only once they have all been +// ruled out. func ResolveVariant(models []*GalleryModel, entry *GalleryModel, env ResolveEnv, pin string) (*GalleryModel, Variant, error) { options, err := variantOptions(models, entry, env) if err != nil { @@ -462,19 +468,40 @@ func InstallModelFromGallery( // back under the entry's own name would miss the record for every custom-named // install and silently re-resolve a deliberately pinned model onto a // different variant, possibly swapping its backend. + // + // recalledPin tracks where the pin came from, because the two sources must + // fail differently when the name no longer resolves. See below. + recalledPin := "" if pin == "" { installName := model.Name if req.Name != "" { installName = req.Name } if previous, err := GetLocalModelConfiguration(systemState.Model.ModelsPath, installName); err == nil && previous != nil { - pin = previous.PinnedVariant + recalledPin = previous.PinnedVariant + pin = recalledPin } } env := HostResolveEnv(ctx, systemState) resolved, variant, err := ResolveVariant(models, model, env, pin) + // A pin the caller supplied on this request must stay fatal: they named + // something this entry cannot give, and installing anything else would + // report success for a request that was not honored. + // + // A pin recalled from disk is different. The gallery can rename or withdraw + // a variant long after it was pinned, and the user is not asking for it + // again on this call. Failing here would turn one gallery edit into a + // permanently unrepairable model whose only remedy is deleting a dotfile + // they have never heard of, so the stale pin is dropped and selection runs + // as if it had never been recorded. + if err != nil && recalledPin != "" && errors.Is(err, ErrPinNotFound) { + xlog.Warn("The recorded variant pin for this model no longer exists in the gallery; re-selecting automatically", + "model", model.Name, "dropped_pin", recalledPin) + pin = "" + resolved, variant, err = ResolveVariant(models, model, env, "") + } if err != nil { return err } diff --git a/core/gallery/resolve_variant.go b/core/gallery/resolve_variant.go index e63bbffb5..34363b55a 100644 --- a/core/gallery/resolve_variant.go +++ b/core/gallery/resolve_variant.go @@ -27,8 +27,10 @@ type VariantOption struct { // precisely why a gallery author never has to describe hardware. Backend string // IsBase marks the declaring entry's own payload. It is exempt from every - // filter and is the answer when nothing else survives, because the entry - // must stay installable on every host and for every client. + // filter, because the entry must stay installable on every host and for + // every client, but it is otherwise an ordinary candidate: it is ranked + // against the declared variants rather than consulted only once they have + // all been rejected. IsBase bool // ProbedMemory is the footprint measured live from the referenced entry's // weights, in bytes. It is the only source of a variant's size. An author @@ -87,9 +89,14 @@ func (e ResolveEnv) backendRuns(backend string) bool { // VariantSelection is the outcome of a selection pass. type VariantSelection struct { Option VariantOption - // FellBackToBase reports that no declared variant survived and the entry's - // own payload was chosen instead. Callers log this, because a host quietly - // taking the base when upgrades were on offer is worth being able to see. + // FellBackToBase reports that no declared variant survived the filters and + // the entry's own payload was all that remained. Callers log this, because a + // host quietly taking the base when upgrades were on offer is worth being + // able to see. + // + // It is deliberately narrower than "the base was selected": the base also + // wins on merit whenever it outranks every surviving variant, and that is an + // ordinary, uninteresting outcome rather than something to warn about. FellBackToBase bool // Reasons explains, one line per rejected variant, why it was dropped. Reasons []string @@ -107,11 +114,13 @@ type VariantSelection struct { // dropped. A variant with an UNKNOWN requirement survives, because nothing // proves it does not fit and refusing on a size the probe could not read // would let a network hiccup silently downgrade what gets installed. -// 4. The largest survivor wins. A bigger footprint is a higher quality -// quantization of the same model, so among things that fit, more is better. -// Unknown requirements rank last, so a proven fit always beats a guess. -// 5. With no survivor the base option wins. The base always installs; this -// never refuses. +// 4. The base survives both filters unconditionally, and then competes. It is +// a candidate like any other, not a last resort: an entry whose own build is +// the largest thing that fits must win against a smaller variant, and a base +// of known size must win against a variant whose size nothing could measure. +// 5. The survivors are ranked and the best one wins, by rankOf below. +// 6. With no survivor at all, which can only happen when the caller supplied no +// base, there is nothing to install and this reports ErrNoVariantMatch. func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (VariantSelection, error) { if pin != "" { for _, o := range options { @@ -125,54 +134,88 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant type ranked struct { option VariantOption memory uint64 - known bool + rank int } - var base *VariantOption survivors := make([]ranked, 0, len(options)) reasons := make([]string, 0, len(options)) + survivingVariants := 0 for i := range options { o := options[i] - if o.IsBase { - base = &options[i] - continue - } - - if !env.backendRuns(o.Backend) { - reasons = append(reasons, fmt.Sprintf("%s needs backend %q, which cannot run on this system", o.Variant.Model, o.Backend)) - continue - } - memory, known := o.EffectiveMemory() - if known && memory > env.AvailableMemory { - reasons = append(reasons, fmt.Sprintf("%s needs %s of memory", o.Variant.Model, humanBytes(memory))) - continue + + // The base skips both gates. There is nothing below it, so refusing it + // would make an entry every older LocalAI installs fine uninstallable on + // newer ones. + if !o.IsBase { + if !env.backendRuns(o.Backend) { + reasons = append(reasons, fmt.Sprintf("%s needs backend %q, which cannot run on this system", o.Variant.Model, o.Backend)) + continue + } + if known && memory > env.AvailableMemory { + reasons = append(reasons, fmt.Sprintf("%s needs %s of memory", o.Variant.Model, humanBytes(memory))) + continue + } + survivingVariants++ } - survivors = append(survivors, ranked{option: o, memory: memory, known: known}) + survivors = append(survivors, ranked{option: o, memory: memory, rank: rankOf(o, env)}) } - if len(survivors) > 0 { - // Stable so that variants with identical requirements keep their - // authored order, which is the only thing order still decides. - sort.SliceStable(survivors, func(i, j int) bool { - if survivors[i].known != survivors[j].known { - return survivors[i].known - } - return survivors[i].memory > survivors[j].memory - }) - return VariantSelection{Option: survivors[0].option, Reasons: reasons}, nil + if len(survivors) == 0 { + return VariantSelection{}, fmt.Errorf( + "%w: %s of memory available; variants: %s", + ErrNoVariantMatch, humanBytes(env.AvailableMemory), strings.Join(reasons, "; "), + ) } - if base != nil { - return VariantSelection{Option: *base, FellBackToBase: true, Reasons: reasons}, nil - } + // Stable so that options within one rank and of identical size keep their + // authored order, which is the only thing order still decides. + sort.SliceStable(survivors, func(i, j int) bool { + if survivors[i].rank != survivors[j].rank { + return survivors[i].rank < survivors[j].rank + } + return survivors[i].memory > survivors[j].memory + }) - return VariantSelection{}, fmt.Errorf( - "%w: %s of memory available; variants: %s", - ErrNoVariantMatch, humanBytes(env.AvailableMemory), strings.Join(reasons, "; "), - ) + winner := survivors[0].option + return VariantSelection{ + Option: winner, + // Only a base that won by default is worth reporting. A base that + // outranked live competition is an ordinary selection. + FellBackToBase: winner.IsBase && survivingVariants == 0, + Reasons: reasons, + }, nil +} + +// Ranks, best first. Within a rank the larger footprint wins, because a bigger +// build is a higher quality quantization of the same model. +const ( + // rankProvenFit is a measured size that the host is measured to satisfy. + rankProvenFit = iota + // rankBase is the entry's own build when it is not a proven fit: either it + // needs more memory than the host reports, or its size could not be + // measured either. It still outranks any unsized variant, because it is the + // payload the entry is guaranteed to be able to install and a variant of + // unmeasurable size is a guess. Taking the guess on a host that cannot be + // shown to accommodate it is how an unreachable network silently changes + // what gets installed. + rankBase + // rankUnknownFit is a variant whose size nothing could measure. Nothing + // proves it does not fit, so it is not dropped, but nothing proves it does + // either, so it ranks last. + rankUnknownFit +) + +func rankOf(o VariantOption, env ResolveEnv) int { + if memory, known := o.EffectiveMemory(); known && memory <= env.AvailableMemory { + return rankProvenFit + } + if o.IsBase { + return rankBase + } + return rankUnknownFit } func humanBytes(b uint64) string { diff --git a/core/gallery/resolve_variant_test.go b/core/gallery/resolve_variant_test.go index 3c3a09474..08713d9c0 100644 --- a/core/gallery/resolve_variant_test.go +++ b/core/gallery/resolve_variant_test.go @@ -47,7 +47,8 @@ var _ = Describe("SelectVariant", func() { } // The base is exempt from every filter, which the fallback specs below pin - // down; its size is carried only so ranking has nothing special to do. + // down, but it is ranked against the variants like any other candidate, so + // its size is load-bearing. base := func(model string, probed uint64) gallery.VariantOption { o := option(model, "llama-cpp", probed) o.IsBase = true @@ -159,10 +160,64 @@ var _ = Describe("SelectVariant", func() { selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(4)}, "") Expect(err).ToNot(HaveOccurred()) - Expect(selection.Option.Variant.Model).To(Equal("m-unknown")) + // Surviving is observable through the rejection reasons: a dropped + // variant is always accounted for there, and this one is not. + Expect(selection.Reasons).ToNot(ContainElement(ContainSubstring("m-unknown"))) + // It survives, but it does not win: the base is a sized, guaranteed + // payload and an unmeasurable variant is a guess. + Expect(selection.Option.Variant.Model).To(Equal("m-base")) Expect(selection.FellBackToBase).To(BeFalse()) }) + It("installs the base rather than an unsized variant on a host too small for either", func() { + // The exact shape 241 of the current index entries have: a referenced + // entry with no files and no size, whose probe can only answer + // "unknown". Ranking it above the base would install an unmeasured + // download on a machine with 2GiB, and would do so silently. + options := []gallery.VariantOption{ + option("m-unknown", "llama-cpp", 0), + base("m-base-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(2)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base-q4")) + Expect(selection.Option.IsBase).To(BeTrue()) + }) + + It("selects the base when the base is the largest option that fits", func() { + // A Q8 base offering a Q4 downgrade for small hosts is a natural + // authoring shape. Treating the base as a last resort would install + // the Q4 on every host large enough for the Q8 and permanently + // downgrade the user. + options := []gallery.VariantOption{ + option("m-q4", "llama-cpp", gib(4)), + base("m-base-q8", gib(8)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base-q8")) + // The Q4 survived every filter, so this is the base winning on rank + // and not the base being fallen back to. + Expect(selection.FellBackToBase).To(BeFalse()) + Expect(selection.Reasons).To(BeEmpty()) + }) + + It("selects a smaller variant when the base does not fit but the variant does", func() { + // The mirror of the spec above: the base competes, it does not win by + // default, so a host that cannot hold it must still take the downgrade + // the entry offers for exactly that case. + options := []gallery.VariantOption{ + option("m-q4", "llama-cpp", gib(4)), + base("m-base-q8", gib(8)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(6)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q4")) + }) + It("reports why a probed size that does not fit was rejected", func() { // Ranking and filtering both run off the probe, so a rejection has to // be traceable back to the figure the probe returned. @@ -213,6 +268,21 @@ var _ = Describe("SelectVariant", func() { selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: 0}, "") Expect(err).ToNot(HaveOccurred()) Expect(selection.Option.Variant.Model).To(Equal("m-base")) + Expect(selection.Option.IsBase).To(BeTrue()) + Expect(selection.FellBackToBase).To(BeTrue()) + }) + + It("prefers the base to an unsized variant even when the base itself is unsized", func() { + // Neither can be shown to fit, so nothing separates them on size. The + // base is still the payload the entry is guaranteed to install. + options := []gallery.VariantOption{ + option("m-unknown", "llama-cpp", 0), + base("m-base", 0), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(8)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) }) It("explains why each variant was rejected", func() { diff --git a/core/gallery/variants_install_test.go b/core/gallery/variants_install_test.go index e67faaecf..900839c52 100644 --- a/core/gallery/variants_install_test.go +++ b/core/gallery/variants_install_test.go @@ -2,6 +2,7 @@ package gallery_test import ( "context" + "fmt" "os" "path/filepath" "runtime" @@ -168,15 +169,17 @@ var _ = Describe("ResolveVariant", func() { Expect(resolved.URL).To(Equal("file://gguf.yaml")) }) - It("completes the install when the probe cannot determine a size", func() { - // A failed probe reports 0. That is an unknown, so the variant is - // not filtered out, and above all the resolution does not fail: a - // network hiccup must never be able to break an install. + It("completes the install on the entry's own payload when no probe answers", func() { + // A failed probe reports 0. That is an unknown, so nothing is filtered + // out and above all the resolution does not fail: a network hiccup + // must never be able to break an install. It must not be able to + // change what gets installed either, so an unmeasurable variant does + // not displace the payload the entry itself ships. resolved, variant, err := gallery.ResolveVariant(models, base, probing(gib(24), map[string]uint64{}), "") Expect(err).ToNot(HaveOccurred()) - Expect(variant.Model).To(Equal("qwen3-8b-vllm-awq")) - Expect(resolved.URL).To(Equal("file://vllm.yaml")) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) }) It("still installs the base when every probe fails and nothing else survives", func() { @@ -317,16 +320,27 @@ var _ = Describe("InstallModelFromGallery with variant entries", func() { var tempdir string var galleries []config.Gallery var systemState *system.SystemState + // galleryRevision keeps every gallery this suite writes distinct from every + // other, so the process-wide gallery cache cannot serve one spec's catalog + // to another. + galleryRevision := 0 // Every entry is described with an inline config_file rather than a URL so // the whole install runs off the local filesystem with no network access. + // + // Each call writes a fresh path and a fresh gallery name because the gallery + // listing is cached on the name and URL pair. A spec that edits its gallery + // mid-flight would otherwise keep reading the version it started with, and + // would silently prove nothing. newGallery := func(entries ...gallery.GalleryModel) { out, err := yaml.Marshal(entries) Expect(err).ToNot(HaveOccurred()) - galleryPath := filepath.Join(tempdir, "gallery.yaml") + name := fmt.Sprintf("test-%d", galleryRevision) + galleryRevision++ + galleryPath := filepath.Join(tempdir, name+".yaml") Expect(os.WriteFile(galleryPath, out, 0600)).To(Succeed()) - galleries = []config.Gallery{{Name: "test", URL: "file://" + galleryPath}} + galleries = []config.Gallery{{Name: name, URL: "file://" + galleryPath}} } entry := func(name, backend string) gallery.GalleryModel { @@ -352,7 +366,7 @@ var _ = Describe("InstallModelFromGallery with variant entries", func() { // entry's own, so only the entry-name overlay can keep the record under the // name the user asked for. The inline config_file branch seeds the name from // the already-renamed resolved entry and so cannot observe the overlay. - urlEntry := func(name, backend string) gallery.GalleryModel { + urlEntry := func(name, backend, size string) gallery.GalleryModel { payload := gallery.ModelConfig{ Name: name, Description: "entry " + name, @@ -367,6 +381,7 @@ var _ = Describe("InstallModelFromGallery with variant entries", func() { m.Name = name m.Description = "entry " + name m.URL = "file://" + payloadPath + m.Size = size return m } @@ -484,10 +499,13 @@ var _ = Describe("InstallModelFromGallery with variant entries", func() { // Without the entry-name overlay the record persists as "qwen3-8b-q8", // and the stable name the entry exists to provide is lost the moment // anything reads the record back. + // The declared sizes make the upgrade the largest build that fits, so + // resolution lands on the variant and the record it writes is the one + // under test here. newGallery( - withVariants(urlEntry("qwen3-8b-q4", "base-backend"), + withVariants(urlEntry("qwen3-8b-q4", "base-backend", "16MiB"), gallery.Variant{Model: "qwen3-8b-q8"}), - urlEntry("qwen3-8b-q8", "upgrade-backend"), + urlEntry("qwen3-8b-q8", "upgrade-backend", "256MiB"), ) Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) @@ -533,10 +551,14 @@ var _ = Describe("InstallModelFromGallery with variant entries", func() { }) It("records a pin and honors it on a plain reinstall", func() { + // The upgrade declares a size and the base does not, so auto-selection + // genuinely prefers the upgrade here. Without that the second install + // below would land on the base whether or not the pin was recalled, and + // the spec would prove nothing. newGallery( withVariants(entry("qwen3-8b-q4", "base-backend"), gallery.Variant{Model: "qwen3-8b-q8"}), - entry("qwen3-8b-q8", "upgrade-backend"), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), ) Expect(install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q4"))).To(Succeed()) @@ -558,10 +580,12 @@ var _ = Describe("InstallModelFromGallery with variant entries", func() { }) It("honors a pin recorded under a custom install name", func() { + // Sized as above so auto-selection would take the upgrade, which is what + // makes the recall observable in the second install. newGallery( withVariants(entry("qwen3-8b-q4", "base-backend"), gallery.Variant{Model: "qwen3-8b-q8"}), - entry("qwen3-8b-q8", "upgrade-backend"), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), ) req := gallery.GalleryModel{} @@ -581,6 +605,57 @@ var _ = Describe("InstallModelFromGallery with variant entries", func() { Expect(installedBackend("prod-llm")).To(Equal("base-backend")) }) + It("re-selects automatically when a recorded pin no longer exists in the gallery", func() { + // A pin is recorded, then the gallery is edited to rename the build it + // named. Without dropping the stale pin every later reinstall and every + // upgrade of this model fails forever, and the only remedy is deleting a + // dotfile the user has never heard of. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + Expect(install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q8"))).To(Succeed()) + + record, err := gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(Equal("qwen3-8b-q8")) + + // The gallery edit: the same build under a new name. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8-v2"}), + sizedEntry("qwen3-8b-q8-v2", "renamed-backend", "16MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + // Auto-selection ran: the renamed build is the largest that fits. + Expect(installedBackend("qwen3-8b-q4")).To(Equal("renamed-backend")) + + // The stale pin is cleared rather than carried forward, so the next + // install does not have to rediscover that it is unusable. + record, err = gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(BeEmpty()) + Expect(record.ResolvedVariant).To(Equal("qwen3-8b-q8-v2")) + }) + + It("still refuses a variant the caller names on this request, recorded pin or not", func() { + // The caller-supplied pin and the recalled one must fail differently. + // This one is a request that cannot be honored, so reporting success + // for it would make a typo indistinguishable from a working choice. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + Expect(install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q8"))).To(Succeed()) + + err := install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q6")) + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("qwen3-8b-q6")) + }) + It("writes each declared url only once into the persisted gallery file", func() { base := withVariants(entry("qwen3-8b-q4", "base-backend"), gallery.Variant{Model: "qwen3-8b-q8"}) diff --git a/core/services/galleryop/model_variant_test.go b/core/services/galleryop/model_variant_test.go index 689a2ecc4..abbe903ac 100644 --- a/core/services/galleryop/model_variant_test.go +++ b/core/services/galleryop/model_variant_test.go @@ -62,13 +62,16 @@ var _ = Describe("LocalModelManager variant selection", func() { Expect(err).ToNot(HaveOccurred()) DeferCleanup(func() { Expect(os.RemoveAll(modelsDir)).To(Succeed()) }) - // The upgrade declares no size and carries no weight files, so the probe - // reports an unknown, the variant survives the filter and auto-selection - // would take it on any machine. Only an honored pin can land the install - // on the base. + // The upgrade declares a size no machine fails to clear and the base + // declares none, so the upgrade is the only measured fit and + // auto-selection takes it everywhere. Only an honored pin can land the + // install on the base. The size is read from the entry itself, so + // nothing here reaches the network. + upgrade := entry("qwen3-8b-q8", "upgrade-backend") + upgrade.Size = "16MiB" entries := []gallery.GalleryModel{ entry("qwen3-8b-q4", "base-backend", gallery.Variant{Model: "qwen3-8b-q8"}), - entry("qwen3-8b-q8", "upgrade-backend"), + upgrade, } out, err := yaml.Marshal(entries) Expect(err).ToNot(HaveOccurred()) diff --git a/docs/content/features/model-gallery.md b/docs/content/features/model-gallery.md index 68b08a0cf..1c66967f2 100644 --- a/docs/content/features/model-gallery.md +++ b/docs/content/features/model-gallery.md @@ -160,10 +160,19 @@ carries a `variants` list, and installing it normally lets LocalAI choose: - variants whose backend cannot run on this machine are dropped; - variants that do not fit the available memory (VRAM on a GPU host, otherwise system RAM) are dropped; -- the largest remaining variant wins, because a bigger footprint means a higher +- the entry's own build is never dropped. It competes with whatever survived + rather than waiting for everything else to fail, so an entry that is itself + the largest build that fits keeps its own payload; +- the largest remaining build wins, because a bigger footprint means a higher quality build of the same model; -- if nothing survives, the entry's own build is installed. The entry is always - installable, on any machine. +- a build whose size could not be measured ranks below the entry's own build, + so an unreadable size never quietly displaces the payload the entry ships; +- if nothing else survives, the entry's own build is installed. The entry is + always installable, on any machine. + +Because the entry's own build competes on size like every other candidate, the +order of the list means nothing and a `variants` list may offer smaller builds, +larger ones, or both. Sizes are measured from the model's weights rather than downloaded, and cached. @@ -181,15 +190,16 @@ curl http://localhost:8080/api/models | jq '.models[] | select(.variants) | "auto_variant": "nanbeige4.1-3b-q8", "variants": [ { "model": "nanbeige4.1-3b-q8", "backend": "llama-cpp", "memory_bytes": 4187593113, "fits": true, "is_base": false }, - { "model": "nanbeige4.1-3b-q4", "backend": "llama-cpp", "memory_bytes": 0, "fits": true, "is_base": true } + { "model": "nanbeige4.1-3b-q4", "backend": "llama-cpp", "fits": true, "is_base": true } ] } ``` `auto_variant` is what installing without a choice would pick right now. `fits` -is whether auto-selection would consider that variant on this machine, and a -`memory_bytes` of 0 means the size could not be determined, not that the build -is free. `is_base` marks the entry's own build. +is whether auto-selection would consider that variant on this machine, and +`is_base` marks the entry's own build. `memory_bytes` is omitted entirely, as on +the second entry above, when the size could not be measured; read a missing +`memory_bytes` as unknown rather than as a free build. To install a specific one, pass its name as `variant`: diff --git a/pkg/mcp/localaitools/prompts/skills/install_chat_model.md b/pkg/mcp/localaitools/prompts/skills/install_chat_model.md index 54d2569d1..08fc4822b 100644 --- a/pkg/mcp/localaitools/prompts/skills/install_chat_model.md +++ b/pkg/mcp/localaitools/prompts/skills/install_chat_model.md @@ -6,7 +6,7 @@ Use this when the user wants to install a chat-capable model — including the c 2. Show the top results as a numbered list with name, gallery, short description, and license. If none match, say so and ask whether to broaden the search. 3. Wait for the user to pick. 4. Summarise the chosen install ("I'll install **`/`** — confirm?") and wait for confirmation. -5. On confirmation, call `install_model` with `gallery_name` and `model_name` from the chosen hit. Leave `variant` empty: LocalAI then auto-selects the largest build this machine's engines and free memory can actually run. Only set `variant` when the user names a specific build (e.g. "the Q8 one"), and pass the name exactly as the entry lists it — an unknown name fails the install rather than falling back to auto-selection. +5. On confirmation, call `install_model` with `gallery_name` and `model_name` from the chosen hit. Leave `variant` empty: LocalAI then auto-selects the largest build this machine's engines and free memory can actually run. Only set `variant` when the user names a specific build (e.g. "the Q8 one"), and pass the name exactly as the entry lists it: an unknown name fails the install rather than falling back to auto-selection. 6. Poll `get_job_status` with the returned job id. Report meaningful progress changes (every ~10–20%, plus completion). 7. When the job reports `processed: true` and no error, call `reload_models`, then `list_installed_models` with `capability: "chat"` to confirm the model is now visible. 8. Tell the user the model is ready and how to use it (its name as the `model` field in chat completions). diff --git a/pkg/system/capabilities.go b/pkg/system/capabilities.go index 9d0d3c3c4..0c8fc25ab 100644 --- a/pkg/system/capabilities.go +++ b/pkg/system/capabilities.go @@ -212,8 +212,10 @@ func (s *SystemState) BackendPreferenceTokens() []string { // // Why this exists alongside Capability: Capability resolves against a caller // supplied map and falls back to "default" then "cpu" when the detected value -// is absent from that map. Callers that express fallback themselves, such as -// model meta entries ordering their own candidates, need the undecorated value. +// is absent from that map, so its answer describes what that caller can serve +// rather than what the hardware is. A caller reasoning about the hardware +// itself, or reporting it to a human, cannot tell a genuinely detected +// "default" apart from a substituted one and needs the undecorated value. func (s *SystemState) DetectedCapability() string { return s.getSystemCapabilities() }