From 77d01de6ffd2ece1bcdab96719385001ec12c3a9 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 18 Jul 2026 23:06:17 +0000 Subject: [PATCH] feat(gallery): expose model variants for selection over API, CLI and MCP A gallery entry may carry `variants:`, alternative builds of the same model. Selection already worked at install time, but nothing could see what an entry offered or ask for a specific build, so the feature was undrivable. Listing: `GET /api/models` now reports `variants` and `auto_variant` for the entries that declare variants. Each variant carries its resolved backend, its measured size and whether it fits this host. `auto_variant` is what installing without a choice would pick right now. The new gallery.DescribeVariants runs the same variantOptions + SelectVariant pass the installer runs, so the reported default cannot drift from what installing actually does, and HostResolveEnv is extracted so both derive the host and share pkg/vram's probe cache from one place. Performance: an entry that declares no variants returns early without touching the probe, so the ~1280 ordinary entries cost exactly what they cost before. Selection: `variant` is accepted on POST /models/apply, as a query param on POST /api/models/install/:id, on the gallery apply file/string request, as `local-ai models install --variant`, and as a parameter on the install_model MCP tool (both the httpapi and inproc clients). Empty means auto-select. An unknown variant name now fails the install naming what was requested. This closes a real hole: an entry declaring no variants short-circuits before selection runs, so a requested variant was previously dropped silently and the install reported success. startup.InstallModels ends in a variadic model list, so install options could not be appended to it; InstallModelsWithOptions is added alongside and InstallModels delegates to it. No caller signature changed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto --- .agents/adding-gallery-models.md | 40 +++ core/cli/models.go | 7 +- core/gallery/describe_variants.go | 85 ++++++ core/gallery/describe_variants_test.go | 247 ++++++++++++++++++ core/gallery/models.go | 48 ++-- core/gallery/variants_install_test.go | 30 +++ core/http/endpoints/localai/gallery.go | 6 + core/http/routes/ui_api.go | 32 ++- core/services/galleryop/managers_local.go | 6 +- core/services/galleryop/model_variant_test.go | 102 ++++++++ core/services/galleryop/models.go | 13 +- core/services/galleryop/operation.go | 9 + core/startup/model_preload.go | 11 +- docs/content/features/model-gallery.md | 64 +++++ pkg/mcp/localaitools/dto.go | 1 + pkg/mcp/localaitools/httpapi/client.go | 3 + pkg/mcp/localaitools/httpapi/client_test.go | 33 +++ pkg/mcp/localaitools/inproc/client.go | 1 + .../prompts/skills/install_chat_model.md | 2 +- pkg/mcp/localaitools/tools_models.go | 2 +- 20 files changed, 719 insertions(+), 23 deletions(-) create mode 100644 core/gallery/describe_variants.go create mode 100644 core/gallery/describe_variants_test.go create mode 100644 core/services/galleryop/model_variant_test.go diff --git a/.agents/adding-gallery-models.md b/.agents/adding-gallery-models.md index fe5a57e87..9e44b0a11 100644 --- a/.agents/adding-gallery-models.md +++ b/.agents/adding-gallery-models.md @@ -91,6 +91,46 @@ To add a variant (e.g., different quantization), use YAML merge: uri: huggingface:////-Q8_0.gguf ``` +## Offering several builds of one model (`variants`) + +When the same model is published in more than one quantization, or is also +servable by another engine, add each build as its own ordinary gallery entry and +then point one of them at the others with `variants`: + +```yaml +- !!merge <<: *chatml + name: "nanbeige4.1-3b-q4" + # ... the usual urls / overrides / files for the Q4 build ... + variants: + - model: nanbeige4.1-3b-q8 +``` + +Rules: + +- The declaring entry is a **complete, normal entry**. It keeps its own + `files`/`overrides` and stays installable on every host and by every older + LocalAI release, which simply ignore `variants`. +- A variant references another gallery entry **by name**. That entry must exist + and must not declare `variants` of its own. +- **Order carries no meaning.** Do not try to encode a preference; write the + list in whatever order reads best. +- **Do not describe hardware.** At install time LocalAI drops variants whose + backend cannot run on the host, drops those that do not fit available memory, + and installs the largest survivor, falling back to the declaring entry's own + build. Sizes are measured live from the weights and cached, so nothing has to + be written down. +- `min_memory` on a single variant (e.g. `min_memory: 20GiB`) overrides the + measured size and suppresses the measurement for that variant. Use it only + when you have measured a real load and know the estimate is wrong; most + variants should not carry it. + +Users can override the automatic choice with `variant` on `POST /models/apply`, +`local-ai models install --variant`, or the `install_model` MCP tool. See +`docs/content/features/model-gallery.md`. + +The gallery lint specs live in `core/gallery`, so run that suite after adding a +`variants` list. + ## Available template configs Look at existing `.yaml` files in `gallery/` to find the right prompt template for your model architecture: diff --git a/core/cli/models.go b/core/cli/models.go index a95462da1..08bbf9847 100644 --- a/core/cli/models.go +++ b/core/cli/models.go @@ -39,6 +39,7 @@ type ModelsInstall struct { RequireBackendIntegrity bool `env:"LOCALAI_REQUIRE_BACKEND_INTEGRITY,REQUIRE_BACKEND_INTEGRITY" help:"If true, reject backend installs without a configured signature verification policy (OCI URIs) or SHA256 (tarball/HTTP URIs)." group:"hardening" default:"false"` AutoloadBackendGalleries bool `env:"LOCALAI_AUTOLOAD_BACKEND_GALLERIES" help:"If true, automatically loads backend galleries" group:"backends" default:"true"` ModelArgs []string `arg:"" optional:"" name:"models" help:"Model configuration URLs to load"` + Variant string `name:"variant" help:"Install a specific variant of a gallery entry that declares them, by the variant's model name. Leave unset to let LocalAI auto-select the largest build this machine can run." group:"models"` ModelsCMDFlags `embed:""` } @@ -147,7 +148,11 @@ func (mi *ModelsInstall) Run(ctx *cliContext.Context) error { } modelLoader := model.NewModelLoader(systemState) - err = startup.InstallModels(context.Background(), galleryService, galleries, backendGalleries, systemState, modelLoader, !mi.DisablePredownloadScan, mi.AutoloadBackendGalleries, mi.RequireBackendIntegrity, progressCallback, modelName) + var installOptions []gallery.InstallOption + if mi.Variant != "" { + installOptions = append(installOptions, gallery.WithVariant(mi.Variant)) + } + err = startup.InstallModelsWithOptions(context.Background(), galleryService, galleries, backendGalleries, systemState, modelLoader, !mi.DisablePredownloadScan, mi.AutoloadBackendGalleries, mi.RequireBackendIntegrity, progressCallback, installOptions, modelName) if err != nil { return err } diff --git a/core/gallery/describe_variants.go b/core/gallery/describe_variants.go new file mode 100644 index 000000000..a44864e00 --- /dev/null +++ b/core/gallery/describe_variants.go @@ -0,0 +1,85 @@ +package gallery + +// VariantView is one selectable build of an entry, as a client sees it. +// +// It is the read-only mirror of what selection decides at install time, so a +// picker can show the user the same facts the installer acts on rather than a +// second, independently-derived opinion that could disagree with it. +type VariantView struct { + // Model is the name to send back as the install request's `variant`. + Model string `json:"model"` + // Backend is the engine the referenced entry resolves to. A client renders + // it, and it is also the reason Fits may be false on a host whose memory + // would be ample. + Backend string `json:"backend"` + // MemoryBytes is the measured footprint, or 0 when it could not be + // determined. Zero is unknown, never "needs nothing", so a client must + // render it as unknown rather than as a free option. + MemoryBytes uint64 `json:"memory_bytes,omitempty"` + // Fits reports whether auto-selection would consider this variant on this + // host: its backend can run here and its known footprint is within budget. + // An unknown footprint counts as fitting, exactly as selection treats it. + Fits bool `json:"fits"` + // IsBase marks the entry's own payload. It is always installable and is + // what auto-selection falls back to, which is why it is listed alongside + // the declared variants rather than hidden. + IsBase bool `json:"is_base"` +} + +// EntryVariants is the variant surface of a single gallery entry. +type EntryVariants struct { + Variants []VariantView `json:"variants"` + // AutoSelected is the variant auto-selection would install on this host + // right now. Clients show it as the default choice. + AutoSelected string `json:"auto_selected"` +} + +// DescribeVariants reports an entry's selectable builds and which one +// auto-selection would currently pick. +// +// It runs the SAME variantOptions + SelectVariant pass the installer runs, so +// the reported auto-selection cannot drift from what installing would actually +// do. The env carries the same probe seam too, so the size shown here and the +// size the installer compares against come from one cache and one round trip. +// +// An entry that declares no variants returns nil, nil WITHOUT touching the +// probe. That is load-bearing: the gallery listing walks entries by the +// thousand and the overwhelming majority declare nothing, so the no-variant +// case must cost nothing at all. +func DescribeVariants(models []*GalleryModel, entry *GalleryModel, env ResolveEnv) (*EntryVariants, error) { + if entry == nil || !entry.HasVariants() { + return nil, nil + } + + options, err := variantOptions(models, entry, env) + if err != nil { + return nil, err + } + + selection, err := SelectVariant(options, env, "") + if err != nil { + return nil, err + } + + views := make([]VariantView, 0, len(options)) + for _, o := range options { + memory, known, err := o.EffectiveMemory() + if err != nil { + return nil, err + } + view := VariantView{ + Model: o.Variant.Model, + Backend: o.Backend, + // The base is exempt from every filter and always installs, so + // reporting it as anything but fitting would misdescribe it. + Fits: o.IsBase || (env.backendRuns(o.Backend) && (!known || memory <= env.AvailableMemory)), + IsBase: o.IsBase, + } + if known { + view.MemoryBytes = memory + } + views = append(views, view) + } + + return &EntryVariants{Variants: views, AutoSelected: selection.Option.Variant.Model}, nil +} diff --git a/core/gallery/describe_variants_test.go b/core/gallery/describe_variants_test.go new file mode 100644 index 000000000..4831aff15 --- /dev/null +++ b/core/gallery/describe_variants_test.go @@ -0,0 +1,247 @@ +package gallery_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/gallery" +) + +var _ = Describe("DescribeVariants", func() { + gib := func(n uint64) uint64 { return n * 1024 * 1024 * 1024 } + + newModel := func(name, backend string) *gallery.GalleryModel { + m := &gallery.GalleryModel{} + m.Name = name + m.Backend = backend + return m + } + + var models []*gallery.GalleryModel + var base *gallery.GalleryModel + // probed records every entry the probe was asked about, so a spec can + // assert on the ABSENCE of a round trip and not merely on the result. + var probed []string + + // probing builds an env whose sizes are injected rather than measured, so + // these specs pin exact footprints without reaching the network. + probing := func(available uint64, sizes map[string]uint64) gallery.ResolveEnv { + return gallery.ResolveEnv{ + AvailableMemory: available, + BackendCompatible: func(string) bool { return true }, + ProbeMemory: func(target *gallery.GalleryModel) uint64 { + probed = append(probed, target.Name) + return sizes[target.Name] + }, + } + } + + byName := func(view *gallery.EntryVariants, name string) gallery.VariantView { + GinkgoHelper() + for _, v := range view.Variants { + if v.Model == name { + return v + } + } + Fail("no variant named " + name + " in the reported view") + return gallery.VariantView{} + } + + BeforeEach(func() { + probed = nil + big := newModel("qwen3-8b-vllm-awq", "vllm") + small := newModel("qwen3-8b-gguf-q8", "llama-cpp") + base = newModel("qwen3-8b-gguf-q4", "llama-cpp") + base.Variants = []gallery.Variant{ + {Model: "qwen3-8b-vllm-awq"}, + {Model: "qwen3-8b-gguf-q8"}, + } + models = []*gallery.GalleryModel{big, small, base} + }) + + Describe("an entry that declares no variants", func() { + It("reports nothing and issues no probe at all", func() { + // This is the performance contract of the gallery listing: the + // listing walks over a thousand entries and virtually none of them + // declare variants, so the ordinary entry must cost zero round + // trips. Asserting on `probed` rather than on timing makes that + // contract testable. + plain := newModel("plain-entry", "llama-cpp") + models = append(models, plain) + + view, err := gallery.DescribeVariants(models, plain, probing(gib(24), map[string]uint64{ + "plain-entry": gib(4), + })) + + Expect(err).ToNot(HaveOccurred()) + Expect(view).To(BeNil()) + Expect(probed).To(BeEmpty()) + }) + + It("reports nothing for a nil entry rather than panicking", func() { + view, err := gallery.DescribeVariants(models, nil, probing(gib(24), nil)) + Expect(err).ToNot(HaveOccurred()) + Expect(view).To(BeNil()) + Expect(probed).To(BeEmpty()) + }) + }) + + Describe("an entry that declares variants", func() { + It("reports every declared variant plus the entry's own build", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(24), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + + names := []string{} + for _, v := range view.Variants { + names = append(names, v.Model) + } + // The base is listed so a picker can offer "decline the upgrade", + // which is a real choice an operator makes. + Expect(names).To(ConsistOf("qwen3-8b-vllm-awq", "qwen3-8b-gguf-q8", "qwen3-8b-gguf-q4")) + Expect(byName(view, "qwen3-8b-gguf-q4").IsBase).To(BeTrue()) + Expect(byName(view, "qwen3-8b-vllm-awq").IsBase).To(BeFalse()) + }) + + It("reports the backend of the referenced entry, not of the declaring one", func() { + // A picker renders this, and it is also the reason a variant can be + // unavailable on a host with ample memory. + view, err := gallery.DescribeVariants(models, base, probing(gib(24), nil)) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Backend).To(Equal("vllm")) + Expect(byName(view, "qwen3-8b-gguf-q8").Backend).To(Equal("llama-cpp")) + }) + + It("reports the probed size of each variant", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(24), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").MemoryBytes).To(Equal(gib(20))) + Expect(byName(view, "qwen3-8b-gguf-q8").MemoryBytes).To(Equal(gib(9))) + }) + + It("lets an authored min_memory win over the probe and skips that probe", func() { + base.Variants = []gallery.Variant{{Model: "qwen3-8b-vllm-awq", MinMemory: "20GiB"}} + + view, err := gallery.DescribeVariants(models, base, probing(gib(24), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(2), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").MemoryBytes).To(Equal(gib(20))) + Expect(probed).ToNot(ContainElement("qwen3-8b-vllm-awq")) + }) + + It("reports a size it could not determine as unknown rather than as free", func() { + // Zero on the wire means unknown. Rendering it as a zero-byte model + // would advertise the largest download on offer as costless. + view, err := gallery.DescribeVariants(models, base, probing(gib(24), map[string]uint64{})) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").MemoryBytes).To(Equal(uint64(0))) + Expect(byName(view, "qwen3-8b-vllm-awq").Fits).To(BeTrue()) + }) + + It("marks a variant too large for this host as not fitting", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(10), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Fits).To(BeFalse()) + Expect(byName(view, "qwen3-8b-gguf-q8").Fits).To(BeTrue()) + }) + + It("marks a variant whose backend cannot run here as not fitting, however much memory there is", func() { + env := probing(gib(1024), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + }) + env.BackendCompatible = func(backend string) bool { return backend != "vllm" } + + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Fits).To(BeFalse()) + Expect(byName(view, "qwen3-8b-gguf-q8").Fits).To(BeTrue()) + }) + + It("always reports the entry's own build as fitting", func() { + // The base is exempt from every filter and always installs, so + // showing it as unavailable would misdescribe it. Neither a hostile + // backend gate nor a zero memory budget may change that. + env := probing(0, map[string]uint64{}) + env.BackendCompatible = func(string) bool { return false } + + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-gguf-q4").Fits).To(BeTrue()) + }) + + It("surfaces a variant that references an entry no gallery declares", func() { + base.Variants = []gallery.Variant{{Model: "does-not-exist"}} + _, err := gallery.DescribeVariants(models, base, probing(gib(24), nil)) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("does-not-exist")) + }) + }) + + Describe("the reported auto-selection", func() { + // These are the specs that keep a picker honest: what the listing shows + // as the default must be what installing with no variant actually does. + // They assert against ResolveVariant rather than against a hardcoded + // name, so a change to the selection rules cannot make the two drift + // without failing here. + agreesWithInstall := func(env gallery.ResolveEnv) { + GinkgoHelper() + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + + _, chosen, err := gallery.ResolveVariant(models, base, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal(chosen.Model)) + } + + It("matches what installing would pick when everything fits", func() { + agreesWithInstall(probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + }) + + It("matches what installing would pick when only the smaller variant fits", func() { + agreesWithInstall(probing(gib(12), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + }) + + It("matches what installing would pick when nothing fits and the base wins", func() { + agreesWithInstall(probing(gib(2), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + }) + + It("names the largest fitting variant, not the first authored", func() { + // Pinned separately from agreesWithInstall so that a mutation making + // BOTH functions pick the wrong variant still fails a spec. + view, err := gallery.DescribeVariants(models, base, probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal("qwen3-8b-vllm-awq")) + }) + + It("names the entry itself when no variant fits", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(2), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal("qwen3-8b-gguf-q4")) + }) + }) +}) diff --git a/core/gallery/models.go b/core/gallery/models.go index 1792d415d..9cc492125 100644 --- a/core/gallery/models.go +++ b/core/gallery/models.go @@ -252,6 +252,31 @@ func ResolveVariant(models []*GalleryModel, entry *GalleryModel, env ResolveEnv, return &resolved, selected.Variant, nil } +// HostResolveEnv describes this machine to variant selection. +// +// It exists so the install path and every read-only surface that reports what +// selection WOULD choose derive the host from one place. Two copies of this +// wiring would eventually disagree, and a picker that shows a different answer +// than the installer produces is worse than no picker at all. +func HostResolveEnv(ctx context.Context, systemState *system.SystemState) ResolveEnv { + return ResolveEnv{ + AvailableMemory: availableModelMemory(systemState), + // The whole hardware gate. IsBackendCompatible already derives + // Darwin-only, NVIDIA-only, ROCm-only and SYCL-only from the backend + // name, so a gallery author never has to describe hardware. + // + // The uri argument is deliberately empty: it exists for backend OCI + // images, and passing a model's gallery url here would let an unrelated + // substring in a download link decide hardware compatibility. + BackendCompatible: func(backend string) bool { + return systemState.IsBackendCompatible(backend, "") + }, + ProbeMemory: func(target *GalleryModel) uint64 { + return probeEntryMemory(ctx, target) + }, + } +} + // noGPUDetected is what SystemState.DetectedCapability() reports when no // usable GPU was found. The constant is unexported in pkg/system. const noGPUDetected = "default" @@ -419,6 +444,12 @@ func InstallModelFromGallery( // still installable as-is; selection below only decides whether one of its // declared alternatives suits this host better. if !model.HasVariants() { + // A caller who named a variant asked for something this entry cannot + // give. Installing the entry anyway would look like success while + // quietly ignoring the choice, so it is refused by name instead. + if installOpts.variant != "" { + return fmt.Errorf("%w: %q was requested but model %q declares no variants", ErrPinNotFound, installOpts.variant, model.Name) + } return applyModel(model, nil) } @@ -443,22 +474,7 @@ func InstallModelFromGallery( } } - env := ResolveEnv{ - AvailableMemory: availableModelMemory(systemState), - // The whole hardware gate. IsBackendCompatible already derives - // Darwin-only, NVIDIA-only, ROCm-only and SYCL-only from the backend - // name, so a gallery author never has to describe hardware. - // - // The uri argument is deliberately empty: it exists for backend OCI - // images, and passing a model's gallery url here would let an unrelated - // substring in a download link decide hardware compatibility. - BackendCompatible: func(backend string) bool { - return systemState.IsBackendCompatible(backend, "") - }, - ProbeMemory: func(target *GalleryModel) uint64 { - return probeEntryMemory(ctx, target) - }, - } + env := HostResolveEnv(ctx, systemState) resolved, variant, err := ResolveVariant(models, model, env, pin) if err != nil { diff --git a/core/gallery/variants_install_test.go b/core/gallery/variants_install_test.go index 424bbfd81..8bf275f33 100644 --- a/core/gallery/variants_install_test.go +++ b/core/gallery/variants_install_test.go @@ -505,6 +505,36 @@ var _ = Describe("InstallModelFromGallery with variant entries", func() { Expect(installedBackend("qwen3-8b-q4")).To(Equal("upgrade-backend")) }) + It("refuses a variant name the entry does not declare, naming what was asked for", func() { + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8", MinMemory: "0GiB"}), + entry("qwen3-8b-q8", "upgrade-backend"), + ) + + err := install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q6")) + + // Auto-selecting instead would report success while silently + // installing something other than what the caller asked for, and a + // typo'd variant name would be indistinguishable from a working one. + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("qwen3-8b-q6")) + Expect(filepath.Join(tempdir, "qwen3-8b-q4.yaml")).ToNot(BeAnExistingFile()) + }) + + It("refuses a variant name when the entry declares no variants at all", func() { + // The entry is installable and selection never runs for it, so without + // an explicit refusal the requested variant would be dropped on the + // floor and the install would look like it honored the choice. + newGallery(entry("qwen3-8b-q4", "base-backend")) + + err := install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q8")) + + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("qwen3-8b-q8")) + Expect(filepath.Join(tempdir, "qwen3-8b-q4.yaml")).ToNot(BeAnExistingFile()) + }) + It("records a pin and honors it on a plain reinstall", func() { newGallery( withVariants(entry("qwen3-8b-q4", "base-backend"), diff --git a/core/http/endpoints/localai/gallery.go b/core/http/endpoints/localai/gallery.go index fabae336d..fb0a77bb4 100644 --- a/core/http/endpoints/localai/gallery.go +++ b/core/http/endpoints/localai/gallery.go @@ -25,6 +25,11 @@ type ModelGalleryEndpointService struct { type GalleryModel struct { ID string `json:"id"` + // Variant installs one specific build of an entry that declares variants, + // named as it appears in the entry's `variants` list (see the `variants` + // and `auto_variant` fields of the gallery listing). Leave it empty to let + // LocalAI auto-select the largest build this host can actually run. + Variant string `json:"variant,omitempty"` gallery.GalleryModel } @@ -86,6 +91,7 @@ func (mgs *ModelGalleryEndpointService) ApplyModelGalleryEndpoint() echo.Handler Req: input.GalleryModel, ID: uuid.String(), GalleryElementName: input.ID, + Variant: input.Variant, Galleries: mgs.galleries, BackendGalleries: mgs.backendGalleries, } diff --git a/core/http/routes/ui_api.go b/core/http/routes/ui_api.go index e737e445e..b30dd2561 100644 --- a/core/http/routes/ui_api.go +++ b/core/http/routes/ui_api.go @@ -413,6 +413,11 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model }) } + // Kept before the filters and pagination below narrow `models`: a + // variant references another gallery entry by name, and that entry may + // well have been filtered off this page. + allModels := models + // Get all available tags allTags := map[string]struct{}{} tags := []string{} @@ -585,6 +590,26 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model "voice_cloning": m.VoiceCloningCapability(appConfig.SystemState.Model.ModelsPath), } + // Variants are described only for the entries that declare them. + // DescribeVariants is what probes model sizes, so calling it for + // every entry would put a network round trip behind each of the + // ~1200 entries in the gallery. The HasVariants guard keeps an + // ordinary entry exactly as cheap as it was before variants + // existed; only a declaring entry pays, and only on the page it + // appears on. + if m.HasVariants() { + env := gallery.HostResolveEnv(c.Request().Context(), appConfig.SystemState) + view, describeErr := gallery.DescribeVariants(allModels, m, env) + if describeErr != nil { + // A malformed variant list must not blank the whole gallery + // page; that entry just renders without a picker. + xlog.Warn("could not describe model variants", "model", m.Name, "error", describeErr) + } else if view != nil { + obj["variants"] = view.Variants + obj["auto_variant"] = view.AutoSelected + } + } + modelsJSON = append(modelsJSON, obj) } @@ -809,7 +834,11 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model "error": "invalid model ID", }) } - xlog.Debug("API job submitted to install", "galleryID", galleryID) + // Optional: one of the variants the listing reported for this entry. + // Absent means auto-select, which is what the listing's auto_variant + // already told the client would happen. + variant := c.QueryParam("variant") + xlog.Debug("API job submitted to install", "galleryID", galleryID, "variant", variant) id, err := uuid.NewUUID() if err != nil { @@ -825,6 +854,7 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model op := galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{ ID: uid, GalleryElementName: galleryID, + Variant: variant, Galleries: appConfig.Galleries, BackendGalleries: appConfig.BackendGalleries, Context: ctx, diff --git a/core/services/galleryop/managers_local.go b/core/services/galleryop/managers_local.go index 10fb29491..70fc018e0 100644 --- a/core/services/galleryop/managers_local.go +++ b/core/services/galleryop/managers_local.go @@ -62,10 +62,14 @@ func (m *LocalModelManager) InstallModel(ctx context.Context, op *ManagementOp[g } return nil case op.GalleryElementName != "": + opts := []gallery.InstallOption{gallery.WithArtifactMaterializer(m.artifactMaterializer)} + if op.Variant != "" { + opts = append(opts, gallery.WithVariant(op.Variant)) + } return gallery.InstallModelFromGallery(ctx, op.Galleries, op.BackendGalleries, m.systemState, m.modelLoader, op.GalleryElementName, op.Req, progressCb, m.enforcePredownloadScans, m.automaticallyInstallBackend, m.requireBackendIntegrity, - gallery.WithArtifactMaterializer(m.artifactMaterializer)) + opts...) default: return installModelFromRemoteConfig(ctx, m.systemState, m.modelLoader, op.Req, progressCb, m.enforcePredownloadScans, m.automaticallyInstallBackend, op.BackendGalleries, m.requireBackendIntegrity, diff --git a/core/services/galleryop/model_variant_test.go b/core/services/galleryop/model_variant_test.go new file mode 100644 index 000000000..ada429130 --- /dev/null +++ b/core/services/galleryop/model_variant_test.go @@ -0,0 +1,102 @@ +package galleryop_test + +import ( + "context" + "os" + "path/filepath" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/gallery" + "github.com/mudler/LocalAI/core/services/galleryop" + "github.com/mudler/LocalAI/pkg/model" + "github.com/mudler/LocalAI/pkg/system" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" +) + +// A variant chosen in the UI or over REST arrives here as ManagementOp.Variant. +// If it is not turned into a gallery.WithVariant install option the install +// still succeeds, just on the auto-selected build, so the user's choice +// disappears without a single error anywhere. These specs pin the threading. +var _ = Describe("LocalModelManager variant selection", func() { + var ( + modelsDir string + mgr *galleryop.LocalModelManager + appConfig *config.ApplicationConfig + systemState *system.SystemState + ) + + // Every entry carries an inline config_file, so the whole install runs off + // the local filesystem and never reaches the network. + entry := func(name, backend string, variants ...gallery.Variant) gallery.GalleryModel { + m := gallery.GalleryModel{ConfigFile: map[string]any{"backend": backend}} + m.Name = name + m.Description = "entry " + name + m.Variants = variants + return m + } + + installedBackend := func(name string) string { + GinkgoHelper() + dat, err := os.ReadFile(filepath.Join(modelsDir, name+".yaml")) + Expect(err).ToNot(HaveOccurred()) + content := map[string]any{} + Expect(yaml.Unmarshal(dat, &content)).To(Succeed()) + return content["backend"].(string) + } + + install := func(variant string) error { + op := &galleryop.ManagementOp[gallery.GalleryModel, gallery.ModelConfig]{ + GalleryElementName: "qwen3-8b-q4", + Variant: variant, + Galleries: appConfig.Galleries, + } + return mgr.InstallModel(context.Background(), op, func(string, string, string, float64) {}) + } + + BeforeEach(func() { + var err error + modelsDir, err = os.MkdirTemp("", "variant-op-*") + Expect(err).ToNot(HaveOccurred()) + DeferCleanup(func() { Expect(os.RemoveAll(modelsDir)).To(Succeed()) }) + + // "0GiB" always fits, so auto-selection would take the upgrade on any + // machine. Only an honored pin can land the install on the base. + entries := []gallery.GalleryModel{ + entry("qwen3-8b-q4", "base-backend", gallery.Variant{Model: "qwen3-8b-q8", MinMemory: "0GiB"}), + entry("qwen3-8b-q8", "upgrade-backend"), + } + out, err := yaml.Marshal(entries) + Expect(err).ToNot(HaveOccurred()) + galleryPath := filepath.Join(modelsDir, "gallery.yaml") + Expect(os.WriteFile(galleryPath, out, 0o600)).To(Succeed()) + + systemState, err = system.GetSystemState(system.WithModelPath(modelsDir)) + Expect(err).ToNot(HaveOccurred()) + appConfig = &config.ApplicationConfig{ + SystemState: systemState, + Galleries: []config.Gallery{{Name: "test", URL: "file://" + galleryPath}}, + } + mgr = galleryop.NewLocalModelManager(appConfig, model.NewModelLoader(systemState)) + }) + + It("installs the entry's own build when the op pins it", func() { + Expect(install("qwen3-8b-q4")).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("base-backend")) + }) + + It("auto-selects when the op names no variant", func() { + // The mirror of the spec above: same gallery, same host, only the + // pin differs, so nothing but the pin can explain the different build. + Expect(install("")).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("upgrade-backend")) + }) + + It("fails the install when the op names a variant the entry does not declare", func() { + err := install("qwen3-8b-q2") + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("qwen3-8b-q2")) + }) +}) diff --git a/core/services/galleryop/models.go b/core/services/galleryop/models.go index 1c6e57c2a..e6a965b28 100644 --- a/core/services/galleryop/models.go +++ b/core/services/galleryop/models.go @@ -6,6 +6,7 @@ import ( "errors" "fmt" "os" + "slices" "github.com/mudler/LocalAI/core/config" "github.com/mudler/LocalAI/core/gallery" @@ -174,6 +175,9 @@ func installModelFromRemoteConfig(ctx context.Context, systemState *system.Syste type galleryModel struct { gallery.GalleryModel `yaml:",inline"` // https://github.com/go-yaml/yaml/issues/63 ID string `json:"id"` + // Variant pins the install to one of the entry's declared variants. Empty + // means auto-select. + Variant string `json:"variant,omitempty" yaml:"variant,omitempty"` } func processRequests(systemState *system.SystemState, modelLoader *model.ModelLoader, enforceScan, automaticallyInstallBackend bool, galleries []config.Gallery, backendGalleries []config.Gallery, requests []galleryModel, requireBackendIntegrity bool, options ...gallery.InstallOption) error { @@ -185,8 +189,15 @@ func processRequests(systemState *system.SystemState, modelLoader *model.ModelLo err = installModelFromRemoteConfig(ctx, systemState, modelLoader, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend, backendGalleries, requireBackendIntegrity, options...) } else { + // Cloned rather than appended to in place: `options` is shared by + // every request in the batch, so appending would leak one request's + // pin onto the next request that reuses the same backing array. + requestOptions := options + if r.Variant != "" { + requestOptions = append(slices.Clone(options), gallery.WithVariant(r.Variant)) + } err = gallery.InstallModelFromGallery( - ctx, galleries, backendGalleries, systemState, modelLoader, r.ID, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend, requireBackendIntegrity, options...) + ctx, galleries, backendGalleries, systemState, modelLoader, r.ID, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend, requireBackendIntegrity, requestOptions...) } } return err diff --git a/core/services/galleryop/operation.go b/core/services/galleryop/operation.go index ed86f75a8..7eb7cedbc 100644 --- a/core/services/galleryop/operation.go +++ b/core/services/galleryop/operation.go @@ -43,6 +43,15 @@ type ManagementOp[T any, E any] struct { // build on one node without touching the rest of the cluster. TargetNodeID string + // Variant pins a model install to one of the gallery entry's declared + // variants, by that variant's model name. Empty means auto-select: LocalAI + // picks the largest variant this host's backend support and memory can + // actually run, and falls back to the entry's own build. + // + // A name that is not among the entry's variants fails the install rather + // than quietly auto-selecting, so a typo cannot masquerade as a choice. + Variant string + // Upgrade is true if this is an upgrade operation (not a fresh install) Upgrade bool diff --git a/core/startup/model_preload.go b/core/startup/model_preload.go index 3ca375b04..4f3bb1683 100644 --- a/core/startup/model_preload.go +++ b/core/startup/model_preload.go @@ -5,6 +5,7 @@ import ( "encoding/json" "errors" "fmt" + "slices" "time" "github.com/google/uuid" @@ -22,9 +23,17 @@ import ( // It will download the model if it is not already present in the model path // It will also try to resolve if the model is an embedded model YAML configuration func InstallModels(ctx context.Context, galleryService *galleryop.GalleryService, galleries, backendGalleries []config.Gallery, systemState *system.SystemState, modelLoader *model.ModelLoader, enforceScan, autoloadBackendGalleries, requireBackendIntegrity bool, downloadStatus func(string, string, string, float64), models ...string) error { + return InstallModelsWithOptions(ctx, galleryService, galleries, backendGalleries, systemState, modelLoader, enforceScan, autoloadBackendGalleries, requireBackendIntegrity, downloadStatus, nil, models...) +} + +// InstallModelsWithOptions is InstallModels with extra install options, which +// is how the CLI passes a variant pin. It is a separate entry point rather than +// a variadic tail on InstallModels because that signature already ends in a +// variadic model list. +func InstallModelsWithOptions(ctx context.Context, galleryService *galleryop.GalleryService, galleries, backendGalleries []config.Gallery, systemState *system.SystemState, modelLoader *model.ModelLoader, enforceScan, autoloadBackendGalleries, requireBackendIntegrity bool, downloadStatus func(string, string, string, float64), extraOptions []gallery.InstallOption, models ...string) error { // create an error that groups all errors var err error - var installOptions []gallery.InstallOption + installOptions := slices.Clone(extraOptions) if galleryService != nil { installOptions = append(installOptions, gallery.WithArtifactMaterializer(galleryService.ModelArtifactMaterializer())) } diff --git a/docs/content/features/model-gallery.md b/docs/content/features/model-gallery.md index 29f72e682..68b08a0cf 100644 --- a/docs/content/features/model-gallery.md +++ b/docs/content/features/model-gallery.md @@ -151,6 +151,70 @@ where: - `bert-embeddings` is the model name in the gallery (read its [config here](https://github.com/mudler/LocalAI/tree/master/gallery/blob/main/bert-embeddings.yaml)). +### Model variants + +Some gallery entries offer several builds of the same model: different +quantizations, or the same weights served by a different engine. Such an entry +carries a `variants` list, and installing it normally lets LocalAI choose: + +- variants whose backend cannot run on this machine are dropped; +- variants that do not fit the available memory (VRAM on a GPU host, otherwise + system RAM) are dropped; +- the largest remaining variant wins, because a bigger footprint means a higher + quality build of the same model; +- if nothing survives, the entry's own build is installed. The entry is always + installable, on any machine. + +Sizes are measured from the model's weights rather than downloaded, and cached. + +The gallery listing reports what an entry offers. Entries with variants carry +two extra fields, `variants` and `auto_variant`: + +```bash +curl http://localhost:8080/api/models | jq '.models[] | select(.variants) | + {name, auto_variant, variants}' +``` + +```json +{ + "name": "nanbeige4.1-3b-q4", + "auto_variant": "nanbeige4.1-3b-q8", + "variants": [ + { "model": "nanbeige4.1-3b-q8", "backend": "llama-cpp", "memory_bytes": 4187593113, "fits": true, "is_base": false }, + { "model": "nanbeige4.1-3b-q4", "backend": "llama-cpp", "memory_bytes": 0, "fits": true, "is_base": true } + ] +} +``` + +`auto_variant` is what installing without a choice would pick right now. `fits` +is whether auto-selection would consider that variant on this machine, and a +`memory_bytes` of 0 means the size could not be determined, not that the build +is free. `is_base` marks the entry's own build. + +To install a specific one, pass its name as `variant`: + +```bash +curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{ + "id": "localai@nanbeige4.1-3b-q4", + "variant": "nanbeige4.1-3b-q8" + }' +``` + +An explicit choice is honored even when the machine looks too small for it, so +you can deliberately install a build LocalAI would not have picked. A `variant` +the entry does not declare fails the install and names what was requested; it +never quietly falls back to auto-selection. The choice is recorded, so a later +reinstall or upgrade of the same model stays on the variant you picked. + +The same option exists on the CLI: + +```bash +local-ai models install nanbeige4.1-3b-q4 --variant nanbeige4.1-3b-q8 +``` + +Entries without a `variants` list are unaffected by any of this and install +exactly as they always have. + ### Artifact-backed models Gallery models with an `artifacts` declaration are fully materialized during diff --git a/pkg/mcp/localaitools/dto.go b/pkg/mcp/localaitools/dto.go index ecc23204a..ab354b31d 100644 --- a/pkg/mcp/localaitools/dto.go +++ b/pkg/mcp/localaitools/dto.go @@ -67,6 +67,7 @@ type InstallModelRequest struct { GalleryName string `json:"gallery_name,omitempty" jsonschema:"The gallery the model lives in (from gallery_search). Optional when ModelName is unique across galleries."` ModelName string `json:"model_name" jsonschema:"The canonical model name as returned by gallery_search."` Overrides map[string]any `json:"overrides,omitempty" jsonschema:"Optional config overrides to merge into the installed model's YAML."` + Variant string `json:"variant,omitempty" jsonschema:"Optional. Installs one specific build of an entry that offers several (different quantizations or engines), named exactly as it appears in that entry's variant list. Leave empty unless the user explicitly asked for a particular build: by default LocalAI auto-selects the largest variant this machine's engine support and free memory can actually run."` } // InstallBackendRequest is the input for install_backend. diff --git a/pkg/mcp/localaitools/httpapi/client.go b/pkg/mcp/localaitools/httpapi/client.go index 771d2908a..8029655ca 100644 --- a/pkg/mcp/localaitools/httpapi/client.go +++ b/pkg/mcp/localaitools/httpapi/client.go @@ -252,6 +252,9 @@ func (c *Client) InstallModel(ctx context.Context, req localaitools.InstallModel if len(req.Overrides) > 0 { body["overrides"] = req.Overrides } + if req.Variant != "" { + body["variant"] = req.Variant + } var resp struct { ID string `json:"uuid"` StatusURL string `json:"status"` diff --git a/pkg/mcp/localaitools/httpapi/client_test.go b/pkg/mcp/localaitools/httpapi/client_test.go index 319ceffee..38eabf283 100644 --- a/pkg/mcp/localaitools/httpapi/client_test.go +++ b/pkg/mcp/localaitools/httpapi/client_test.go @@ -129,6 +129,39 @@ var _ = Describe("httpapi.Client against the LocalAI admin REST surface", func() Expect(err).ToNot(HaveOccurred()) Expect(id).To(Equal("job-123")) }) + + It("forwards a chosen variant to the apply endpoint", func() { + // The assistant's whole ability to honor "install the Q8 one" + // rests on this field surviving the hop to REST. + var body map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + Expect(json.NewDecoder(r.Body).Decode(&body)).To(Succeed()) + _ = json.NewEncoder(w).Encode(map[string]any{"uuid": "job-123"}) + })) + DeferCleanup(srv.Close) + + _, err := New(srv.URL, "").InstallModel(context.Background(), localaitools.InstallModelRequest{ + ModelName: "qwen2.5-7b-instruct", + Variant: "qwen2.5-7b-instruct-q8", + }) + Expect(err).ToNot(HaveOccurred()) + Expect(body).To(HaveKeyWithValue("variant", "qwen2.5-7b-instruct-q8")) + }) + + It("omits the variant when none was chosen, so the server auto-selects", func() { + var body map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + Expect(json.NewDecoder(r.Body).Decode(&body)).To(Succeed()) + _ = json.NewEncoder(w).Encode(map[string]any{"uuid": "job-123"}) + })) + DeferCleanup(srv.Close) + + _, err := New(srv.URL, "").InstallModel(context.Background(), localaitools.InstallModelRequest{ + ModelName: "qwen2.5-7b-instruct", + }) + Expect(err).ToNot(HaveOccurred()) + Expect(body).ToNot(HaveKey("variant")) + }) }) Describe("GetJobStatus", func() { diff --git a/pkg/mcp/localaitools/inproc/client.go b/pkg/mcp/localaitools/inproc/client.go index 9f6eb8fe2..d3054f614 100644 --- a/pkg/mcp/localaitools/inproc/client.go +++ b/pkg/mcp/localaitools/inproc/client.go @@ -203,6 +203,7 @@ func (c *Client) InstallModel(ctx context.Context, req localaitools.InstallModel Req: gallery.GalleryModel{ Metadata: gallery.Metadata{Name: req.ModelName}, }, + Variant: req.Variant, Galleries: galleries, BackendGalleries: c.AppConfig.BackendGalleries, } diff --git a/pkg/mcp/localaitools/prompts/skills/install_chat_model.md b/pkg/mcp/localaitools/prompts/skills/install_chat_model.md index ca6bd12e2..54d2569d1 100644 --- a/pkg/mcp/localaitools/prompts/skills/install_chat_model.md +++ b/pkg/mcp/localaitools/prompts/skills/install_chat_model.md @@ -6,7 +6,7 @@ Use this when the user wants to install a chat-capable model — including the c 2. Show the top results as a numbered list with name, gallery, short description, and license. If none match, say so and ask whether to broaden the search. 3. Wait for the user to pick. 4. Summarise the chosen install ("I'll install **`/`** — confirm?") and wait for confirmation. -5. On confirmation, call `install_model` with `gallery_name` and `model_name` from the chosen hit. +5. On confirmation, call `install_model` with `gallery_name` and `model_name` from the chosen hit. Leave `variant` empty: LocalAI then auto-selects the largest build this machine's engines and free memory can actually run. Only set `variant` when the user names a specific build (e.g. "the Q8 one"), and pass the name exactly as the entry lists it — an unknown name fails the install rather than falling back to auto-selection. 6. Poll `get_job_status` with the returned job id. Report meaningful progress changes (every ~10–20%, plus completion). 7. When the job reports `processed: true` and no error, call `reload_models`, then `list_installed_models` with `capability: "chat"` to confirm the model is now visible. 8. Tell the user the model is ready and how to use it (its name as the `model` field in chat completions). diff --git a/pkg/mcp/localaitools/tools_models.go b/pkg/mcp/localaitools/tools_models.go index d652198a4..f38e6edce 100644 --- a/pkg/mcp/localaitools/tools_models.go +++ b/pkg/mcp/localaitools/tools_models.go @@ -83,7 +83,7 @@ func registerModelTools(s *mcp.Server, client LocalAIClient, opts Options) { mcp.AddTool(s, &mcp.Tool{ Name: ToolInstallModel, - Description: "Install a model from a gallery. Requires explicit user confirmation per safety rule 1. Returns a job id; poll with get_job_status.", + Description: "Install a model from a gallery. Some entries offer several variants (quantizations or engines) of the same model; leaving `variant` empty auto-selects the largest one this machine can actually run, which is almost always what the user wants. Set `variant` only when the user names a specific build. Requires explicit user confirmation per safety rule 1. Returns a job id; poll with get_job_status.", }, func(ctx context.Context, _ *mcp.CallToolRequest, args InstallModelRequest) (*mcp.CallToolResult, any, error) { // Empty-string check at the tool layer: the SDK schema validator // only enforces presence, not non-empty, and we want a consistent