diff --git a/.agents/adding-backends.md b/.agents/adding-backends.md index e02635a69..34253ce34 100644 --- a/.agents/adding-backends.md +++ b/.agents/adding-backends.md @@ -220,8 +220,9 @@ docker-build-backends: ... docker-build- A gallery entry can declare `variants`, alternative builds of the same weights, and LocalAI picks one per host: it drops builds whose backend cannot run here or -that do not fit memory, then ranks the survivors by **engine preference first, -size second** (`SelectVariant` in `core/gallery/resolve_variant.go`). +that do not fit memory, then ranks the survivors by **engine preference +first, serving feature second, size third** (`SelectVariant` in +`core/gallery/resolve_variant.go`). Ask whether your backend should outrank another one on some hardware. If it should, add it to `engineNamePreferenceRules` in `pkg/system/capabilities.go`, @@ -232,13 +233,26 @@ best engine first for that capability: + {Nvidia, []string{engineVLLM, engineSGLang, engineMyEngine, engineLlamaCpp}}, ``` -That is the ENGINE NAME table, matched against a gallery entry's `backend:` -value. The `backendBuildTagPreferenceRules` table next to it is a different -vocabulary: build tags (`cuda`, `rocm`, `metal`) matched against installed build -directory names for alias resolution. **Putting an engine name in the build tag -table, or a build tag in the engine table, matches nothing and does not error**: -every candidate scores equal and size alone decides, so the preference silently -stops existing. The block comment above both tables spells the contract out. +That is the ENGINE NAME table, matched as a substring of a gallery entry's +`backend:` value. Two sibling tables in the same file speak different +vocabularies and are matched against different things: + +| Table | Vocabulary | Matched against | Consumer | +|-------|-----------|-----------------|----------| +| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` | +| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking | +| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery ENTRY NAME, as a whole segment | gallery variant ranking, one rank below the engine | + +**Putting a token in the wrong table matches nothing and does not error**: every +candidate scores equal and the next sort key decides, so the preference silently +stops existing. The block comment above all three tables spells the contract out. + +The serving feature table is the odd one: it is not keyed by capability, because +no hardware prefers a plain build over an equivalent faster build of the same +weights, and it matches whole name segments rather than substrings, because +entry names are author-supplied free text where a short marker like `mtp` can +turn up inside an unrelated word. A backend never needs to appear in it; it +ranks builds, not engines. Leaving your backend out is a valid choice when no ordering can be justified for it. It then ranks below every known engine and selection falls back to size, @@ -292,7 +306,7 @@ After adding a new backend, verify: - [ ] No Makefile syntax errors (check with linter) - [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern) - [ ] **`Load` validates its input and refuses models it can't serve.** When a model config has no explicit `backend:`, the model loader greedily probes *every* installed backend with the model's name and binds to the first `Load` that succeeds — an accept-anything `Load` will capture arbitrary LLMs (issue #9287). Backends that load a real artefact get this for free (the load fails); backends with no artefact must gate on the name: `opus` accepts only its own name (or none), `local-store` requires the `store.NamespacePrefix` namespace marker sent by `core/backend/stores.go`. -- [ ] **Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets size decide. +- [ ] **Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`, NOT `servingFeaturePreferenceTokens`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets the next sort key decide. - [ ] Documented: added to the category list in `docs/content/features/backends.md` (and any new endpoint/realtime capability documented under `docs/content/`) - [ ] If it is an in-house native C/C++/GGML engine, added to the maintained-engines table in the top-level `README.md` diff --git a/.agents/adding-gallery-models.md b/.agents/adding-gallery-models.md index db706bae5..55d1b5019 100644 --- a/.agents/adding-gallery-models.md +++ b/.agents/adding-gallery-models.md @@ -136,6 +136,17 @@ Rules: per-capability order lives in `engineNamePreferenceRules` (`pkg/system/capabilities.go`); see [adding-backends.md](adding-backends.md) for how a backend gets into it. +- **Serving feature preference sits between engine and size.** Among builds on + an equally preferred engine, one that speculates or predicts several tokens + per step beats the plain build of the same weights, because it answers faster + for the same output: a `-dflash` entry beats an `-mtp` one, and either beats a + plain build. The order lives in `servingFeaturePreferenceTokens` + (`pkg/system/capabilities.go`) and is matched against whole segments of the + variant's entry name, since no gallery field declares it. Engine deliberately + outranks it: a serving feature makes the right engine faster, it does not make + a wrong engine right. Fit still outranks both, so a drafter pairing (strictly + larger than the plain build, since it ships a drafter alongside it) is dropped + on a host too small for it before this order is ever consulted. - A variant is nothing but a name; there is no per-variant memory field. When the measured size for a build is wrong, correct it on the referenced entry by setting that entry's own `size:` (e.g. `size: "20GiB"`). The estimator prefers diff --git a/core/gallery/models.go b/core/gallery/models.go index 0601f0741..e37267213 100644 --- a/core/gallery/models.go +++ b/core/gallery/models.go @@ -283,6 +283,10 @@ func HostResolveEnv(ctx context.Context, systemState *system.SystemState) Resolv // matched on its gallery `backend:` engine name, and the latter reports // build tags that no engine name contains. EnginePreference: systemState.EnginePreferenceTokens(), + // The third vocabulary, and the only one that is not host derived: no + // hardware prefers a plain build over an equivalent faster one, so this + // comes from a package function rather than from systemState. + ServingFeaturePreference: system.ServingFeaturePreferenceTokens(), ProbeMemory: func(target *GalleryModel) uint64 { return probeEntryMemory(ctx, target) }, diff --git a/core/gallery/resolve_variant.go b/core/gallery/resolve_variant.go index 9758dec63..4fab73aea 100644 --- a/core/gallery/resolve_variant.go +++ b/core/gallery/resolve_variant.go @@ -3,8 +3,10 @@ package gallery import ( "errors" "fmt" + "slices" "sort" "strings" + "unicode" ) var ( @@ -91,6 +93,20 @@ type ResolveEnv struct { // view of the hardware, and it is what every host looked like before // preference existed. EnginePreference []string + // ServingFeaturePreference lists the SERVING FEATURES to prefer, best first, + // as system.ServingFeaturePreferenceTokens reports them (currently + // ["dflash", "mtp"]). A token is matched against whole SEGMENTS of the + // variant's ENTRY NAME, splitting on every non-alphanumeric run. + // + // A third vocabulary, matched against a third thing. It is neither a build + // tag nor an engine name: these name a way of serving the same weights + // faster, and no gallery field declares them, so the entry name is the only + // signal there is. pkg/system/capabilities.go documents all three tables + // together and justifies matching on segments rather than substrings. + // + // An empty list ranks every build equally on this axis, which is what every + // host looked like before serving features were ranked at all. + ServingFeaturePreference []string // ProbeMemory measures how much memory a referenced gallery entry needs, // without downloading it. A zero result means "could not tell", never // "needs nothing". @@ -123,16 +139,58 @@ func (e ResolveEnv) backendRuns(backend string) bool { // an unrecognised backend, an unrecognised capability and an absent preference // list all degrade to the same predictable place: ordering by size alone. func (e ResolveEnv) preferenceRank(backend string) int { - if len(e.EnginePreference) == 0 { + name := strings.ToLower(backend) + return preferenceIndex(e.EnginePreference, func(token string) bool { + return strings.Contains(name, token) + }) +} + +// servingFeatureRank scores a variant's entry name against the preferred +// serving feature order, lower being better. +// +// It names no feature, exactly as preferenceRank names no engine: the ordered +// list comes from pkg/system, so teaching LocalAI about a new serving feature +// is a one-line edit to that table and never reaches this file. +// +// A build naming no feature is a plain build and scores just below the least +// preferred known one, which is the whole point: whenever a faster way to serve +// the same weights survived the filters, it outranks the plain build. +func (e ResolveEnv) servingFeatureRank(entryName string) int { + segments := nameSegments(entryName) + return preferenceIndex(e.ServingFeaturePreference, func(token string) bool { + return slices.Contains(segments, token) + }) +} + +// preferenceIndex returns the position of the first token `matches` accepts. +// +// A subject matching no token scores just below the least preferred known one. +// It is never dropped, and a list where nothing matches scores uniformly, so an +// unrecognised subject and an absent preference list degrade to the same +// predictable place: this axis stops discriminating and the next key decides. +func preferenceIndex(tokens []string, matches func(token string) bool) int { + if len(tokens) == 0 { return 0 } - name := strings.ToLower(backend) - for i, token := range e.EnginePreference { - if token != "" && strings.Contains(name, token) { + for i, token := range tokens { + if token != "" && matches(token) { return i } } - return len(e.EnginePreference) + return len(tokens) +} + +// nameSegments splits a gallery entry name into its lowercased alphanumeric +// runs, so "qwen3.6-27b-nvfp4-mtp" yields ["qwen3", "6", "27b", "nvfp4", "mtp"]. +// +// Whole-segment matching is what keeps a short marker from matching inside an +// unrelated word. Entry names are author-supplied free text, unlike the engine +// names preferenceRank matches as substrings, and those are a closed vocabulary +// LocalAI defines itself. +func nameSegments(name string) []string { + return strings.FieldsFunc(strings.ToLower(name), func(r rune) bool { + return !unicode.IsLetter(r) && !unicode.IsDigit(r) + }) } // VariantSelection is the outcome of a selection pass. @@ -168,8 +226,11 @@ type VariantSelection struct { // the largest thing that fits must win against a smaller variant, and a base // of known size must win against a variant whose size nothing could measure. // 5. The survivors are ranked and the best one wins: fit tier first (rankOf -// below), then this host's backend preference, then size. Preference beats -// size deliberately, so a Mac takes an MLX build over a larger GGUF one. +// below), then this host's engine preference, then the serving feature +// preference, then size. Both preferences beat size deliberately, so a Mac +// takes an MLX build over a larger GGUF one, and a host that can hold a +// speculative-decoding build of the same weights takes it over the plain +// one. // 6. With no survivor at all, which can only happen when the caller supplied no // base, there is nothing to install and this reports ErrNoVariantMatch. func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (VariantSelection, error) { @@ -187,6 +248,7 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant memory uint64 rank int preference int + feature int } survivors := make([]ranked, 0, len(options)) @@ -217,6 +279,7 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant memory: memory, rank: rankOf(o, env), preference: env.preferenceRank(o.Backend), + feature: env.servingFeatureRank(o.Variant.Model), }) } @@ -227,7 +290,7 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant ) } - // Three keys, in this order, and the order is the whole design: + // Four keys, in this order, and the order is the whole design: // // 1. rank, the fit tier. Fit is a fact about whether the host can hold the // build at all, so nothing outranks it. @@ -236,12 +299,24 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant // the smaller download. A Mac given an MLX build and a larger llama.cpp // build should run MLX; ranking by size first would install the GGUF and // leave the accelerated build unused. Do NOT move this below size. - // 3. memory, largest first, because among builds on equally preferred - // runtimes a bigger build is a higher quality quantization of the same - // model. Preference must not flatten this: two builds on one backend are - // still ordered by size. + // 3. feature, the serving feature order. Among builds on an equally + // preferred runtime, one that speculates or predicts several tokens per + // step beats the plain build of the same weights, because it answers + // faster for the same output. // - // Stable so that options equal on all three keep their authored order, which + // Engine outranks this deliberately. A serving feature makes the right + // engine faster; it does not make a wrong engine right. Wearing the + // wrong engine costs the whole GPU, whereas a plain build on the native + // engine is merely slower than it could be, so a DFlash build on the + // portable engine must not beat a plain build on the engine this host + // actually serves well. + // 4. memory, largest first, because among builds on equally preferred + // runtimes with the same serving feature a bigger build is a higher + // quality quantization of the same model. Neither preference key may + // flatten this: two plain builds on one backend are still ordered by + // size. + // + // Stable so that options equal on all four keep their authored order, which // is the only thing order still decides. sort.SliceStable(survivors, func(i, j int) bool { if survivors[i].rank != survivors[j].rank { @@ -250,6 +325,9 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant if survivors[i].preference != survivors[j].preference { return survivors[i].preference < survivors[j].preference } + if survivors[i].feature != survivors[j].feature { + return survivors[i].feature < survivors[j].feature + } return survivors[i].memory > survivors[j].memory }) @@ -263,8 +341,9 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant }, nil } -// Fit tiers, best first. Within a tier the host's preferred runtime wins, and -// within one runtime the larger footprint wins; see the sort in SelectVariant. +// Fit tiers, best first. Within a tier the host's preferred runtime wins, then +// the preferred serving feature, then the larger footprint; see the sort in +// SelectVariant. const ( // rankProvenFit is a measured size that the host is measured to satisfy. rankProvenFit = iota diff --git a/core/gallery/resolve_variant_test.go b/core/gallery/resolve_variant_test.go index 671d8789d..b90cf246f 100644 --- a/core/gallery/resolve_variant_test.go +++ b/core/gallery/resolve_variant_test.go @@ -482,6 +482,177 @@ var _ = Describe("SelectVariant", func() { }) }) + Describe("ranking by serving feature", func() { + // These are SERVING FEATURES, exactly what + // system.ServingFeaturePreferenceTokens reports, and a third vocabulary + // after build tags and engine names. They are matched against whole + // segments of the variant's ENTRY NAME rather than against a backend, + // because nothing on a gallery entry declares them. + // + // Spelled out rather than read from pkg/system so these specs pin the + // intended ordering rather than restating whatever the table says. + features := []string{"dflash", "mtp"} + nvidia := []string{"vllm", "sglang", "llama-cpp"} + + It("prefers a speculative build to the plain build of the same weights", func() { + // The rule the feature table exists to express. Both builds fit and + // both run on the same engine, and the plain build is deliberately + // the larger one, so without this axis size would take it and the + // drafter pairing would never be installed. Emptying + // ServingFeaturePreference fails this spec. + options := []gallery.VariantOption{ + option("m-dflash", "llama-cpp", gib(14)), + option("m-q8", "llama-cpp", gib(20)), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-dflash")) + }) + + It("takes the larger plain build once the feature preference is unknown", func() { + // The mirror of the spec above on the identical option set, proving + // the DFlash win comes from the feature list rather than from + // anything intrinsic to the set. + options := []gallery.VariantOption{ + option("m-dflash", "llama-cpp", gib(14)), + option("m-q8", "llama-cpp", gib(20)), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + }) + + It("prefers dflash to mtp", func() { + // Both are speculative pairings, so only the table's order can + // separate them, and the MTP build is deliberately the larger one so + // size cannot be what decides. + options := []gallery.VariantOption{ + option("m-nvfp4-mtp", "llama-cpp", gib(20)), + option("m-dflash", "llama-cpp", gib(14)), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-dflash")) + }) + + It("does not let a preferred feature rescue a build that does not fit", func() { + // A drafter pairing is strictly larger than the plain build, so this + // is the ordinary case on a small host rather than a corner one. Fit + // is a filter; the feature preference only ranks survivors. + options := []gallery.VariantOption{ + option("m-dflash", "llama-cpp", gib(48)), + option("m-q8", "llama-cpp", gib(12)), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-dflash"))) + }) + + It("lets the host engine preference outrank the serving feature", func() { + // A serving feature makes the right engine faster; it does not make + // a wrong engine right. On nvidia the plain vLLM build therefore + // beats a DFlash llama.cpp build even though both fit and the + // llama.cpp one is larger. + options := []gallery.VariantOption{ + option("m-dflash", "llama-cpp", gib(24)), + option("m-vllm-awq", "vllm", gib(8)), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm-awq")) + }) + + It("still picks the largest fitting build among equally featured builds", func() { + // Neither preference key may flatten size ordering: with one engine + // and one feature in play there is nothing left for them to decide. + options := []gallery.VariantOption{ + option("m-dflash-q4", "llama-cpp", gib(8)), + option("m-dflash-q8", "llama-cpp", gib(14)), + option("m-dflash-f16", "llama-cpp", gib(48)), + base("m-q4", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-dflash-q8")) + }) + + It("does not read a feature token out of the middle of an unrelated word", func() { + // The guard on matching whole name segments rather than substrings. + // "smtp-assistant" is a mail model and carries no MTP heads, yet + // strings.Contains would rank it as an MTP build, and being the + // larger of the two it would then win. Entry names are + // author-supplied free text, unlike the closed engine vocabulary + // preferenceRank matches as substrings. + options := []gallery.VariantOption{ + option("m-smtp-assistant-q8", "llama-cpp", gib(24)), + option("m-nvfp4-mtp", "llama-cpp", gib(14)), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-nvfp4-mtp")) + }) + + It("leaves an unfeatured build ranked last rather than dropping it", func() { + // Ranking never filters. With only plain builds on offer the axis + // scores them uniformly and selection degrades to size alone. + options := []gallery.VariantOption{ + option("m-q8", "llama-cpp", gib(12)), + option("m-q4", "llama-cpp", gib(6)), + base("m-q2", gib(3)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + Expect(selection.Reasons).To(BeEmpty()) + }) + }) + Describe("falling back to the base", func() { It("selects the base when nothing else fits", func() { options := []gallery.VariantOption{ diff --git a/gallery/index.yaml b/gallery/index.yaml index 3aeb87806..b8a4e02ad 100644 --- a/gallery/index.yaml +++ b/gallery/index.yaml @@ -386,6 +386,7 @@ - model: deepseek-v4-flash-q2 - model: deepseek-v4-flash-q2-q4 - model: deepseek-v4-flash-q4-ssd + - model: deepseek-v4-flash-q2-mtp url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: - https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF @@ -508,6 +509,8 @@ sha256: e7a8eafdd8013443b6bcc4b6fb47b2d2025f772d359650b9ceb7d75971e22cad uri: https://huggingface.co/unsloth/Qwen-AgentWorld-35B-A3B-GGUF/resolve/main/Qwen-AgentWorld-35B-A3B-UD-Q4_K_M.gguf - name: "ornith-1.0-9b" + variants: + - model: ornith-1.0-9b-mtp url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: - https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B-GGUF @@ -1795,6 +1798,8 @@ sha256: ef269e294502d6ee3722cbf129681b2586c2e6ceb79d0507963c92146e058cd4 uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF/resolve/main/mmproj-F32.gguf - name: "gemma-4-12b-it-qat-q4_0" + variants: + - model: gemma-4-12b-it-qat-mtp url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: - https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf @@ -1848,6 +1853,8 @@ sha256: e70b0e5cd80323d5d588b4ed06780356b7b1ba03995a4b8164c6ae9db0ff5989 uri: https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf/resolve/main/mmproj-gemma-4-12b-it-qat-q4_0.gguf - name: "gemma-4-e2b-it-qat-q4_0" + variants: + - model: gemma-4-e2b-it-qat-mtp url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: - https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf @@ -1890,6 +1897,8 @@ sha256: 58c187648007cab392bd5678b87e862c3e8794017deb945feea2cf256195e96a uri: https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf/resolve/main/gemma-4-E2B-it-mmproj.gguf - name: "gemma-4-e4b-it-qat-q4_0" + variants: + - model: gemma-4-e4b-it-qat-mtp url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: - https://huggingface.co/google/gemma-4-E4B-it-qat-q4_0-gguf @@ -1975,6 +1984,8 @@ sha256: d8e2de16e17515d9061b23c9a002715f996f9e0c87b93a9354264611bfab9239 uri: https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf/resolve/main/gemma-4-26B-it-mmproj.gguf - name: "gemma-4-31b-it-qat-q4_0" + variants: + - model: gemma-4-31b-it-qat-mtp url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: - https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf @@ -3151,6 +3162,9 @@ sha256: 13bd039f95c9ea46ef1d75905faa7be6ca4e47a5af9d4cf62e298a738a5b195f uri: https://huggingface.co/KyleHessling1/Qwopus-GLM-18B-Merged-GGUF/resolve/main/Qwopus-GLM-18B-Healed-Q4_K_M.gguf - name: qwen3.6-27b + variants: + - model: qwen3.6-27b-dflash + - model: qwen3.6-27b-nvfp4-mtp url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/unsloth/Qwen3.6-27B-GGUF @@ -3441,6 +3455,9 @@ sha256: b5aa0676be588bf6ef3bbdb89905d7d239b2a809637f0766a6ce23aed6c6b5b4 uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-Quality.gguf - name: qwen3.6-35b-a3b + variants: + - model: qwen3.6-35b-a3b-dflash + - model: qwen3.6-35b-a3b-nvfp4-mtp url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF @@ -3579,6 +3596,8 @@ sha256: 2aa99ffb47033ead4a3f1584fec5283905302c1c16fed59c99e0eec131c6dc53 uri: huggingface://ggml-org/gemma-4-26B-A4B-it-GGUF/mmproj-gemma-4-26B-A4B-it-bf16.gguf - name: gemma-4-e2b-it + variants: + - model: "gemma-4-e2b-it:sglang-mtp" url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/google/gemma-4-E2B-it @@ -3622,6 +3641,8 @@ sha256: e42083b71a9e31e0f722171d551f6d92b101544001c4dde040306a8f2160fe8c uri: huggingface://ggml-org/gemma-4-E2B-it-GGUF/mmproj-gemma-4-E2B-it-bf16.gguf - name: gemma-4-e4b-it + variants: + - model: "gemma-4-e4b-it:sglang-mtp" url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/google/gemma-4-E4B-it @@ -4039,6 +4060,8 @@ sha256: 8221b6a48c714db6829a92760c31034d7ecd436f830c61624ccc92b461b4a1c4 uri: https://huggingface.co/mradermacher/Q3.5-BlueStar-27B-GGUF/resolve/main/Q3.5-BlueStar-27B.mmproj-f16.gguf - name: qwen3.5-9b + variants: + - model: qwen3.5-9b-dflash url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/unsloth/Qwen3.5-9B-GGUF @@ -12070,6 +12093,8 @@ sha256: 376902d50612ecfc5bd8b268f376c04d10ad7e480f99a1483b833f04344a549e uri: huggingface://MaziyarPanahi/Qwen3-8B-GGUF/Qwen3-8B.Q4_K_M.gguf - name: qwen3-4b + variants: + - model: qwen3-4b-dflash url: github:mudler/LocalAI/gallery/qwen3.yaml@master urls: - https://huggingface.co/Qwen/Qwen3-4B diff --git a/pkg/system/capabilities.go b/pkg/system/capabilities.go index 5ceea9307..a95690a45 100644 --- a/pkg/system/capabilities.go +++ b/pkg/system/capabilities.go @@ -53,29 +53,43 @@ const ( engineSGLang = "sglang" engineLlamaCpp = "llama-cpp" engineMLX = "mlx" + + // Serving feature names (private). A third vocabulary again: not a build + // tag and not an engine, but the inference-time strategy a published build + // enables. They are matched against a gallery ENTRY NAME, so they are + // spelled the way gallery authors spell them in an entry name segment. + servingFeatureDFlash = "dflash" + servingFeatureMTP = "mtp" ) -// There are TWO preference tables below and they speak DIFFERENT VOCABULARIES. -// Merging them looks tempting and silently breaks one of the two consumers, -// because a token that means something in one vocabulary means nothing in the -// other. Read this before editing either. +// There are THREE preference tables below and they speak DIFFERENT VOCABULARIES, +// matched against three different things. Merging any two looks tempting and +// silently breaks a consumer, because a token that means something in one +// vocabulary means nothing in the others. Read this before editing any of them. // // - backendBuildTagPreferenceRules holds BUILD TAGS ("cuda", "rocm", "metal"). -// They are matched against INSTALLED BACKEND BUILD DIRECTORY NAMES such as -// "llama-cpp-cuda-12" or "cuda12-vllm". Consumer: alias resolution in -// ListSystemBackends (core/gallery/backends.go), which picks which installed -// build of one alias to run. +// They are matched as substrings of INSTALLED BACKEND BUILD DIRECTORY NAMES +// such as "llama-cpp-cuda-12" or "cuda12-vllm". Consumer: alias resolution +// in ListSystemBackends (core/gallery/backends.go), which picks which +// installed build of one alias to run. // // - engineNamePreferenceRules holds ENGINE NAMES ("vllm", "llama-cpp", "mlx"). -// They are matched against a gallery entry's `backend:` value, which never -// carries a build tag: no entry in gallery/index.yaml contains "cuda", -// "rocm", "sycl" or "vulkan" anywhere in its backend name. Consumer: gallery -// variant auto-selection (core/gallery/resolve_variant.go), which picks -// which build of one model's weights to install. +// They are matched as substrings of a gallery entry's `backend:` value, +// which never carries a build tag: no entry in gallery/index.yaml contains +// "cuda", "rocm", "sycl" or "vulkan" anywhere in its backend name. Consumer: +// gallery variant auto-selection (core/gallery/resolve_variant.go), which +// picks which build of one model's weights to install. +// +// - servingFeaturePreferenceTokens holds SERVING FEATURES ("dflash", "mtp"). +// They are matched against WHOLE SEGMENTS of a gallery ENTRY NAME, not +// against a backend or an engine, because no field on a gallery entry +// declares them. Same consumer as the engine table, applied one rank below +// it. See the token list for why segment matching rather than substring. // // Feeding build tags to the variant ranker matches nothing, which does not // error: every candidate simply scores equal and size alone decides, so the // preference silently stops existing. That is exactly the bug this split fixes. +// A serving feature fed to either of the other two tables fails the same way. // backendPreferenceRule maps a detected capability to preferred tokens, best // first. Both tables share this shape; only their vocabulary differs. @@ -180,6 +194,53 @@ var engineNamePreferenceRules = []backendPreferenceRule{ // list as and is the behaviour that predates preference. var defaultEnginePreferenceTokens = []string{} +// servingFeaturePreferenceTokens is the SERVING FEATURE table, best first. See +// the block comment above for the vocabulary contract. +// +// A serving feature is a way of running the same weights faster rather than a +// different set of weights: a DFlash build pairs the base GGUF with a drafter +// so the target accepts several speculated tokens per step, and an MTP build +// carries extra prediction heads to the same end. Both produce the same +// distribution as the plain build, so whenever one fits there is no reason to +// install the plain build instead. +// +// DFlash leads because it is the newer pairing and accepts more tokens per step +// in practice; MTP follows; a build naming neither is plain and ranks last. +// A build's extra weights make it strictly larger than the plain build, which +// is why this ranks BELOW fit: the size filter drops the pairing on a host too +// small for it before this list is ever consulted. +// +// Unlike the two tables above this one is NOT keyed by capability. A serving +// feature is a property of the published build, and no hardware prefers the +// plain build over an equivalent faster one, so there is no host-shaped +// ordering to express. Should one ever appear, this becomes a rule table like +// its neighbours without any consumer changing. +// +// Matching is against WHOLE SEGMENTS of the entry name, splitting on every +// non-alphanumeric run, rather than the substring matching the tables above +// use. The other two vocabularies are closed sets that LocalAI itself defines, +// whereas an entry name is author-supplied free text where a short token can +// turn up inside an unrelated word. Segment matching also keeps this honest +// about what it cannot know: "qwen3.6-27b-mtp-pi-tune" is a separate finetune +// and not a variant of anything, so it never reaches ranking at all, but if a +// future finetune were grouped, the name is all there is to go on either way. +// +// The name is the signal because nothing else is reliable. Tags would be the +// natural declaration, but they are optional and demonstrably inconsistent: +// "gemma-4-e2b-it:sglang-mtp" carries an "mtp" tag while "ornith-1.0-9b-mtp" +// and "qwen3.6-27b-nvfp4-mtp" carry none, so keying on tags would rank two of +// those three as plain builds. +var servingFeaturePreferenceTokens = []string{servingFeatureDFlash, servingFeatureMTP} + +// ServingFeaturePreferenceTokens returns the serving features to prefer, best +// first, for ranking gallery model variants against each other. +// +// It is a package function rather than a SystemState method precisely because +// the answer does not depend on the host; see the table for why. +func ServingFeaturePreferenceTokens() []string { + return slices.Clone(servingFeaturePreferenceTokens) +} + var ( cuda13DirExists bool cuda12DirExists bool diff --git a/pkg/system/capabilities_test.go b/pkg/system/capabilities_test.go index 2e589327f..efd808090 100644 --- a/pkg/system/capabilities_test.go +++ b/pkg/system/capabilities_test.go @@ -342,3 +342,46 @@ var _ = Describe("DetectedCapability", func() { Expect(state.DetectedCapability()).NotTo(Equal("default")) }) }) + +var _ = Describe("ServingFeaturePreferenceTokens", func() { + It("puts dflash ahead of mtp", func() { + // The rule the serving feature table exists to express. Both are ways + // of serving the same weights faster; only this order separates them. + Expect(ServingFeaturePreferenceTokens()).To(Equal([]string{"dflash", "mtp"})) + }) + + It("carries neither a build tag nor an engine name", func() { + // The third vocabulary, and the same merge hazard as the other two. A + // build tag or an engine name here would be matched against gallery + // entry names, where it means nothing. + Expect(ServingFeaturePreferenceTokens()).ToNot(ContainElements( + "cuda", "rocm", "hip", "sycl", "metal", "vulkan", "cpu", "darwin-x86", + "vllm", "sglang", "llama-cpp", "mlx", + )) + }) + + It("does not depend on the host capability", func() { + // Deliberately not a SystemState method: no hardware prefers the plain + // build over an equivalent faster one, so there is no host-shaped + // ordering to express here. + previous := os.Getenv(capabilityEnv) + defer func() { + if previous != "" { + os.Setenv(capabilityEnv, previous) + } else { + os.Unsetenv(capabilityEnv) + } + }() + + Expect(os.Setenv(capabilityEnv, "nvidia-cuda-12")).To(Succeed()) + onNvidia := ServingFeaturePreferenceTokens() + Expect(os.Setenv(capabilityEnv, metal)).To(Succeed()) + Expect(ServingFeaturePreferenceTokens()).To(Equal(onNvidia)) + }) + + It("hands out a copy so one caller cannot corrupt another's lookup", func() { + first := ServingFeaturePreferenceTokens() + first[0] = "clobbered" + Expect(ServingFeaturePreferenceTokens()[0]).To(Equal("dflash")) + }) +})