diff --git a/.agents/adding-backends.md b/.agents/adding-backends.md index c6e4506e4..603c9dbc8 100644 --- a/.agents/adding-backends.md +++ b/.agents/adding-backends.md @@ -241,7 +241,7 @@ vocabularies and are matched against different things: |-------|-----------|-----------------|----------| | `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` | | `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking | -| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole; failing that its ENTRY NAME, as a whole segment | gallery variant ranking, one rank below the engine | +| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole and case-insensitively, and nothing else | gallery variant ranking, one rank below the engine | **Putting a token in the wrong table matches nothing and does not error**: every candidate scores equal and the next sort key decides, so the preference silently @@ -249,11 +249,19 @@ stops existing. The block comment above all three tables spells the contract out The serving feature table is the odd one: it is not keyed by capability, because no hardware prefers a plain build over an equivalent faster build of the same -weights. It reads a declared tag first, since a tag is a deliberate statement, -and falls back to whole name segments rather than substrings, because entry -names are author-supplied free text where a short marker like `mtp` can turn up -inside an unrelated word. A backend never needs to appear in it; it ranks -builds, not engines. +weights. It reads a declared tag and nothing else. The entry name was the +original signal and is gone: a naming convention is not a contract, and names +are author-supplied free text where a short marker like `mtp` turns up inside +unrelated words or on weights whose entry enables nothing. +`overrides.options` was rejected for the mirror-image reason: `spec_type:` is +llama.cpp's config vocabulary, whereas a cross-backend ranking decision must +work the same for `ds4`'s `mtp_path:` and `sglang`'s `speculative_algorithm:`. + +**If your backend can serve the same weights faster** (speculative decoding, +multi-token prediction), say so in the docs for its gallery entries so curators +tag them: the tagging rule and the per-backend evidence table live in +[adding-gallery-models.md](adding-gallery-models.md). A backend never needs to +appear in the token table itself; it ranks builds, not engines. Leaving your backend out is a valid choice when no ordering can be justified for it. It then ranks below every known engine and selection falls back to size, diff --git a/.agents/adding-gallery-models.md b/.agents/adding-gallery-models.md index 3d87c0c9e..155495e62 100644 --- a/.agents/adding-gallery-models.md +++ b/.agents/adding-gallery-models.md @@ -139,13 +139,11 @@ Rules: - **Serving feature preference sits between engine and size.** Among builds on an equally preferred engine, one that speculates or predicts several tokens per step beats the plain build of the same weights, because it answers faster - for the same output: a `-dflash` entry beats an `-mtp` one, and either beats a + for the same output: a `dflash` build beats an `mtp` one, and either beats a plain build. The order lives in `servingFeaturePreferenceTokens` - (`pkg/system/capabilities.go`) and is matched against the entry's `tags:`, - falling back to whole segments of its entry name. **Tag every speculative or - MTP build** (`- dflash` / `- mtp`) so ranking recognises it; the fallback - covers entries nobody tagged, but do not rely on it, since a build whose name - spells the feature nowhere is invisible without the tag. Engine deliberately + (`pkg/system/capabilities.go`) and is matched against the entry's `tags:` and + **nothing else**: not the entry name, not `overrides.options`. See + [the tagging rule](#the-dflash--mtp-tagging-rule) below. Engine deliberately outranks it: a serving feature makes the right engine faster, it does not make a wrong engine right. Fit still outranks both, so a drafter pairing (strictly larger than the plain build, since it ships a drafter alongside it) is dropped @@ -163,6 +161,36 @@ Users can override the automatic choice with `variant` on `POST /models/apply`, The gallery lint specs live in `core/gallery`, so run that suite after adding a `variants` list. +### The `dflash` / `mtp` tagging rule + +**Tag an entry `dflash` or `mtp` when the entry actually configures that +feature. Variant ranking reads the tag and nothing else.** + +Decide by looking at what the entry configures, in whatever vocabulary its +backend uses: + +| Backend | Configures the feature when it declares | +|---------|------------------------------------------| +| `llama-cpp` | `overrides.options` contains `spec_type:draft-dflash` or `spec_type:draft-mtp` | +| `ds4` | `overrides.options` contains `mtp_path:` / `mtp_draft:` | +| `sglang` | the referenced `gallery/*.yaml` sets `speculative_algorithm:` | + +That check is curation-time only. `spec_type` is llama.cpp's config vocabulary, +and a cross-backend ranking decision must not depend on one backend's option +syntax, which is precisely why the ranker reads the tag instead of the options. + +Two mistakes the rule exists to prevent: + +- **Weights that carry the heads are not an entry that enables them.** The + NVFP4 GGUF entries ship MTP-bearing weights but set only `use_jinja:true`, so + they enable no speculative decoding and must NOT be tagged. Tagging them wins + them the feature axis without being any faster. +- **A name is not a declaration.** An entry whose name spells `-mtp` while + configuring nothing gets no tag, and an entry that configures the feature is + tagged even when its name says nothing (`hy3`, `glm-5.2`). Ranking never reads + the name, so an untagged build that does enable the feature is simply ranked + as plain rather than promoted on a marker nobody meant. + ## Available template configs Look at existing `.yaml` files in `gallery/` to find the right prompt template for your model architecture: diff --git a/core/gallery/resolve_variant.go b/core/gallery/resolve_variant.go index 8a27a9def..fec25b532 100644 --- a/core/gallery/resolve_variant.go +++ b/core/gallery/resolve_variant.go @@ -6,7 +6,6 @@ import ( "slices" "sort" "strings" - "unicode" ) var ( @@ -44,13 +43,17 @@ type VariantOption struct { // a zero requirement: a probe that cannot reach the network must never be // able to break an install. ProbedMemory uint64 - // Tags are the referenced entry's declared tags, and they are the - // AUTHORITATIVE serving feature signal: a tag is something an author wrote - // down on purpose, whereas an entry name only happens to contain a marker. + // Tags are the referenced entry's declared tags, and they are the SOLE + // serving feature signal. A tag is something an author wrote down on + // purpose, whereas an entry name only happens to contain a marker, so + // nothing else is consulted. // - // Empty does not mean "no features". An entry nobody has tagged yet falls - // back to its name, so tagging discipline improves ranking without being a - // precondition for it. + // Empty therefore means "declares no serving feature", and an entry that + // enables one without saying so ranks as a plain build. That is the cost of + // making the signal a declaration, and it is the right cost: an install + // silently downgraded to the plain build is recoverable, whereas a build + // promoted on a marker nobody meant is not visible at all. + // .agents/adding-gallery-models.md states the tagging rule. Tags []string } @@ -104,15 +107,13 @@ type ResolveEnv struct { // ServingFeaturePreference lists the SERVING FEATURES to prefer, best first, // as system.ServingFeaturePreferenceTokens reports them (currently // ["dflash", "mtp"]). A token matches when it equals one of the variant's - // declared TAGS, or failing that when it equals a whole SEGMENT of its - // ENTRY NAME, splitting on every non-alphanumeric run. + // declared TAGS, compared whole and case-insensitively. Nothing else is + // matched: not the entry name, not its backend options. // // A third vocabulary, matched against a third thing. It is neither a build // tag nor an engine name: these name a way of serving the same weights - // faster. A gallery tag is the declaration and takes precedence; the entry - // name remains a fallback for entries nobody has tagged. - // pkg/system/capabilities.go documents all three tables together and - // justifies why the name half matches segments rather than substrings. + // faster. pkg/system/capabilities.go documents all three tables together + // and justifies why this one reads a declaration rather than free text. // // An empty list ranks every build equally on this axis, which is what every // host looked like before serving features were ranked at all. @@ -162,30 +163,30 @@ func (e ResolveEnv) preferenceRank(backend string) int { // list comes from pkg/system, so teaching LocalAI about a new serving feature // is a one-line edit to that table and never reaches this file. // -// Two signals, either of which is enough. A declared TAG is the authoritative -// one and is compared whole, because a tag is a deliberate statement rather -// than free text: an author who writes "mtp" in a tag list means the feature, -// so the substring risk that segment matching exists to avoid does not arise -// there. The ENTRY NAME stays as a fallback, matched by whole segment as -// before, so an entry nobody has tagged yet ranks exactly the way it did -// before tags were read at all. Tags-only would have been a regression on the -// day it shipped, since most MTP entries carried no tag. +// A declared TAG is the ONLY signal, compared whole and case-insensitively. An +// entry name is deliberately not consulted: a naming convention is not a +// contract, and an author is free to spell a build however they like, so +// reading a marker out of a name infers a capability nobody declared. Tags are +// a LocalAI-level vocabulary the gallery curators control and they mean the +// same thing on every backend. `overrides.options` was considered as the +// signal and rejected for the opposite reason: spec_type:draft-mtp is what +// actually enables the feature, but that spelling is llama.cpp's config +// vocabulary, and a cross-backend ranking decision must not depend on one +// backend's option syntax. Curators check the options at tagging time; the +// ranker reads only the tag. // -// A build declaring no feature by either route is a plain build and scores -// just below the least preferred known one, which is the whole point: whenever -// a faster way to serve the same weights survived the filters, it outranks the -// plain build. +// A build declaring no feature is a plain build and scores just below the +// least preferred known one, which is the whole point: whenever a faster way +// to serve the same weights survived the filters, it outranks the plain build. func (e ResolveEnv) servingFeatureRank(o VariantOption) int { - segments := nameSegments(o.Variant.Model) tags := lowercased(o.Tags) return preferenceIndex(e.ServingFeaturePreference, func(token string) bool { - return slices.Contains(tags, token) || slices.Contains(segments, token) + return slices.Contains(tags, token) }) } // lowercased folds a tag list for comparison, so a gallery author writing -// "MTP" declares the same feature as one writing "mtp". Names are already -// folded by nameSegments, and the two signals must not disagree about case. +// "MTP" declares the same feature as one writing "mtp". func lowercased(tags []string) []string { if len(tags) == 0 { return nil @@ -215,19 +216,6 @@ func preferenceIndex(tokens []string, matches func(token string) bool) int { return len(tokens) } -// nameSegments splits a gallery entry name into its lowercased alphanumeric -// runs, so "qwen3.6-27b-nvfp4-mtp" yields ["qwen3", "6", "27b", "nvfp4", "mtp"]. -// -// Whole-segment matching is what keeps a short marker from matching inside an -// unrelated word. Entry names are author-supplied free text, unlike the engine -// names preferenceRank matches as substrings, and those are a closed vocabulary -// LocalAI defines itself. -func nameSegments(name string) []string { - return strings.FieldsFunc(strings.ToLower(name), func(r rune) bool { - return !unicode.IsLetter(r) && !unicode.IsDigit(r) - }) -} - // VariantSelection is the outcome of a selection pass. type VariantSelection struct { Option VariantOption diff --git a/core/gallery/resolve_variant_test.go b/core/gallery/resolve_variant_test.go index fa7efd3b1..c5bd86481 100644 --- a/core/gallery/resolve_variant_test.go +++ b/core/gallery/resolve_variant_test.go @@ -486,14 +486,23 @@ var _ = Describe("SelectVariant", func() { // These are SERVING FEATURES, exactly what // system.ServingFeaturePreferenceTokens reports, and a third vocabulary // after build tags and engine names. They are matched against a - // variant's declared TAGS, and failing that against whole segments of - // its ENTRY NAME, rather than against a backend. + // variant's declared TAGS and against nothing else. // // Spelled out rather than read from pkg/system so these specs pin the // intended ordering rather than restating whatever the table says. features := []string{"dflash", "mtp"} nvidia := []string{"vllm", "sglang", "llama-cpp"} + // A tag is the whole signal, so every candidate that is meant to carry + // a feature declares it here. Candidates built with `option` carry no + // tags and are therefore plain builds no matter what their name says, + // which several specs below rely on. + tagged := func(model string, probed uint64, tags ...string) gallery.VariantOption { + o := option(model, "llama-cpp", probed) + o.Tags = tags + return o + } + It("prefers a speculative build to the plain build of the same weights", func() { // The rule the feature table exists to express. Both builds fit and // both run on the same engine, and the plain build is deliberately @@ -501,8 +510,8 @@ var _ = Describe("SelectVariant", func() { // drafter pairing would never be installed. Emptying // ServingFeaturePreference fails this spec. options := []gallery.VariantOption{ - option("m-dflash", "llama-cpp", gib(14)), - option("m-q8", "llama-cpp", gib(20)), + tagged("m-turbo-q4", gib(14), "llm", "dflash"), + tagged("m-q8", gib(20), "llm"), base("m-q4", gib(6)), } @@ -512,7 +521,7 @@ var _ = Describe("SelectVariant", func() { ServingFeaturePreference: features, }, "") Expect(err).ToNot(HaveOccurred()) - Expect(selection.Option.Variant.Model).To(Equal("m-dflash")) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4")) }) It("takes the larger plain build once the feature preference is unknown", func() { @@ -520,8 +529,8 @@ var _ = Describe("SelectVariant", func() { // the DFlash win comes from the feature list rather than from // anything intrinsic to the set. options := []gallery.VariantOption{ - option("m-dflash", "llama-cpp", gib(14)), - option("m-q8", "llama-cpp", gib(20)), + tagged("m-turbo-q4", gib(14), "llm", "dflash"), + tagged("m-q8", gib(20), "llm"), base("m-q4", gib(6)), } @@ -538,8 +547,8 @@ var _ = Describe("SelectVariant", func() { // separate them, and the MTP build is deliberately the larger one so // size cannot be what decides. options := []gallery.VariantOption{ - option("m-nvfp4-mtp", "llama-cpp", gib(20)), - option("m-dflash", "llama-cpp", gib(14)), + tagged("m-alpha-q8", gib(20), "llm", "mtp"), + tagged("m-beta-q4", gib(14), "llm", "dflash"), base("m-q4", gib(6)), } @@ -549,7 +558,7 @@ var _ = Describe("SelectVariant", func() { ServingFeaturePreference: features, }, "") Expect(err).ToNot(HaveOccurred()) - Expect(selection.Option.Variant.Model).To(Equal("m-dflash")) + Expect(selection.Option.Variant.Model).To(Equal("m-beta-q4")) }) It("does not let a preferred feature rescue a build that does not fit", func() { @@ -557,8 +566,8 @@ var _ = Describe("SelectVariant", func() { // is the ordinary case on a small host rather than a corner one. Fit // is a filter; the feature preference only ranks survivors. options := []gallery.VariantOption{ - option("m-dflash", "llama-cpp", gib(48)), - option("m-q8", "llama-cpp", gib(12)), + tagged("m-turbo-f16", gib(48), "llm", "dflash"), + tagged("m-q8", gib(12), "llm"), base("m-q4", gib(6)), } @@ -569,7 +578,7 @@ var _ = Describe("SelectVariant", func() { }, "") Expect(err).ToNot(HaveOccurred()) Expect(selection.Option.Variant.Model).To(Equal("m-q8")) - Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-dflash"))) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-turbo-f16"))) }) It("lets the host engine preference outrank the serving feature", func() { @@ -578,7 +587,7 @@ var _ = Describe("SelectVariant", func() { // beats a DFlash llama.cpp build even though both fit and the // llama.cpp one is larger. options := []gallery.VariantOption{ - option("m-dflash", "llama-cpp", gib(24)), + tagged("m-turbo-q8", gib(24), "llm", "dflash"), option("m-vllm-awq", "vllm", gib(8)), base("m-q4", gib(6)), } @@ -596,9 +605,9 @@ var _ = Describe("SelectVariant", func() { // Neither preference key may flatten size ordering: with one engine // and one feature in play there is nothing left for them to decide. options := []gallery.VariantOption{ - option("m-dflash-q4", "llama-cpp", gib(8)), - option("m-dflash-q8", "llama-cpp", gib(14)), - option("m-dflash-f16", "llama-cpp", gib(48)), + tagged("m-turbo-q4", gib(8), "llm", "dflash"), + tagged("m-turbo-q8", gib(14), "llm", "dflash"), + tagged("m-turbo-f16", gib(48), "llm", "dflash"), base("m-q4", gib(2)), } @@ -608,29 +617,7 @@ var _ = Describe("SelectVariant", func() { ServingFeaturePreference: features, }, "") Expect(err).ToNot(HaveOccurred()) - Expect(selection.Option.Variant.Model).To(Equal("m-dflash-q8")) - }) - - It("does not read a feature token out of the middle of an unrelated word", func() { - // The guard on matching whole name segments rather than substrings. - // "smtp-assistant" is a mail model and carries no MTP heads, yet - // strings.Contains would rank it as an MTP build, and being the - // larger of the two it would then win. Entry names are - // author-supplied free text, unlike the closed engine vocabulary - // preferenceRank matches as substrings. - options := []gallery.VariantOption{ - option("m-smtp-assistant-q8", "llama-cpp", gib(24)), - option("m-nvfp4-mtp", "llama-cpp", gib(14)), - base("m-q4", gib(6)), - } - - selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ - AvailableMemory: gib(64), - EnginePreference: nvidia, - ServingFeaturePreference: features, - }, "") - Expect(err).ToNot(HaveOccurred()) - Expect(selection.Option.Variant.Model).To(Equal("m-nvfp4-mtp")) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8")) }) It("leaves an unfeatured build ranked last rather than dropping it", func() { @@ -653,16 +640,6 @@ var _ = Describe("SelectVariant", func() { }) Describe("reading the declared tags", func() { - // A tag is the authoritative signal and the name is the fallback, - // so each of the two has to be pinned on its own: a spec whose - // candidate declares the feature both ways would keep passing after - // either half was deleted. - tagged := func(model string, probed uint64, tags ...string) gallery.VariantOption { - o := option(model, "llama-cpp", probed) - o.Tags = tags - return o - } - It("prefers a build whose tag declares the feature its name does not", func() { // The case tags exist for. "m-turbo-q8" carries MTP heads but // spells nothing in its name, which is the shape most of the @@ -683,10 +660,16 @@ var _ = Describe("SelectVariant", func() { Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8")) }) - It("still prefers a build the name alone declares, with no tags at all", func() { - // The fallback, pinned independently of the tag path. An entry - // nobody has tagged must rank exactly as it did before tags - // were read, so deleting the name half fails here alone. + It("does NOT prefer a build that only its name declares, with no tags at all", func() { + // The inversion of the old name-fallback spec, and the + // regression this change exists to guard. A name is + // author-supplied free text and a naming convention is not a + // contract: "m-nvfp4-mtp" is exactly the shape of the gallery's + // NVFP4 entries, whose weights carry MTP heads while the entry + // enables no speculative decoding at all, so ranking it as a + // speculative build made it win the feature axis without being + // any faster. With no tag it is a plain build and the larger + // plain build takes it on size. options := []gallery.VariantOption{ option("m-nvfp4-mtp", "llama-cpp", gib(14)), tagged("m-plain-q8", gib(20), "llm", "gguf"), @@ -699,13 +682,35 @@ var _ = Describe("SelectVariant", func() { ServingFeaturePreference: features, }, "") Expect(err).ToNot(HaveOccurred()) - Expect(selection.Option.Variant.Model).To(Equal("m-nvfp4-mtp")) + Expect(selection.Option.Variant.Model).To(Equal("m-plain-q8")) + }) + + It("lets a tagged build beat a larger one whose name says the same feature", func() { + // The sharper form of the spec above: the name half is not + // merely unnecessary, it is not consulted. The untagged + // "m-dflash" would outrank the tagged MTP build on the old + // name-first ordering, and is the larger build besides, so it + // wins on either of the two ways the name could still be read. + options := []gallery.VariantOption{ + option("m-dflash", "llama-cpp", gib(20)), + tagged("m-turbo-q4", gib(14), "llm", "mtp"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4")) }) It("matches a tag regardless of the case it was written in", func() { // Gallery tags are author-supplied, so the same declaration - // arrives in whatever case the author typed. Name segments are - // already folded and the two signals must not disagree. + // arrives in whatever case the author typed. A curator who + // writes "MTP" has declared the feature just as plainly as one + // who writes "mtp", and the sole signal must not turn on that. options := []gallery.VariantOption{ tagged("m-turbo-q8", gib(14), "LLM", "MTP"), tagged("m-plain-q8", gib(20), "llm", "gguf"), @@ -722,9 +727,9 @@ var _ = Describe("SelectVariant", func() { }) It("prefers dflash to mtp when both are declared by tag", func() { - // The table's order has to survive the new signal: with neither - // name saying anything, only the tags separate these two, and - // the MTP build is deliberately the larger one. + // The table's order has to survive on the tag path with both + // features spelled out explicitly, and the MTP build is + // deliberately the larger one so size cannot be what decides. options := []gallery.VariantOption{ tagged("m-alpha-q8", gib(20), "llm", "mtp"), tagged("m-beta-q8", gib(14), "llm", "dflash"), @@ -741,11 +746,11 @@ var _ = Describe("SelectVariant", func() { }) It("does not read a feature out of an unrelated tag", func() { - // Tags are compared whole, exactly as name segments are, so a - // tag that merely contains a feature token declares nothing. - // "multimodal" contains no feature; "smtp" does contain "mtp" - // as a substring and must not count either. The tagged build is - // the larger one, so a false positive would hand it the win. + // Tags are compared whole, so a tag that merely contains a + // feature token declares nothing. "multimodal" contains no + // feature; "smtp" does contain "mtp" as a substring and must + // not count either. The mail model is the larger build, so a + // false positive would hand it the win. options := []gallery.VariantOption{ tagged("m-mail-q8", gib(24), "llm", "multimodal", "smtp"), tagged("m-turbo-q4", gib(14), "llm", "mtp"), diff --git a/gallery/index.yaml b/gallery/index.yaml index 5b48b2519..70205d322 100644 --- a/gallery/index.yaml +++ b/gallery/index.yaml @@ -59,6 +59,7 @@ tags: - llm - gguf + - mtp overrides: backend: llama-cpp function: @@ -1123,6 +1124,7 @@ - vision - multimodal - reasoning + - mtp overrides: backend: llama-cpp function: @@ -1157,6 +1159,7 @@ tags: - llm - gguf + - mtp icon: https://raw.githubusercontent.com/zai-org/GLM-5/refs/heads/main/resources/bench_52.png overrides: backend: llama-cpp @@ -1246,7 +1249,6 @@ - llm - gguf - qwen - - mtp icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.6/Figures/qwen3.6_35b_a3b_score.png overrides: backend: llama-cpp @@ -1284,7 +1286,6 @@ tags: - llm - gguf - - mtp overrides: backend: llama-cpp # NVFP4 GGUFs use a quant type the GGUF metadata parser cannot read, so @@ -1314,7 +1315,6 @@ tags: - llm - gguf - - mtp icon: https://cdn-uploads.huggingface.co/production/uploads/66309bd090589b7c65950665/sGQKmrMc6L6guMoaB5_Y2.png overrides: backend: llama-cpp @@ -1368,7 +1368,6 @@ tags: - llm - gguf - - mtp icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.6/Figures/qwen3.6_27b_score.png overrides: backend: llama-cpp diff --git a/pkg/system/capabilities.go b/pkg/system/capabilities.go index 5756af663..7286e2ee0 100644 --- a/pkg/system/capabilities.go +++ b/pkg/system/capabilities.go @@ -56,9 +56,8 @@ const ( // Serving feature names (private). A third vocabulary again: not a build // tag and not an engine, but the inference-time strategy a published build - // enables. They are matched against a gallery entry's declared TAGS, and - // failing that against its ENTRY NAME, so they are spelled the way gallery - // authors spell them in both places. + // enables. They are matched against a gallery entry's declared TAGS and + // nothing else, so they are spelled the way gallery authors spell a tag. servingFeatureDFlash = "dflash" servingFeatureMTP = "mtp" ) @@ -82,11 +81,10 @@ const ( // picks which build of one model's weights to install. // // - servingFeaturePreferenceTokens holds SERVING FEATURES ("dflash", "mtp"). -// They are matched against a gallery entry's declared TAGS, and failing -// that against WHOLE SEGMENTS of its ENTRY NAME, rather than against a -// backend or an engine. Same consumer as the engine table, applied one rank -// below it. See the token list for why the name half matches segments -// rather than substrings. +// They are matched against a gallery entry's declared TAGS, whole and +// case-insensitively, rather than against a backend, an engine or the entry +// name. Same consumer as the engine table, applied one rank below it. See +// the token list for why a tag is the only signal. // // Feeding build tags to the variant ranker matches nothing, which does not // error: every candidate simply scores equal and size alone decides, so the @@ -218,25 +216,31 @@ var defaultEnginePreferenceTokens = []string{} // ordering to express. Should one ever appear, this becomes a rule table like // its neighbours without any consumer changing. // -// A token is matched first against an entry's declared TAGS, compared whole -// and case-insensitively. A tag is the authoritative declaration: an author -// writing "mtp" in a tag list means the feature, so it can be compared exactly -// without the ambiguity free text carries. +// A token is matched against an entry's declared TAGS, compared whole and +// case-insensitively, and against NOTHING ELSE. A tag is the declaration: an +// author writing "mtp" in a tag list means the feature, so it can be compared +// exactly without the ambiguity free text carries. // -// Failing a tag it is matched against WHOLE SEGMENTS of the entry name, -// splitting on every non-alphanumeric run, rather than the substring matching -// the tables above use. The other two vocabularies are closed sets that LocalAI -// itself defines, whereas an entry name is author-supplied free text where a -// short token can turn up inside an unrelated word. Segment matching also keeps -// this honest about what it cannot know: "qwen3.6-27b-mtp-pi-tune" is a -// separate finetune and not a variant of anything, so it never reaches ranking -// at all, but if a future finetune were grouped, the name is all there is to go -// on when nobody tagged it. +// Two rejected alternatives, because both look like obvious improvements: // -// The name half stays because tagging discipline is still being established: -// dropping it would immediately misrank every MTP entry that predates the tag -// sweep. Authors should tag; the fallback exists so a forgotten tag costs -// nothing rather than silently downgrading an install. +// - The ENTRY NAME. It was the original signal and is now gone. A naming +// convention is not a contract, and a name is author-supplied free text +// where a short marker turns up inside unrelated words ("smtp-assistant"), +// or names weights that carry MTP heads without the entry enabling +// anything ("qwen3.6-27b-nvfp4-mtp", whose only option is use_jinja). +// Reading a name infers a capability nobody declared. +// +// - overrides.options, where "spec_type:draft-mtp" is what actually turns +// the feature on. That spelling is llama.cpp's config vocabulary. ds4 +// spells the same feature "mtp_path" and sglang spells it +// "speculative_algorithm" in a referenced config file, so keying a +// cross-backend ranking decision on one backend's option syntax would +// rank the other backends' builds as plain. +// +// Options are the CURATION-TIME check instead: a gallery entry earns the tag +// when it configures the feature in whatever vocabulary its backend uses. The +// rule is written down in .agents/adding-gallery-models.md, and the +// backend-specific syntax never reaches the selection logic. var servingFeaturePreferenceTokens = []string{servingFeatureDFlash, servingFeatureMTP} // ServingFeaturePreferenceTokens returns the serving features to prefer, best diff --git a/pkg/system/capabilities_test.go b/pkg/system/capabilities_test.go index 8d0ffe8f4..0844e52f4 100644 --- a/pkg/system/capabilities_test.go +++ b/pkg/system/capabilities_test.go @@ -352,8 +352,8 @@ var _ = Describe("ServingFeaturePreferenceTokens", func() { It("carries neither a build tag nor an engine name", func() { // The third vocabulary, and the same merge hazard as the other two. A - // build tag or an engine name here would be matched against gallery - // entry names, where it means nothing. + // build tag or an engine name here would be matched against a gallery + // entry's declared tags, where it means nothing. Expect(ServingFeaturePreferenceTokens()).ToNot(ContainElements( "cuda", "rocm", "hip", "sycl", "metal", "vulkan", "cpu", "darwin-x86", "vllm", "sglang", "llama-cpp", "mlx",