mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-08-01 02:49:51 -04:00
Code Issues Packages Projects Releases Wiki Activity

feat(gallery): make a declared tag the sole serving-feature signal

Browse Source 
Variant auto-selection ranks survivors by fit, then engine, then serving
feature, then size. The serving-feature lookup recognised a speculative build
by either a declared tag or a whole segment of its entry name. Drop the name
half: a tag is now the only signal.

A name is author-supplied free text and a naming convention is not a contract,
so reading a marker out of one infers a capability nobody declared. The gallery
already had the failure in it: the four NVFP4 entries name MTP-bearing weights
while setting no option that enables speculative decoding, and being live
variants they were winning the feature axis without answering any faster.

overrides.options was considered as the replacement and rejected. It carries
spec_type:draft-mtp / spec_type:draft-dflash, which is what actually turns the
feature on, but that spelling is llama.cpp's config vocabulary: ds4 spells the
same feature mtp_path and sglang spells it speculative_algorithm in a
referenced config. Keying a cross-backend ranking decision on one backend's
option syntax would rank the other backends' builds as plain. Options are the
curation-time check instead, and never reach the selection logic.

With no fallback left, tag correctness is load bearing, so audit every entry
against the rule "tagged when the entry configures that feature, in whatever
vocabulary its backend uses". Three entries configure MTP untagged and gain the
tag (hy3, glm-5.2, qwythos-9b-claude-mythos-5-1m, all spec_type:draft-mtp with
no marker in their names). Four carry the tag while configuring nothing and
lose it: qwen3.6-27b-nvfp4-mtp, qwen3.6-35b-a3b-nvfp4-mtp,
qwopus3.6-27b-coder-mtp-nvfp4 and qwopus3.6-27b-v2-mtp-nvfp4, whose only option
is use_jinja:true. The dflash side was checked independently rather than assumed
consistent: all five dflash entries declare spec_type:draft-dflash and all five
are tagged, so it needed no edits.

Four entries keep a tag that a literal spec_type-only reading would strip,
because they configure MTP through a different backend: deepseek-v4-flash-q2-mtp
via ds4's mtp_path/mtp_draft, and the three sglang entries via
speculative_algorithm in their referenced configs. Stripping those would
contradict the reason spec_type was rejected as the signal and would demote four
genuinely faster builds to plain.

The index was edited by line insertion and deletion only, never round-tripped
through a serializer. A resolved-tag diff across all 1272 named entries, taken
after merge keys are applied, shows exactly these 7 changing and no entry
gaining or losing a tag through an anchor.

The two specs that pinned the name fallback are inverted rather than deleted,
since a name silently promoting a build is the regression worth guarding. The
whole-token guard survives on the tag path, where smtp must still not match mtp.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
Ettore Di Giacinto
2026-07-20 07:01:19 +00:00
parent 20ea4feba3
commit 72260d8b03
7 changed files with 182 additions and 150 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
.agents/adding-backends.md
View File

@@ -241,7 +241,7 @@ vocabularies and are matched against different things:
|-------|-----------|-----------------|----------|
| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` |
| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking |
| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole; failing that its ENTRY NAME, as a whole segment | gallery variant ranking, one rank below the engine |
| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole and case-insensitively, and nothing else | gallery variant ranking, one rank below the engine |
**Putting a token in the wrong table matches nothing and does not error**: every
candidate scores equal and the next sort key decides, so the preference silently
@@ -249,11 +249,19 @@ stops existing. The block comment above all three tables spells the contract out
The serving feature table is the odd one: it is not keyed by capability, because
no hardware prefers a plain build over an equivalent faster build of the same
weights. It reads a declared tag first, since a tag is a deliberate statement,
and falls back to whole name segments rather than substrings, because entry
names are author-supplied free text where a short marker like `mtp` can turn up
inside an unrelated word. A backend never needs to appear in it; it ranks
builds, not engines.
weights. It reads a declared tag and nothing else. The entry name was the
original signal and is gone: a naming convention is not a contract, and names
are author-supplied free text where a short marker like `mtp` turns up inside
unrelated words or on weights whose entry enables nothing.
`overrides.options` was rejected for the mirror-image reason: `spec_type:` is
llama.cpp's config vocabulary, whereas a cross-backend ranking decision must
work the same for `ds4`'s `mtp_path:` and `sglang`'s `speculative_algorithm:`.
**If your backend can serve the same weights faster** (speculative decoding,
multi-token prediction), say so in the docs for its gallery entries so curators
tag them: the tagging rule and the per-backend evidence table live in
[adding-gallery-models.md](adding-gallery-models.md). A backend never needs to
appear in the token table itself; it ranks builds, not engines.
Leaving your backend out is a valid choice when no ordering can be justified for
it. It then ranks below every known engine and selection falls back to size,

40
.agents/adding-gallery-models.md
View File

@@ -139,13 +139,11 @@ Rules:
- **Serving feature preference sits between engine and size.** Among builds on
an equally preferred engine, one that speculates or predicts several tokens
per step beats the plain build of the same weights, because it answers faster
for the same output: a `-dflash` entry beats an `-mtp` one, and either beats a
for the same output: a `dflash` build beats an `mtp` one, and either beats a
plain build. The order lives in `servingFeaturePreferenceTokens`
(`pkg/system/capabilities.go`) and is matched against the entry's `tags:`,
falling back to whole segments of its entry name. **Tag every speculative or
MTP build** (`- dflash` / `- mtp`) so ranking recognises it; the fallback
covers entries nobody tagged, but do not rely on it, since a build whose name
spells the feature nowhere is invisible without the tag. Engine deliberately
(`pkg/system/capabilities.go`) and is matched against the entry's `tags:` and
**nothing else**: not the entry name, not `overrides.options`. See
[the tagging rule](#the-dflash--mtp-tagging-rule) below. Engine deliberately
outranks it: a serving feature makes the right engine faster, it does not make
a wrong engine right. Fit still outranks both, so a drafter pairing (strictly
larger than the plain build, since it ships a drafter alongside it) is dropped
@@ -163,6 +161,36 @@ Users can override the automatic choice with `variant` on `POST /models/apply`,
The gallery lint specs live in `core/gallery`, so run that suite after adding a
`variants` list.
### The `dflash` / `mtp` tagging rule
**Tag an entry `dflash` or `mtp` when the entry actually configures that
feature. Variant ranking reads the tag and nothing else.**
Decide by looking at what the entry configures, in whatever vocabulary its
backend uses:
| Backend | Configures the feature when it declares |
|---------|------------------------------------------|
| `llama-cpp` | `overrides.options` contains `spec_type:draft-dflash` or `spec_type:draft-mtp` |
| `ds4` | `overrides.options` contains `mtp_path:` / `mtp_draft:` |
| `sglang` | the referenced `gallery/*.yaml` sets `speculative_algorithm:` |
That check is curation-time only. `spec_type` is llama.cpp's config vocabulary,
and a cross-backend ranking decision must not depend on one backend's option
syntax, which is precisely why the ranker reads the tag instead of the options.
Two mistakes the rule exists to prevent:
- **Weights that carry the heads are not an entry that enables them.** The
NVFP4 GGUF entries ship MTP-bearing weights but set only `use_jinja:true`, so
they enable no speculative decoding and must NOT be tagged. Tagging them wins
them the feature axis without being any faster.
- **A name is not a declaration.** An entry whose name spells `-mtp` while
configuring nothing gets no tag, and an entry that configures the feature is
tagged even when its name says nothing (`hy3`, `glm-5.2`). Ranking never reads
the name, so an untagged build that does enable the feature is simply ranked
as plain rather than promoted on a marker nobody meant.
## Available template configs
Look at existing `.yaml` files in `gallery/` to find the right prompt template for your model architecture:

72
core/gallery/resolve_variant.go
View File

@@ -6,7 +6,6 @@ import (
"slices"
"sort"
"strings"
"unicode"
)
var (
@@ -44,13 +43,17 @@ type VariantOption struct {
// a zero requirement: a probe that cannot reach the network must never be
// able to break an install.
ProbedMemory uint64
// Tags are the referenced entry's declared tags, and they are the
// AUTHORITATIVE serving feature signal: a tag is something an author wrote
// down on purpose, whereas an entry name only happens to contain a marker.
// Tags are the referenced entry's declared tags, and they are the SOLE
// serving feature signal. A tag is something an author wrote down on
// purpose, whereas an entry name only happens to contain a marker, so
// nothing else is consulted.
//
// Empty does not mean "no features". An entry nobody has tagged yet falls
// back to its name, so tagging discipline improves ranking without being a
// precondition for it.
// Empty therefore means "declares no serving feature", and an entry that
// enables one without saying so ranks as a plain build. That is the cost of
// making the signal a declaration, and it is the right cost: an install
// silently downgraded to the plain build is recoverable, whereas a build
// promoted on a marker nobody meant is not visible at all.
// .agents/adding-gallery-models.md states the tagging rule.
Tags []string
}
@@ -104,15 +107,13 @@ type ResolveEnv struct {
// ServingFeaturePreference lists the SERVING FEATURES to prefer, best first,
// as system.ServingFeaturePreferenceTokens reports them (currently
// ["dflash", "mtp"]). A token matches when it equals one of the variant's
// declared TAGS, or failing that when it equals a whole SEGMENT of its
// ENTRY NAME, splitting on every non-alphanumeric run.
// declared TAGS, compared whole and case-insensitively. Nothing else is
// matched: not the entry name, not its backend options.
//
// A third vocabulary, matched against a third thing. It is neither a build
// tag nor an engine name: these name a way of serving the same weights
// faster. A gallery tag is the declaration and takes precedence; the entry
// name remains a fallback for entries nobody has tagged.
// pkg/system/capabilities.go documents all three tables together and
// justifies why the name half matches segments rather than substrings.
// faster. pkg/system/capabilities.go documents all three tables together
// and justifies why this one reads a declaration rather than free text.
//
// An empty list ranks every build equally on this axis, which is what every
// host looked like before serving features were ranked at all.
@@ -162,30 +163,30 @@ func (e ResolveEnv) preferenceRank(backend string) int {
// list comes from pkg/system, so teaching LocalAI about a new serving feature
// is a one-line edit to that table and never reaches this file.
//
// Two signals, either of which is enough. A declared TAG is the authoritative
// one and is compared whole, because a tag is a deliberate statement rather
// than free text: an author who writes "mtp" in a tag list means the feature,
// so the substring risk that segment matching exists to avoid does not arise
// there. The ENTRY NAME stays as a fallback, matched by whole segment as
// before, so an entry nobody has tagged yet ranks exactly the way it did
// before tags were read at all. Tags-only would have been a regression on the
// day it shipped, since most MTP entries carried no tag.
// A declared TAG is the ONLY signal, compared whole and case-insensitively. An
// entry name is deliberately not consulted: a naming convention is not a
// contract, and an author is free to spell a build however they like, so
// reading a marker out of a name infers a capability nobody declared. Tags are
// a LocalAI-level vocabulary the gallery curators control and they mean the
// same thing on every backend. `overrides.options` was considered as the
// signal and rejected for the opposite reason: spec_type:draft-mtp is what
// actually enables the feature, but that spelling is llama.cpp's config
// vocabulary, and a cross-backend ranking decision must not depend on one
// backend's option syntax. Curators check the options at tagging time; the
// ranker reads only the tag.
//
// A build declaring no feature by either route is a plain build and scores
// just below the least preferred known one, which is the whole point: whenever
// a faster way to serve the same weights survived the filters, it outranks the
// plain build.
// A build declaring no feature is a plain build and scores just below the
// least preferred known one, which is the whole point: whenever a faster way
// to serve the same weights survived the filters, it outranks the plain build.
func (e ResolveEnv) servingFeatureRank(o VariantOption) int {
segments := nameSegments(o.Variant.Model)
tags := lowercased(o.Tags)
return preferenceIndex(e.ServingFeaturePreference, func(token string) bool {
return slices.Contains(tags, token) || slices.Contains(segments, token)
return slices.Contains(tags, token)
})
}
// lowercased folds a tag list for comparison, so a gallery author writing
// "MTP" declares the same feature as one writing "mtp". Names are already
// folded by nameSegments, and the two signals must not disagree about case.
// "MTP" declares the same feature as one writing "mtp".
func lowercased(tags []string) []string {
if len(tags) == 0 {
return nil
@@ -215,19 +216,6 @@ func preferenceIndex(tokens []string, matches func(token string) bool) int {
return len(tokens)
}
// nameSegments splits a gallery entry name into its lowercased alphanumeric
// runs, so "qwen3.6-27b-nvfp4-mtp" yields ["qwen3", "6", "27b", "nvfp4", "mtp"].
//
// Whole-segment matching is what keeps a short marker from matching inside an
// unrelated word. Entry names are author-supplied free text, unlike the engine
// names preferenceRank matches as substrings, and those are a closed vocabulary
// LocalAI defines itself.
func nameSegments(name string) []string {
return strings.FieldsFunc(strings.ToLower(name), func(r rune) bool {
return !unicode.IsLetter(r) && !unicode.IsDigit(r)
})
}
// VariantSelection is the outcome of a selection pass.
type VariantSelection struct {
Option VariantOption

135
core/gallery/resolve_variant_test.go
View File

@@ -486,14 +486,23 @@ var _ = Describe("SelectVariant", func() {
// These are SERVING FEATURES, exactly what
// system.ServingFeaturePreferenceTokens reports, and a third vocabulary
// after build tags and engine names. They are matched against a
// variant's declared TAGS, and failing that against whole segments of
// its ENTRY NAME, rather than against a backend.
// variant's declared TAGS and against nothing else.
//
// Spelled out rather than read from pkg/system so these specs pin the
// intended ordering rather than restating whatever the table says.
features := []string{"dflash", "mtp"}
nvidia := []string{"vllm", "sglang", "llama-cpp"}
// A tag is the whole signal, so every candidate that is meant to carry
// a feature declares it here. Candidates built with `option` carry no
// tags and are therefore plain builds no matter what their name says,
// which several specs below rely on.
tagged := func(model string, probed uint64, tags ...string) gallery.VariantOption {
o := option(model, "llama-cpp", probed)
o.Tags = tags
return o
}
It("prefers a speculative build to the plain build of the same weights", func() {
// The rule the feature table exists to express. Both builds fit and
// both run on the same engine, and the plain build is deliberately
@@ -501,8 +510,8 @@ var _ = Describe("SelectVariant", func() {
// drafter pairing would never be installed. Emptying
// ServingFeaturePreference fails this spec.
options := []gallery.VariantOption{
option("m-dflash", "llama-cpp", gib(14)),
option("m-q8", "llama-cpp", gib(20)),
tagged("m-turbo-q4", gib(14), "llm", "dflash"),
tagged("m-q8", gib(20), "llm"),
base("m-q4", gib(6)),
}
@@ -512,7 +521,7 @@ var _ = Describe("SelectVariant", func() {
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-dflash"))
Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4"))
})
It("takes the larger plain build once the feature preference is unknown", func() {
@@ -520,8 +529,8 @@ var _ = Describe("SelectVariant", func() {
// the DFlash win comes from the feature list rather than from
// anything intrinsic to the set.
options := []gallery.VariantOption{
option("m-dflash", "llama-cpp", gib(14)),
option("m-q8", "llama-cpp", gib(20)),
tagged("m-turbo-q4", gib(14), "llm", "dflash"),
tagged("m-q8", gib(20), "llm"),
base("m-q4", gib(6)),
}
@@ -538,8 +547,8 @@ var _ = Describe("SelectVariant", func() {
// separate them, and the MTP build is deliberately the larger one so
// size cannot be what decides.
options := []gallery.VariantOption{
option("m-nvfp4-mtp", "llama-cpp", gib(20)),
option("m-dflash", "llama-cpp", gib(14)),
tagged("m-alpha-q8", gib(20), "llm", "mtp"),
tagged("m-beta-q4", gib(14), "llm", "dflash"),
base("m-q4", gib(6)),
}
@@ -549,7 +558,7 @@ var _ = Describe("SelectVariant", func() {
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-dflash"))
Expect(selection.Option.Variant.Model).To(Equal("m-beta-q4"))
})
It("does not let a preferred feature rescue a build that does not fit", func() {
@@ -557,8 +566,8 @@ var _ = Describe("SelectVariant", func() {
// is the ordinary case on a small host rather than a corner one. Fit
// is a filter; the feature preference only ranks survivors.
options := []gallery.VariantOption{
option("m-dflash", "llama-cpp", gib(48)),
option("m-q8", "llama-cpp", gib(12)),
tagged("m-turbo-f16", gib(48), "llm", "dflash"),
tagged("m-q8", gib(12), "llm"),
base("m-q4", gib(6)),
}
@@ -569,7 +578,7 @@ var _ = Describe("SelectVariant", func() {
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-q8"))
Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-dflash")))
Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-turbo-f16")))
})
It("lets the host engine preference outrank the serving feature", func() {
@@ -578,7 +587,7 @@ var _ = Describe("SelectVariant", func() {
// beats a DFlash llama.cpp build even though both fit and the
// llama.cpp one is larger.
options := []gallery.VariantOption{
option("m-dflash", "llama-cpp", gib(24)),
tagged("m-turbo-q8", gib(24), "llm", "dflash"),
option("m-vllm-awq", "vllm", gib(8)),
base("m-q4", gib(6)),
}
@@ -596,9 +605,9 @@ var _ = Describe("SelectVariant", func() {
// Neither preference key may flatten size ordering: with one engine
// and one feature in play there is nothing left for them to decide.
options := []gallery.VariantOption{
option("m-dflash-q4", "llama-cpp", gib(8)),
option("m-dflash-q8", "llama-cpp", gib(14)),
option("m-dflash-f16", "llama-cpp", gib(48)),
tagged("m-turbo-q4", gib(8), "llm", "dflash"),
tagged("m-turbo-q8", gib(14), "llm", "dflash"),
tagged("m-turbo-f16", gib(48), "llm", "dflash"),
base("m-q4", gib(2)),
}
@@ -608,29 +617,7 @@ var _ = Describe("SelectVariant", func() {
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-dflash-q8"))
})
It("does not read a feature token out of the middle of an unrelated word", func() {
// The guard on matching whole name segments rather than substrings.
// "smtp-assistant" is a mail model and carries no MTP heads, yet
// strings.Contains would rank it as an MTP build, and being the
// larger of the two it would then win. Entry names are
// author-supplied free text, unlike the closed engine vocabulary
// preferenceRank matches as substrings.
options := []gallery.VariantOption{
option("m-smtp-assistant-q8", "llama-cpp", gib(24)),
option("m-nvfp4-mtp", "llama-cpp", gib(14)),
base("m-q4", gib(6)),
}
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
AvailableMemory: gib(64),
EnginePreference: nvidia,
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-nvfp4-mtp"))
Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8"))
})
It("leaves an unfeatured build ranked last rather than dropping it", func() {
@@ -653,16 +640,6 @@ var _ = Describe("SelectVariant", func() {
})
Describe("reading the declared tags", func() {
// A tag is the authoritative signal and the name is the fallback,
// so each of the two has to be pinned on its own: a spec whose
// candidate declares the feature both ways would keep passing after
// either half was deleted.
tagged := func(model string, probed uint64, tags ...string) gallery.VariantOption {
o := option(model, "llama-cpp", probed)
o.Tags = tags
return o
}
It("prefers a build whose tag declares the feature its name does not", func() {
// The case tags exist for. "m-turbo-q8" carries MTP heads but
// spells nothing in its name, which is the shape most of the
@@ -683,10 +660,16 @@ var _ = Describe("SelectVariant", func() {
Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8"))
})
It("still prefers a build the name alone declares, with no tags at all", func() {
// The fallback, pinned independently of the tag path. An entry
// nobody has tagged must rank exactly as it did before tags
// were read, so deleting the name half fails here alone.
It("does NOT prefer a build that only its name declares, with no tags at all", func() {
// The inversion of the old name-fallback spec, and the
// regression this change exists to guard. A name is
// author-supplied free text and a naming convention is not a
// contract: "m-nvfp4-mtp" is exactly the shape of the gallery's
// NVFP4 entries, whose weights carry MTP heads while the entry
// enables no speculative decoding at all, so ranking it as a
// speculative build made it win the feature axis without being
// any faster. With no tag it is a plain build and the larger
// plain build takes it on size.
options := []gallery.VariantOption{
option("m-nvfp4-mtp", "llama-cpp", gib(14)),
tagged("m-plain-q8", gib(20), "llm", "gguf"),
@@ -699,13 +682,35 @@ var _ = Describe("SelectVariant", func() {
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-nvfp4-mtp"))
Expect(selection.Option.Variant.Model).To(Equal("m-plain-q8"))
})
It("lets a tagged build beat a larger one whose name says the same feature", func() {
// The sharper form of the spec above: the name half is not
// merely unnecessary, it is not consulted. The untagged
// "m-dflash" would outrank the tagged MTP build on the old
// name-first ordering, and is the larger build besides, so it
// wins on either of the two ways the name could still be read.
options := []gallery.VariantOption{
option("m-dflash", "llama-cpp", gib(20)),
tagged("m-turbo-q4", gib(14), "llm", "mtp"),
base("m-q4", gib(6)),
}
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
AvailableMemory: gib(32),
EnginePreference: nvidia,
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4"))
})
It("matches a tag regardless of the case it was written in", func() {
// Gallery tags are author-supplied, so the same declaration
// arrives in whatever case the author typed. Name segments are
// already folded and the two signals must not disagree.
// arrives in whatever case the author typed. A curator who
// writes "MTP" has declared the feature just as plainly as one
// who writes "mtp", and the sole signal must not turn on that.
options := []gallery.VariantOption{
tagged("m-turbo-q8", gib(14), "LLM", "MTP"),
tagged("m-plain-q8", gib(20), "llm", "gguf"),
@@ -722,9 +727,9 @@ var _ = Describe("SelectVariant", func() {
})
It("prefers dflash to mtp when both are declared by tag", func() {
// The table's order has to survive the new signal: with neither
// name saying anything, only the tags separate these two, and
// the MTP build is deliberately the larger one.
// The table's order has to survive on the tag path with both
// features spelled out explicitly, and the MTP build is
// deliberately the larger one so size cannot be what decides.
options := []gallery.VariantOption{
tagged("m-alpha-q8", gib(20), "llm", "mtp"),
tagged("m-beta-q8", gib(14), "llm", "dflash"),
@@ -741,11 +746,11 @@ var _ = Describe("SelectVariant", func() {
})
It("does not read a feature out of an unrelated tag", func() {
// Tags are compared whole, exactly as name segments are, so a
// tag that merely contains a feature token declares nothing.
// "multimodal" contains no feature; "smtp" does contain "mtp"
// as a substring and must not count either. The tagged build is
// the larger one, so a false positive would hand it the win.
// Tags are compared whole, so a tag that merely contains a
// feature token declares nothing. "multimodal" contains no
// feature; "smtp" does contain "mtp" as a substring and must
// not count either. The mail model is the larger build, so a
// false positive would hand it the win.
options := []gallery.VariantOption{
tagged("m-mail-q8", gib(24), "llm", "multimodal", "smtp"),
tagged("m-turbo-q4", gib(14), "llm", "mtp"),

7
gallery/index.yaml
View File

@@ -59,6 +59,7 @@
tags:
- llm
- gguf
- mtp
overrides:
backend: llama-cpp
function:
@@ -1123,6 +1124,7 @@
- vision
- multimodal
- reasoning
- mtp
overrides:
backend: llama-cpp
function:
@@ -1157,6 +1159,7 @@
tags:
- llm
- gguf
- mtp
icon: https://raw.githubusercontent.com/zai-org/GLM-5/refs/heads/main/resources/bench_52.png
overrides:
backend: llama-cpp
@@ -1246,7 +1249,6 @@
- llm
- gguf
- qwen
- mtp
icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.6/Figures/qwen3.6_35b_a3b_score.png
overrides:
backend: llama-cpp
@@ -1284,7 +1286,6 @@
tags:
- llm
- gguf
- mtp
overrides:
backend: llama-cpp
# NVFP4 GGUFs use a quant type the GGUF metadata parser cannot read, so
@@ -1314,7 +1315,6 @@
tags:
- llm
- gguf
- mtp
icon: https://cdn-uploads.huggingface.co/production/uploads/66309bd090589b7c65950665/sGQKmrMc6L6guMoaB5_Y2.png
overrides:
backend: llama-cpp
@@ -1368,7 +1368,6 @@
tags:
- llm
- gguf
- mtp
icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.6/Figures/qwen3.6_27b_score.png
overrides:
backend: llama-cpp

54
pkg/system/capabilities.go
View File

@@ -56,9 +56,8 @@ const (
// Serving feature names (private). A third vocabulary again: not a build
// tag and not an engine, but the inference-time strategy a published build
// enables. They are matched against a gallery entry's declared TAGS, and
// failing that against its ENTRY NAME, so they are spelled the way gallery
// authors spell them in both places.
// enables. They are matched against a gallery entry's declared TAGS and
// nothing else, so they are spelled the way gallery authors spell a tag.
servingFeatureDFlash = "dflash"
servingFeatureMTP = "mtp"
)
@@ -82,11 +81,10 @@ const (
// picks which build of one model's weights to install.
//
// - servingFeaturePreferenceTokens holds SERVING FEATURES ("dflash", "mtp").
// They are matched against a gallery entry's declared TAGS, and failing
// that against WHOLE SEGMENTS of its ENTRY NAME, rather than against a
// backend or an engine. Same consumer as the engine table, applied one rank
// below it. See the token list for why the name half matches segments
// rather than substrings.
// They are matched against a gallery entry's declared TAGS, whole and
// case-insensitively, rather than against a backend, an engine or the entry
// name. Same consumer as the engine table, applied one rank below it. See
// the token list for why a tag is the only signal.
//
// Feeding build tags to the variant ranker matches nothing, which does not
// error: every candidate simply scores equal and size alone decides, so the
@@ -218,25 +216,31 @@ var defaultEnginePreferenceTokens = []string{}
// ordering to express. Should one ever appear, this becomes a rule table like
// its neighbours without any consumer changing.
//
// A token is matched first against an entry's declared TAGS, compared whole
// and case-insensitively. A tag is the authoritative declaration: an author
// writing "mtp" in a tag list means the feature, so it can be compared exactly
// without the ambiguity free text carries.
// A token is matched against an entry's declared TAGS, compared whole and
// case-insensitively, and against NOTHING ELSE. A tag is the declaration: an
// author writing "mtp" in a tag list means the feature, so it can be compared
// exactly without the ambiguity free text carries.
//
// Failing a tag it is matched against WHOLE SEGMENTS of the entry name,
// splitting on every non-alphanumeric run, rather than the substring matching
// the tables above use. The other two vocabularies are closed sets that LocalAI
// itself defines, whereas an entry name is author-supplied free text where a
// short token can turn up inside an unrelated word. Segment matching also keeps
// this honest about what it cannot know: "qwen3.6-27b-mtp-pi-tune" is a
// separate finetune and not a variant of anything, so it never reaches ranking
// at all, but if a future finetune were grouped, the name is all there is to go
// on when nobody tagged it.
// Two rejected alternatives, because both look like obvious improvements:
//
// The name half stays because tagging discipline is still being established:
// dropping it would immediately misrank every MTP entry that predates the tag
// sweep. Authors should tag; the fallback exists so a forgotten tag costs
// nothing rather than silently downgrading an install.
// - The ENTRY NAME. It was the original signal and is now gone. A naming
// convention is not a contract, and a name is author-supplied free text
// where a short marker turns up inside unrelated words ("smtp-assistant"),
// or names weights that carry MTP heads without the entry enabling
// anything ("qwen3.6-27b-nvfp4-mtp", whose only option is use_jinja).
// Reading a name infers a capability nobody declared.
//
// - overrides.options, where "spec_type:draft-mtp" is what actually turns
// the feature on. That spelling is llama.cpp's config vocabulary. ds4
// spells the same feature "mtp_path" and sglang spells it
// "speculative_algorithm" in a referenced config file, so keying a
// cross-backend ranking decision on one backend's option syntax would
// rank the other backends' builds as plain.
//
// Options are the CURATION-TIME check instead: a gallery entry earns the tag
// when it configures the feature in whatever vocabulary its backend uses. The
// rule is written down in .agents/adding-gallery-models.md, and the
// backend-specific syntax never reaches the selection logic.
var servingFeaturePreferenceTokens = []string{servingFeatureDFlash, servingFeatureMTP}
// ServingFeaturePreferenceTokens returns the serving features to prefer, best

4
pkg/system/capabilities_test.go
View File

@@ -352,8 +352,8 @@ var _ = Describe("ServingFeaturePreferenceTokens", func() {
It("carries neither a build tag nor an engine name", func() {
// The third vocabulary, and the same merge hazard as the other two. A
// build tag or an engine name here would be matched against gallery
// entry names, where it means nothing.
// build tag or an engine name here would be matched against a gallery
// entry's declared tags, where it means nothing.
Expect(ServingFeaturePreferenceTokens()).ToNot(ContainElements(
"cuda", "rocm", "hip", "sycl", "metal", "vulkan", "cpu", "darwin-x86",
"vllm", "sglang", "llama-cpp", "mlx",