mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 18:09:05 -04:00
feat(gallery): make the mtp tag authoritative for serving-feature ranking
Variant auto-selection ranks survivors by fit, then engine, then serving feature, then size. The serving-feature lookup read only whole alphanumeric segments of a variant's entry name, because tags were inconsistent: every dflash entry carried a dflash tag, but only 7 of 20 MTP entries carried an mtp tag. Tag the 13 untagged MTP entries, then teach the lookup to read tags as well as names. A tag is now the authoritative signal and is compared whole and case-insensitively, which is safe precisely because a tag is a deliberate declaration rather than free text: there is no word-inside-a-word failure mode, so the segment splitting the name half needs is unnecessary there. The name check stays as a fallback rather than being replaced. Switching to tags only would have regressed the six already-grouped entries on the day it shipped, and would depend on tagging discipline that does not exist yet. The lookup still names no feature, so adding one remains a one-line edit to servingFeaturePreferenceTokens. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
@@ -241,7 +241,7 @@ vocabularies and are matched against different things:
|
||||
|-------|-----------|-----------------|----------|
|
||||
| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` |
|
||||
| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking |
|
||||
| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery ENTRY NAME, as a whole segment | gallery variant ranking, one rank below the engine |
|
||||
| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole; failing that its ENTRY NAME, as a whole segment | gallery variant ranking, one rank below the engine |
|
||||
|
||||
**Putting a token in the wrong table matches nothing and does not error**: every
|
||||
candidate scores equal and the next sort key decides, so the preference silently
|
||||
@@ -249,10 +249,11 @@ stops existing. The block comment above all three tables spells the contract out
|
||||
|
||||
The serving feature table is the odd one: it is not keyed by capability, because
|
||||
no hardware prefers a plain build over an equivalent faster build of the same
|
||||
weights, and it matches whole name segments rather than substrings, because
|
||||
entry names are author-supplied free text where a short marker like `mtp` can
|
||||
turn up inside an unrelated word. A backend never needs to appear in it; it
|
||||
ranks builds, not engines.
|
||||
weights. It reads a declared tag first, since a tag is a deliberate statement,
|
||||
and falls back to whole name segments rather than substrings, because entry
|
||||
names are author-supplied free text where a short marker like `mtp` can turn up
|
||||
inside an unrelated word. A backend never needs to appear in it; it ranks
|
||||
builds, not engines.
|
||||
|
||||
Leaving your backend out is a valid choice when no ordering can be justified for
|
||||
it. It then ranks below every known engine and selection falls back to size,
|
||||
|
||||
@@ -141,8 +141,11 @@ Rules:
|
||||
per step beats the plain build of the same weights, because it answers faster
|
||||
for the same output: a `-dflash` entry beats an `-mtp` one, and either beats a
|
||||
plain build. The order lives in `servingFeaturePreferenceTokens`
|
||||
(`pkg/system/capabilities.go`) and is matched against whole segments of the
|
||||
variant's entry name, since no gallery field declares it. Engine deliberately
|
||||
(`pkg/system/capabilities.go`) and is matched against the entry's `tags:`,
|
||||
falling back to whole segments of its entry name. **Tag every speculative or
|
||||
MTP build** (`- dflash` / `- mtp`) so ranking recognises it; the fallback
|
||||
covers entries nobody tagged, but do not rely on it, since a build whose name
|
||||
spells the feature nowhere is invisible without the tag. Engine deliberately
|
||||
outranks it: a serving feature makes the right engine faster, it does not make
|
||||
a wrong engine right. Fit still outranks both, so a drafter pairing (strictly
|
||||
larger than the plain build, since it ships a drafter alongside it) is dropped
|
||||
|
||||
@@ -104,7 +104,10 @@ func variantOptions(models []*GalleryModel, entry *GalleryModel, env ResolveEnv)
|
||||
return nil, fmt.Errorf("model %q references variant %q which declares variants of its own; resolution is a single pass, so those would be silently ignored", entry.Name, v.Model)
|
||||
}
|
||||
|
||||
option := VariantOption{Variant: v, Backend: target.Backend}
|
||||
// Tags come from the REFERENCED entry, not from the declaring one: the
|
||||
// serving feature being ranked is a property of the build that would be
|
||||
// installed, and a parent's tags describe the model family.
|
||||
option := VariantOption{Variant: v, Backend: target.Backend, Tags: target.Tags}
|
||||
if env.ProbeMemory != nil {
|
||||
option.ProbedMemory = env.ProbeMemory(target)
|
||||
}
|
||||
@@ -118,6 +121,7 @@ func variantOptions(models []*GalleryModel, entry *GalleryModel, env ResolveEnv)
|
||||
Variant: Variant{Model: entry.Name},
|
||||
Backend: entry.Backend,
|
||||
IsBase: true,
|
||||
Tags: entry.Tags,
|
||||
}
|
||||
if env.ProbeMemory != nil {
|
||||
base.ProbedMemory = env.ProbeMemory(entry)
|
||||
|
||||
@@ -44,6 +44,14 @@ type VariantOption struct {
|
||||
// a zero requirement: a probe that cannot reach the network must never be
|
||||
// able to break an install.
|
||||
ProbedMemory uint64
|
||||
// Tags are the referenced entry's declared tags, and they are the
|
||||
// AUTHORITATIVE serving feature signal: a tag is something an author wrote
|
||||
// down on purpose, whereas an entry name only happens to contain a marker.
|
||||
//
|
||||
// Empty does not mean "no features". An entry nobody has tagged yet falls
|
||||
// back to its name, so tagging discipline improves ranking without being a
|
||||
// precondition for it.
|
||||
Tags []string
|
||||
}
|
||||
|
||||
// EffectiveMemory returns this option's memory requirement in bytes and whether
|
||||
@@ -95,14 +103,16 @@ type ResolveEnv struct {
|
||||
EnginePreference []string
|
||||
// ServingFeaturePreference lists the SERVING FEATURES to prefer, best first,
|
||||
// as system.ServingFeaturePreferenceTokens reports them (currently
|
||||
// ["dflash", "mtp"]). A token is matched against whole SEGMENTS of the
|
||||
// variant's ENTRY NAME, splitting on every non-alphanumeric run.
|
||||
// ["dflash", "mtp"]). A token matches when it equals one of the variant's
|
||||
// declared TAGS, or failing that when it equals a whole SEGMENT of its
|
||||
// ENTRY NAME, splitting on every non-alphanumeric run.
|
||||
//
|
||||
// A third vocabulary, matched against a third thing. It is neither a build
|
||||
// tag nor an engine name: these name a way of serving the same weights
|
||||
// faster, and no gallery field declares them, so the entry name is the only
|
||||
// signal there is. pkg/system/capabilities.go documents all three tables
|
||||
// together and justifies matching on segments rather than substrings.
|
||||
// faster. A gallery tag is the declaration and takes precedence; the entry
|
||||
// name remains a fallback for entries nobody has tagged.
|
||||
// pkg/system/capabilities.go documents all three tables together and
|
||||
// justifies why the name half matches segments rather than substrings.
|
||||
//
|
||||
// An empty list ranks every build equally on this axis, which is what every
|
||||
// host looked like before serving features were ranked at all.
|
||||
@@ -145,23 +155,48 @@ func (e ResolveEnv) preferenceRank(backend string) int {
|
||||
})
|
||||
}
|
||||
|
||||
// servingFeatureRank scores a variant's entry name against the preferred
|
||||
// serving feature order, lower being better.
|
||||
// servingFeatureRank scores a variant against the preferred serving feature
|
||||
// order, lower being better.
|
||||
//
|
||||
// It names no feature, exactly as preferenceRank names no engine: the ordered
|
||||
// list comes from pkg/system, so teaching LocalAI about a new serving feature
|
||||
// is a one-line edit to that table and never reaches this file.
|
||||
//
|
||||
// A build naming no feature is a plain build and scores just below the least
|
||||
// preferred known one, which is the whole point: whenever a faster way to serve
|
||||
// the same weights survived the filters, it outranks the plain build.
|
||||
func (e ResolveEnv) servingFeatureRank(entryName string) int {
|
||||
segments := nameSegments(entryName)
|
||||
// Two signals, either of which is enough. A declared TAG is the authoritative
|
||||
// one and is compared whole, because a tag is a deliberate statement rather
|
||||
// than free text: an author who writes "mtp" in a tag list means the feature,
|
||||
// so the substring risk that segment matching exists to avoid does not arise
|
||||
// there. The ENTRY NAME stays as a fallback, matched by whole segment as
|
||||
// before, so an entry nobody has tagged yet ranks exactly the way it did
|
||||
// before tags were read at all. Tags-only would have been a regression on the
|
||||
// day it shipped, since most MTP entries carried no tag.
|
||||
//
|
||||
// A build declaring no feature by either route is a plain build and scores
|
||||
// just below the least preferred known one, which is the whole point: whenever
|
||||
// a faster way to serve the same weights survived the filters, it outranks the
|
||||
// plain build.
|
||||
func (e ResolveEnv) servingFeatureRank(o VariantOption) int {
|
||||
segments := nameSegments(o.Variant.Model)
|
||||
tags := lowercased(o.Tags)
|
||||
return preferenceIndex(e.ServingFeaturePreference, func(token string) bool {
|
||||
return slices.Contains(segments, token)
|
||||
return slices.Contains(tags, token) || slices.Contains(segments, token)
|
||||
})
|
||||
}
|
||||
|
||||
// lowercased folds a tag list for comparison, so a gallery author writing
|
||||
// "MTP" declares the same feature as one writing "mtp". Names are already
|
||||
// folded by nameSegments, and the two signals must not disagree about case.
|
||||
func lowercased(tags []string) []string {
|
||||
if len(tags) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(tags))
|
||||
for _, t := range tags {
|
||||
out = append(out, strings.ToLower(strings.TrimSpace(t)))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// preferenceIndex returns the position of the first token `matches` accepts.
|
||||
//
|
||||
// A subject matching no token scores just below the least preferred known one.
|
||||
@@ -279,7 +314,7 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant
|
||||
memory: memory,
|
||||
rank: rankOf(o, env),
|
||||
preference: env.preferenceRank(o.Backend),
|
||||
feature: env.servingFeatureRank(o.Variant.Model),
|
||||
feature: env.servingFeatureRank(o),
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -485,9 +485,9 @@ var _ = Describe("SelectVariant", func() {
|
||||
Describe("ranking by serving feature", func() {
|
||||
// These are SERVING FEATURES, exactly what
|
||||
// system.ServingFeaturePreferenceTokens reports, and a third vocabulary
|
||||
// after build tags and engine names. They are matched against whole
|
||||
// segments of the variant's ENTRY NAME rather than against a backend,
|
||||
// because nothing on a gallery entry declares them.
|
||||
// after build tags and engine names. They are matched against a
|
||||
// variant's declared TAGS, and failing that against whole segments of
|
||||
// its ENTRY NAME, rather than against a backend.
|
||||
//
|
||||
// Spelled out rather than read from pkg/system so these specs pin the
|
||||
// intended ordering rather than restating whatever the table says.
|
||||
@@ -651,6 +651,170 @@ var _ = Describe("SelectVariant", func() {
|
||||
Expect(selection.Option.Variant.Model).To(Equal("m-q8"))
|
||||
Expect(selection.Reasons).To(BeEmpty())
|
||||
})
|
||||
|
||||
Describe("reading the declared tags", func() {
|
||||
// A tag is the authoritative signal and the name is the fallback,
|
||||
// so each of the two has to be pinned on its own: a spec whose
|
||||
// candidate declares the feature both ways would keep passing after
|
||||
// either half was deleted.
|
||||
tagged := func(model string, probed uint64, tags ...string) gallery.VariantOption {
|
||||
o := option(model, "llama-cpp", probed)
|
||||
o.Tags = tags
|
||||
return o
|
||||
}
|
||||
|
||||
It("prefers a build whose tag declares the feature its name does not", func() {
|
||||
// The case tags exist for. "m-turbo-q8" carries MTP heads but
|
||||
// spells nothing in its name, which is the shape most of the
|
||||
// gallery's MTP entries had before they were tagged. It is also
|
||||
// the smaller build, so only the feature axis can lift it.
|
||||
options := []gallery.VariantOption{
|
||||
tagged("m-turbo-q8", gib(14), "llm", "gguf", "mtp"),
|
||||
tagged("m-plain-q8", gib(20), "llm", "gguf"),
|
||||
base("m-q4", gib(6)),
|
||||
}
|
||||
|
||||
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
|
||||
AvailableMemory: gib(32),
|
||||
EnginePreference: nvidia,
|
||||
ServingFeaturePreference: features,
|
||||
}, "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8"))
|
||||
})
|
||||
|
||||
It("still prefers a build the name alone declares, with no tags at all", func() {
|
||||
// The fallback, pinned independently of the tag path. An entry
|
||||
// nobody has tagged must rank exactly as it did before tags
|
||||
// were read, so deleting the name half fails here alone.
|
||||
options := []gallery.VariantOption{
|
||||
option("m-nvfp4-mtp", "llama-cpp", gib(14)),
|
||||
tagged("m-plain-q8", gib(20), "llm", "gguf"),
|
||||
base("m-q4", gib(6)),
|
||||
}
|
||||
|
||||
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
|
||||
AvailableMemory: gib(32),
|
||||
EnginePreference: nvidia,
|
||||
ServingFeaturePreference: features,
|
||||
}, "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(selection.Option.Variant.Model).To(Equal("m-nvfp4-mtp"))
|
||||
})
|
||||
|
||||
It("matches a tag regardless of the case it was written in", func() {
|
||||
// Gallery tags are author-supplied, so the same declaration
|
||||
// arrives in whatever case the author typed. Name segments are
|
||||
// already folded and the two signals must not disagree.
|
||||
options := []gallery.VariantOption{
|
||||
tagged("m-turbo-q8", gib(14), "LLM", "MTP"),
|
||||
tagged("m-plain-q8", gib(20), "llm", "gguf"),
|
||||
base("m-q4", gib(6)),
|
||||
}
|
||||
|
||||
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
|
||||
AvailableMemory: gib(32),
|
||||
EnginePreference: nvidia,
|
||||
ServingFeaturePreference: features,
|
||||
}, "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8"))
|
||||
})
|
||||
|
||||
It("prefers dflash to mtp when both are declared by tag", func() {
|
||||
// The table's order has to survive the new signal: with neither
|
||||
// name saying anything, only the tags separate these two, and
|
||||
// the MTP build is deliberately the larger one.
|
||||
options := []gallery.VariantOption{
|
||||
tagged("m-alpha-q8", gib(20), "llm", "mtp"),
|
||||
tagged("m-beta-q8", gib(14), "llm", "dflash"),
|
||||
base("m-q4", gib(6)),
|
||||
}
|
||||
|
||||
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
|
||||
AvailableMemory: gib(32),
|
||||
EnginePreference: nvidia,
|
||||
ServingFeaturePreference: features,
|
||||
}, "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(selection.Option.Variant.Model).To(Equal("m-beta-q8"))
|
||||
})
|
||||
|
||||
It("does not read a feature out of an unrelated tag", func() {
|
||||
// Tags are compared whole, exactly as name segments are, so a
|
||||
// tag that merely contains a feature token declares nothing.
|
||||
// "multimodal" contains no feature; "smtp" does contain "mtp"
|
||||
// as a substring and must not count either. The tagged build is
|
||||
// the larger one, so a false positive would hand it the win.
|
||||
options := []gallery.VariantOption{
|
||||
tagged("m-mail-q8", gib(24), "llm", "multimodal", "smtp"),
|
||||
tagged("m-turbo-q4", gib(14), "llm", "mtp"),
|
||||
base("m-q4", gib(6)),
|
||||
}
|
||||
|
||||
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
|
||||
AvailableMemory: gib(64),
|
||||
EnginePreference: nvidia,
|
||||
ServingFeaturePreference: features,
|
||||
}, "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4"))
|
||||
})
|
||||
|
||||
It("does not let a tag rescue a build that does not fit", func() {
|
||||
// Fit still outranks the feature axis, whichever signal
|
||||
// declared it.
|
||||
options := []gallery.VariantOption{
|
||||
tagged("m-turbo-f16", gib(48), "llm", "mtp"),
|
||||
tagged("m-plain-q8", gib(12), "llm"),
|
||||
base("m-q4", gib(6)),
|
||||
}
|
||||
|
||||
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
|
||||
AvailableMemory: gib(16),
|
||||
EnginePreference: nvidia,
|
||||
ServingFeaturePreference: features,
|
||||
}, "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(selection.Option.Variant.Model).To(Equal("m-plain-q8"))
|
||||
Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-turbo-f16")))
|
||||
})
|
||||
|
||||
It("lets the host engine preference outrank a tagged serving feature", func() {
|
||||
// Engine beats feature no matter which signal carried the
|
||||
// feature: a tag does not make a wrong engine right.
|
||||
options := []gallery.VariantOption{
|
||||
tagged("m-turbo-q8", gib(24), "llm", "mtp"),
|
||||
option("m-vllm-awq", "vllm", gib(8)),
|
||||
base("m-q4", gib(6)),
|
||||
}
|
||||
|
||||
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
|
||||
AvailableMemory: gib(64),
|
||||
EnginePreference: nvidia,
|
||||
ServingFeaturePreference: features,
|
||||
}, "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(selection.Option.Variant.Model).To(Equal("m-vllm-awq"))
|
||||
})
|
||||
|
||||
It("ignores tags entirely when no feature preference is configured", func() {
|
||||
// The axis stops discriminating with an empty list, tags or no
|
||||
// tags, and selection degrades to size alone as it always did.
|
||||
options := []gallery.VariantOption{
|
||||
tagged("m-turbo-q8", gib(14), "llm", "mtp"),
|
||||
tagged("m-plain-q8", gib(20), "llm"),
|
||||
base("m-q4", gib(6)),
|
||||
}
|
||||
|
||||
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
|
||||
AvailableMemory: gib(32),
|
||||
EnginePreference: nvidia,
|
||||
}, "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(selection.Option.Variant.Model).To(Equal("m-plain-q8"))
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("falling back to the base", func() {
|
||||
|
||||
@@ -118,6 +118,46 @@ var _ = Describe("ResolveVariant", func() {
|
||||
Expect(resolved.URL).To(Equal("file://gguf.yaml"))
|
||||
})
|
||||
|
||||
It("carries the referenced entry's tags into the serving feature ranking", func() {
|
||||
// The tags that rank a variant are the REFERENCED entry's, not the
|
||||
// declaring entry's: the feature belongs to the build that would be
|
||||
// installed. Only the GGUF build here is tagged with a feature, and it
|
||||
// is deliberately the smaller of the two, so it can only win if the
|
||||
// lookup read a tag off the entry rather than off the variant name or
|
||||
// off the parent.
|
||||
speculative := gallery.GalleryModel{}
|
||||
speculative.Name = "qwen3-8b-gguf-turbo"
|
||||
speculative.URL = "file://turbo.yaml"
|
||||
speculative.Backend = "llama-cpp"
|
||||
speculative.Tags = []string{"llm", "mtp"}
|
||||
|
||||
parent := gallery.GalleryModel{}
|
||||
parent.Name = "qwen3-8b-gguf-q4"
|
||||
parent.URL = "file://gguf.yaml"
|
||||
parent.Backend = "llama-cpp"
|
||||
parent.Variants = []gallery.Variant{{Model: "qwen3-8b-vllm-awq"}, {Model: "qwen3-8b-gguf-turbo"}}
|
||||
|
||||
catalog := []*gallery.GalleryModel{models[0], &speculative, &parent}
|
||||
|
||||
resolved, variant, err := gallery.ResolveVariant(catalog, &parent, gallery.ResolveEnv{
|
||||
AvailableMemory: gib(64),
|
||||
BackendCompatible: runsEverything,
|
||||
// Only llama-cpp is ranked, so the vLLM build cannot win on engine
|
||||
// and the contest comes down to the feature axis.
|
||||
EnginePreference: []string{"llama-cpp"},
|
||||
ServingFeaturePreference: []string{"dflash", "mtp"},
|
||||
ProbeMemory: func(target *gallery.GalleryModel) uint64 {
|
||||
if target.Name == "qwen3-8b-gguf-turbo" {
|
||||
return gib(10)
|
||||
}
|
||||
return gib(20)
|
||||
},
|
||||
}, "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(variant.Model).To(Equal("qwen3-8b-gguf-turbo"))
|
||||
Expect(resolved.URL).To(Equal("file://turbo.yaml"))
|
||||
})
|
||||
|
||||
It("falls back to the entry's own payload when no variant fits", func() {
|
||||
resolved, variant, err := gallery.ResolveVariant(models, base, env(gib(8)), "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
@@ -423,6 +423,7 @@
|
||||
- gguf
|
||||
- vision
|
||||
- multimodal
|
||||
- mtp
|
||||
icon: https://cdn-uploads.huggingface.co/production/uploads/66309bd090589b7c65950665/ztbyGV_zGhzcLuTCSVyq3.png
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
@@ -458,6 +459,7 @@
|
||||
tags:
|
||||
- llm
|
||||
- gguf
|
||||
- mtp
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
function:
|
||||
@@ -618,6 +620,7 @@
|
||||
- llm
|
||||
- gguf
|
||||
- reasoning
|
||||
- mtp
|
||||
icon: https://storage.ko-fi.com/cdn/kofi6.png
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
@@ -689,6 +692,7 @@
|
||||
- vision
|
||||
- multimodal
|
||||
- reasoning
|
||||
- mtp
|
||||
icon: https://cdn-uploads.huggingface.co/production/uploads/66309bd090589b7c65950665/sGQKmrMc6L6guMoaB5_Y2.png
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
@@ -1242,6 +1246,7 @@
|
||||
- llm
|
||||
- gguf
|
||||
- qwen
|
||||
- mtp
|
||||
icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.6/Figures/qwen3.6_35b_a3b_score.png
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
@@ -1279,6 +1284,7 @@
|
||||
tags:
|
||||
- llm
|
||||
- gguf
|
||||
- mtp
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
# NVFP4 GGUFs use a quant type the GGUF metadata parser cannot read, so
|
||||
@@ -1308,6 +1314,7 @@
|
||||
tags:
|
||||
- llm
|
||||
- gguf
|
||||
- mtp
|
||||
icon: https://cdn-uploads.huggingface.co/production/uploads/66309bd090589b7c65950665/sGQKmrMc6L6guMoaB5_Y2.png
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
@@ -1361,6 +1368,7 @@
|
||||
tags:
|
||||
- llm
|
||||
- gguf
|
||||
- mtp
|
||||
icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.6/Figures/qwen3.6_27b_score.png
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
@@ -1474,6 +1482,7 @@
|
||||
- vision
|
||||
- multimodal
|
||||
- reasoning
|
||||
- mtp
|
||||
icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.6/Figures/qwen3.6_27b_score.png
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
@@ -1716,6 +1725,7 @@
|
||||
- vision
|
||||
- multimodal
|
||||
- reasoning
|
||||
- mtp
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
function:
|
||||
@@ -2544,6 +2554,7 @@
|
||||
- llm
|
||||
- gguf
|
||||
- reasoning
|
||||
- mtp
|
||||
icon: https://cdn-uploads.huggingface.co/production/uploads/66309bd090589b7c65950665/9EnS13MSxNU3snpAgEiLq.jpeg
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
@@ -2582,6 +2593,7 @@
|
||||
- llm
|
||||
- gguf
|
||||
- reasoning
|
||||
- mtp
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
function:
|
||||
@@ -36958,6 +36970,7 @@
|
||||
- gguf
|
||||
- llm
|
||||
- chat
|
||||
- mtp
|
||||
overrides:
|
||||
backend: ds4
|
||||
options:
|
||||
|
||||
@@ -56,8 +56,9 @@ const (
|
||||
|
||||
// Serving feature names (private). A third vocabulary again: not a build
|
||||
// tag and not an engine, but the inference-time strategy a published build
|
||||
// enables. They are matched against a gallery ENTRY NAME, so they are
|
||||
// spelled the way gallery authors spell them in an entry name segment.
|
||||
// enables. They are matched against a gallery entry's declared TAGS, and
|
||||
// failing that against its ENTRY NAME, so they are spelled the way gallery
|
||||
// authors spell them in both places.
|
||||
servingFeatureDFlash = "dflash"
|
||||
servingFeatureMTP = "mtp"
|
||||
)
|
||||
@@ -81,10 +82,11 @@ const (
|
||||
// picks which build of one model's weights to install.
|
||||
//
|
||||
// - servingFeaturePreferenceTokens holds SERVING FEATURES ("dflash", "mtp").
|
||||
// They are matched against WHOLE SEGMENTS of a gallery ENTRY NAME, not
|
||||
// against a backend or an engine, because no field on a gallery entry
|
||||
// declares them. Same consumer as the engine table, applied one rank below
|
||||
// it. See the token list for why segment matching rather than substring.
|
||||
// They are matched against a gallery entry's declared TAGS, and failing
|
||||
// that against WHOLE SEGMENTS of its ENTRY NAME, rather than against a
|
||||
// backend or an engine. Same consumer as the engine table, applied one rank
|
||||
// below it. See the token list for why the name half matches segments
|
||||
// rather than substrings.
|
||||
//
|
||||
// Feeding build tags to the variant ranker matches nothing, which does not
|
||||
// error: every candidate simply scores equal and size alone decides, so the
|
||||
@@ -216,20 +218,25 @@ var defaultEnginePreferenceTokens = []string{}
|
||||
// ordering to express. Should one ever appear, this becomes a rule table like
|
||||
// its neighbours without any consumer changing.
|
||||
//
|
||||
// Matching is against WHOLE SEGMENTS of the entry name, splitting on every
|
||||
// non-alphanumeric run, rather than the substring matching the tables above
|
||||
// use. The other two vocabularies are closed sets that LocalAI itself defines,
|
||||
// whereas an entry name is author-supplied free text where a short token can
|
||||
// turn up inside an unrelated word. Segment matching also keeps this honest
|
||||
// about what it cannot know: "qwen3.6-27b-mtp-pi-tune" is a separate finetune
|
||||
// and not a variant of anything, so it never reaches ranking at all, but if a
|
||||
// future finetune were grouped, the name is all there is to go on either way.
|
||||
// A token is matched first against an entry's declared TAGS, compared whole
|
||||
// and case-insensitively. A tag is the authoritative declaration: an author
|
||||
// writing "mtp" in a tag list means the feature, so it can be compared exactly
|
||||
// without the ambiguity free text carries.
|
||||
//
|
||||
// The name is the signal because nothing else is reliable. Tags would be the
|
||||
// natural declaration, but they are optional and demonstrably inconsistent:
|
||||
// "gemma-4-e2b-it:sglang-mtp" carries an "mtp" tag while "ornith-1.0-9b-mtp"
|
||||
// and "qwen3.6-27b-nvfp4-mtp" carry none, so keying on tags would rank two of
|
||||
// those three as plain builds.
|
||||
// Failing a tag it is matched against WHOLE SEGMENTS of the entry name,
|
||||
// splitting on every non-alphanumeric run, rather than the substring matching
|
||||
// the tables above use. The other two vocabularies are closed sets that LocalAI
|
||||
// itself defines, whereas an entry name is author-supplied free text where a
|
||||
// short token can turn up inside an unrelated word. Segment matching also keeps
|
||||
// this honest about what it cannot know: "qwen3.6-27b-mtp-pi-tune" is a
|
||||
// separate finetune and not a variant of anything, so it never reaches ranking
|
||||
// at all, but if a future finetune were grouped, the name is all there is to go
|
||||
// on when nobody tagged it.
|
||||
//
|
||||
// The name half stays because tagging discipline is still being established:
|
||||
// dropping it would immediately misrank every MTP entry that predates the tag
|
||||
// sweep. Authors should tag; the fallback exists so a forgotten tag costs
|
||||
// nothing rather than silently downgrading an install.
|
||||
var servingFeaturePreferenceTokens = []string{servingFeatureDFlash, servingFeatureMTP}
|
||||
|
||||
// ServingFeaturePreferenceTokens returns the serving features to prefer, best
|
||||
|
||||
Reference in New Issue
Block a user