mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-31 10:28:43 -04:00
Code Issues Packages Projects Releases Wiki Activity

gallery: prefer speculative-decoding builds when they fit

Browse Source 
Rank serving features between engine preference and size, so a host that can
hold a DFlash or MTP build of a model's weights installs it instead of the
plain build. Both answer faster for the same output, so whenever one survives
the filters there is no reason to take the plain build.

Precedence is now fit, then engine, then serving feature, then size. Engine
outranks the feature deliberately: a serving feature makes the right engine
faster, it does not make a wrong engine right, so a plain vLLM build still
beats a DFlash llama.cpp build on NVIDIA. Fit outranks both, and a drafter
pairing is strictly larger than the plain build, so the existing size filter
drops it on a host too small for it before this axis is consulted.

The order lives in a third preference table in pkg/system, alongside the build
tag and engine name tables. It is the odd one of the three: not keyed by
capability, because no hardware prefers a plain build over an equivalent
faster one, and matched against whole segments of a gallery ENTRY NAME rather
than as a substring of a backend value. Nothing on a gallery entry declares a
serving feature, and tags are not a usable substitute: gemma-4-e2b-it:sglang-mtp
carries an mtp tag while ornith-1.0-9b-mtp and qwen3.6-27b-nvfp4-mtp carry
none. Entry names are author-supplied free text, unlike the closed engine
vocabulary, so a short marker can turn up inside an unrelated word and whole
segment matching is what keeps smtp-assistant from ranking as an MTP build.
The block comment over the tables now documents all three together and states
what each is matched against; the ranking code names no feature, so adding one
stays a one-line edit to the table.

29c49203b rejected these entries as serving configurations rather than
alternative builds of the same weights. The definition is now "alternative ways
to serve the same model", which includes them, so regroup 14 entries under 12
parents. Judged by the files each entry points at: the qwen3.6, qwen3.5, qwen3
and deepseek pairings are the base GGUF plus a drafter, the gemma-4 QAT MTP
entries are the same QAT weights at a different quantization plus an MTP
drafter, and the two sglang MTP entries describe themselves as the same model
served with speculative decoding. Left separate: qwen3.6-27b-mtp-pi-tune, a
finetune with its own weights, and every entry whose base model LocalAI does
not ship as its own row, which is the whole Qwopus line plus gemmable-4-12b-mtp,
mimo-7b-mtp:sglang and qwen3.5-4b-dflash.

None of the twelve parents defines a YAML anchor, so no variants key can leak
through a merge key and no empty override was needed this time. The index was
edited by line insertion only.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
Ettore Di Giacinto
2026-07-19 22:43:13 +00:00
parent 2070b002be
commit 460ee79f3c
8 changed files with 446 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
.agents/adding-backends.md
View File

@@ -220,8 +220,9 @@ docker-build-backends: ... docker-build-<backend-name>
A gallery entry can declare `variants`, alternative builds of the same weights,
and LocalAI picks one per host: it drops builds whose backend cannot run here or
that do not fit memory, then ranks the survivors by **engine preference first,
size second** (`SelectVariant` in `core/gallery/resolve_variant.go`).
that do not fit memory, then ranks the survivors by **engine preference
first, serving feature second, size third** (`SelectVariant` in
`core/gallery/resolve_variant.go`).
Ask whether your backend should outrank another one on some hardware. If it
should, add it to `engineNamePreferenceRules` in `pkg/system/capabilities.go`,
@@ -232,13 +233,26 @@ best engine first for that capability:
+ {Nvidia, []string{engineVLLM, engineSGLang, engineMyEngine, engineLlamaCpp}},
```
That is the ENGINE NAME table, matched against a gallery entry's `backend:`
value. The `backendBuildTagPreferenceRules` table next to it is a different
vocabulary: build tags (`cuda`, `rocm`, `metal`) matched against installed build
directory names for alias resolution. **Putting an engine name in the build tag
table, or a build tag in the engine table, matches nothing and does not error**:
every candidate scores equal and size alone decides, so the preference silently
stops existing. The block comment above both tables spells the contract out.
That is the ENGINE NAME table, matched as a substring of a gallery entry's
`backend:` value. Two sibling tables in the same file speak different
vocabularies and are matched against different things:
| Table | Vocabulary | Matched against | Consumer |
|-------|-----------|-----------------|----------|
| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` |
| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking |
| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery ENTRY NAME, as a whole segment | gallery variant ranking, one rank below the engine |
**Putting a token in the wrong table matches nothing and does not error**: every
candidate scores equal and the next sort key decides, so the preference silently
stops existing. The block comment above all three tables spells the contract out.
The serving feature table is the odd one: it is not keyed by capability, because
no hardware prefers a plain build over an equivalent faster build of the same
weights, and it matches whole name segments rather than substrings, because
entry names are author-supplied free text where a short marker like `mtp` can
turn up inside an unrelated word. A backend never needs to appear in it; it
ranks builds, not engines.
Leaving your backend out is a valid choice when no ordering can be justified for
it. It then ranks below every known engine and selection falls back to size,
@@ -292,7 +306,7 @@ After adding a new backend, verify:
- [ ] No Makefile syntax errors (check with linter)
- [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern)
- [ ] **`Load` validates its input and refuses models it can't serve.** When a model config has no explicit `backend:`, the model loader greedily probes *every* installed backend with the model's name and binds to the first `Load` that succeeds — an accept-anything `Load` will capture arbitrary LLMs (issue #9287). Backends that load a real artefact get this for free (the load fails); backends with no artefact must gate on the name: `opus` accepts only its own name (or none), `local-store` requires the `store.NamespacePrefix` namespace marker sent by `core/backend/stores.go`.
- [ ] **Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets size decide.
- [ ] **Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`, NOT `servingFeaturePreferenceTokens`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets the next sort key decide.
- [ ] Documented: added to the category list in `docs/content/features/backends.md` (and any new endpoint/realtime capability documented under `docs/content/`)
- [ ] If it is an in-house native C/C++/GGML engine, added to the maintained-engines table in the top-level `README.md`

11
.agents/adding-gallery-models.md
View File

@@ -136,6 +136,17 @@ Rules:
per-capability order lives in `engineNamePreferenceRules`
(`pkg/system/capabilities.go`); see
[adding-backends.md](adding-backends.md) for how a backend gets into it.
- **Serving feature preference sits between engine and size.** Among builds on
an equally preferred engine, one that speculates or predicts several tokens
per step beats the plain build of the same weights, because it answers faster
for the same output: a `-dflash` entry beats an `-mtp` one, and either beats a
plain build. The order lives in `servingFeaturePreferenceTokens`
(`pkg/system/capabilities.go`) and is matched against whole segments of the
variant's entry name, since no gallery field declares it. Engine deliberately
outranks it: a serving feature makes the right engine faster, it does not make
a wrong engine right. Fit still outranks both, so a drafter pairing (strictly
larger than the plain build, since it ships a drafter alongside it) is dropped
on a host too small for it before this order is ever consulted.
- A variant is nothing but a name; there is no per-variant memory field. When
the measured size for a build is wrong, correct it on the referenced entry by
setting that entry's own `size:` (e.g. `size: "20GiB"`). The estimator prefers

4
core/gallery/models.go
View File

@@ -283,6 +283,10 @@ func HostResolveEnv(ctx context.Context, systemState *system.SystemState) Resolv
// matched on its gallery `backend:` engine name, and the latter reports
// build tags that no engine name contains.
EnginePreference: systemState.EnginePreferenceTokens(),
// The third vocabulary, and the only one that is not host derived: no
// hardware prefers a plain build over an equivalent faster one, so this
// comes from a package function rather than from systemState.
ServingFeaturePreference: system.ServingFeaturePreferenceTokens(),
ProbeMemory: func(target *GalleryModel) uint64 {
return probeEntryMemory(ctx, target)
},

109
core/gallery/resolve_variant.go
View File

@@ -3,8 +3,10 @@ package gallery
import (
"errors"
"fmt"
"slices"
"sort"
"strings"
"unicode"
)
var (
@@ -91,6 +93,20 @@ type ResolveEnv struct {
// view of the hardware, and it is what every host looked like before
// preference existed.
EnginePreference []string
// ServingFeaturePreference lists the SERVING FEATURES to prefer, best first,
// as system.ServingFeaturePreferenceTokens reports them (currently
// ["dflash", "mtp"]). A token is matched against whole SEGMENTS of the
// variant's ENTRY NAME, splitting on every non-alphanumeric run.
//
// A third vocabulary, matched against a third thing. It is neither a build
// tag nor an engine name: these name a way of serving the same weights
// faster, and no gallery field declares them, so the entry name is the only
// signal there is. pkg/system/capabilities.go documents all three tables
// together and justifies matching on segments rather than substrings.
//
// An empty list ranks every build equally on this axis, which is what every
// host looked like before serving features were ranked at all.
ServingFeaturePreference []string
// ProbeMemory measures how much memory a referenced gallery entry needs,
// without downloading it. A zero result means "could not tell", never
// "needs nothing".
@@ -123,16 +139,58 @@ func (e ResolveEnv) backendRuns(backend string) bool {
// an unrecognised backend, an unrecognised capability and an absent preference
// list all degrade to the same predictable place: ordering by size alone.
func (e ResolveEnv) preferenceRank(backend string) int {
if len(e.EnginePreference) == 0 {
name := strings.ToLower(backend)
return preferenceIndex(e.EnginePreference, func(token string) bool {
return strings.Contains(name, token)
})
}
// servingFeatureRank scores a variant's entry name against the preferred
// serving feature order, lower being better.
//
// It names no feature, exactly as preferenceRank names no engine: the ordered
// list comes from pkg/system, so teaching LocalAI about a new serving feature
// is a one-line edit to that table and never reaches this file.
//
// A build naming no feature is a plain build and scores just below the least
// preferred known one, which is the whole point: whenever a faster way to serve
// the same weights survived the filters, it outranks the plain build.
func (e ResolveEnv) servingFeatureRank(entryName string) int {
segments := nameSegments(entryName)
return preferenceIndex(e.ServingFeaturePreference, func(token string) bool {
return slices.Contains(segments, token)
})
}
// preferenceIndex returns the position of the first token `matches` accepts.
//
// A subject matching no token scores just below the least preferred known one.
// It is never dropped, and a list where nothing matches scores uniformly, so an
// unrecognised subject and an absent preference list degrade to the same
// predictable place: this axis stops discriminating and the next key decides.
func preferenceIndex(tokens []string, matches func(token string) bool) int {
if len(tokens) == 0 {
return 0
}
name := strings.ToLower(backend)
for i, token := range e.EnginePreference {
if token != "" && strings.Contains(name, token) {
for i, token := range tokens {
if token != "" && matches(token) {
return i
}
}
return len(e.EnginePreference)
return len(tokens)
}
// nameSegments splits a gallery entry name into its lowercased alphanumeric
// runs, so "qwen3.6-27b-nvfp4-mtp" yields ["qwen3", "6", "27b", "nvfp4", "mtp"].
//
// Whole-segment matching is what keeps a short marker from matching inside an
// unrelated word. Entry names are author-supplied free text, unlike the engine
// names preferenceRank matches as substrings, and those are a closed vocabulary
// LocalAI defines itself.
func nameSegments(name string) []string {
return strings.FieldsFunc(strings.ToLower(name), func(r rune) bool {
return !unicode.IsLetter(r) && !unicode.IsDigit(r)
})
}
// VariantSelection is the outcome of a selection pass.
@@ -168,8 +226,11 @@ type VariantSelection struct {
// the largest thing that fits must win against a smaller variant, and a base
// of known size must win against a variant whose size nothing could measure.
// 5. The survivors are ranked and the best one wins: fit tier first (rankOf
// below), then this host's backend preference, then size. Preference beats
// size deliberately, so a Mac takes an MLX build over a larger GGUF one.
// below), then this host's engine preference, then the serving feature
// preference, then size. Both preferences beat size deliberately, so a Mac
// takes an MLX build over a larger GGUF one, and a host that can hold a
// speculative-decoding build of the same weights takes it over the plain
// one.
// 6. With no survivor at all, which can only happen when the caller supplied no
// base, there is nothing to install and this reports ErrNoVariantMatch.
func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (VariantSelection, error) {
@@ -187,6 +248,7 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant
memory uint64
rank int
preference int
feature int
}
survivors := make([]ranked, 0, len(options))
@@ -217,6 +279,7 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant
memory: memory,
rank: rankOf(o, env),
preference: env.preferenceRank(o.Backend),
feature: env.servingFeatureRank(o.Variant.Model),
})
}
@@ -227,7 +290,7 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant
)
}
// Three keys, in this order, and the order is the whole design:
// Four keys, in this order, and the order is the whole design:
//
// 1. rank, the fit tier. Fit is a fact about whether the host can hold the
// build at all, so nothing outranks it.
@@ -236,12 +299,24 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant
// the smaller download. A Mac given an MLX build and a larger llama.cpp
// build should run MLX; ranking by size first would install the GGUF and
// leave the accelerated build unused. Do NOT move this below size.
// 3. memory, largest first, because among builds on equally preferred
// runtimes a bigger build is a higher quality quantization of the same
// model. Preference must not flatten this: two builds on one backend are
// still ordered by size.
// 3. feature, the serving feature order. Among builds on an equally
// preferred runtime, one that speculates or predicts several tokens per
// step beats the plain build of the same weights, because it answers
// faster for the same output.
//
// Stable so that options equal on all three keep their authored order, which
// Engine outranks this deliberately. A serving feature makes the right
// engine faster; it does not make a wrong engine right. Wearing the
// wrong engine costs the whole GPU, whereas a plain build on the native
// engine is merely slower than it could be, so a DFlash build on the
// portable engine must not beat a plain build on the engine this host
// actually serves well.
// 4. memory, largest first, because among builds on equally preferred
// runtimes with the same serving feature a bigger build is a higher
// quality quantization of the same model. Neither preference key may
// flatten this: two plain builds on one backend are still ordered by
// size.
//
// Stable so that options equal on all four keep their authored order, which
// is the only thing order still decides.
sort.SliceStable(survivors, func(i, j int) bool {
if survivors[i].rank != survivors[j].rank {
@@ -250,6 +325,9 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant
if survivors[i].preference != survivors[j].preference {
return survivors[i].preference < survivors[j].preference
}
if survivors[i].feature != survivors[j].feature {
return survivors[i].feature < survivors[j].feature
}
return survivors[i].memory > survivors[j].memory
})
@@ -263,8 +341,9 @@ func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (Variant
}, nil
}
// Fit tiers, best first. Within a tier the host's preferred runtime wins, and
// within one runtime the larger footprint wins; see the sort in SelectVariant.
// Fit tiers, best first. Within a tier the host's preferred runtime wins, then
// the preferred serving feature, then the larger footprint; see the sort in
// SelectVariant.
const (
// rankProvenFit is a measured size that the host is measured to satisfy.
rankProvenFit = iota

171
core/gallery/resolve_variant_test.go
View File

@@ -482,6 +482,177 @@ var _ = Describe("SelectVariant", func() {
})
})
Describe("ranking by serving feature", func() {
// These are SERVING FEATURES, exactly what
// system.ServingFeaturePreferenceTokens reports, and a third vocabulary
// after build tags and engine names. They are matched against whole
// segments of the variant's ENTRY NAME rather than against a backend,
// because nothing on a gallery entry declares them.
//
// Spelled out rather than read from pkg/system so these specs pin the
// intended ordering rather than restating whatever the table says.
features := []string{"dflash", "mtp"}
nvidia := []string{"vllm", "sglang", "llama-cpp"}
It("prefers a speculative build to the plain build of the same weights", func() {
// The rule the feature table exists to express. Both builds fit and
// both run on the same engine, and the plain build is deliberately
// the larger one, so without this axis size would take it and the
// drafter pairing would never be installed. Emptying
// ServingFeaturePreference fails this spec.
options := []gallery.VariantOption{
option("m-dflash", "llama-cpp", gib(14)),
option("m-q8", "llama-cpp", gib(20)),
base("m-q4", gib(6)),
}
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
AvailableMemory: gib(32),
EnginePreference: nvidia,
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-dflash"))
})
It("takes the larger plain build once the feature preference is unknown", func() {
// The mirror of the spec above on the identical option set, proving
// the DFlash win comes from the feature list rather than from
// anything intrinsic to the set.
options := []gallery.VariantOption{
option("m-dflash", "llama-cpp", gib(14)),
option("m-q8", "llama-cpp", gib(20)),
base("m-q4", gib(6)),
}
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
AvailableMemory: gib(32),
EnginePreference: nvidia,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-q8"))
})
It("prefers dflash to mtp", func() {
// Both are speculative pairings, so only the table's order can
// separate them, and the MTP build is deliberately the larger one so
// size cannot be what decides.
options := []gallery.VariantOption{
option("m-nvfp4-mtp", "llama-cpp", gib(20)),
option("m-dflash", "llama-cpp", gib(14)),
base("m-q4", gib(6)),
}
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
AvailableMemory: gib(32),
EnginePreference: nvidia,
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-dflash"))
})
It("does not let a preferred feature rescue a build that does not fit", func() {
// A drafter pairing is strictly larger than the plain build, so this
// is the ordinary case on a small host rather than a corner one. Fit
// is a filter; the feature preference only ranks survivors.
options := []gallery.VariantOption{
option("m-dflash", "llama-cpp", gib(48)),
option("m-q8", "llama-cpp", gib(12)),
base("m-q4", gib(6)),
}
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
AvailableMemory: gib(16),
EnginePreference: nvidia,
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-q8"))
Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-dflash")))
})
It("lets the host engine preference outrank the serving feature", func() {
// A serving feature makes the right engine faster; it does not make
// a wrong engine right. On nvidia the plain vLLM build therefore
// beats a DFlash llama.cpp build even though both fit and the
// llama.cpp one is larger.
options := []gallery.VariantOption{
option("m-dflash", "llama-cpp", gib(24)),
option("m-vllm-awq", "vllm", gib(8)),
base("m-q4", gib(6)),
}
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
AvailableMemory: gib(64),
EnginePreference: nvidia,
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-vllm-awq"))
})
It("still picks the largest fitting build among equally featured builds", func() {
// Neither preference key may flatten size ordering: with one engine
// and one feature in play there is nothing left for them to decide.
options := []gallery.VariantOption{
option("m-dflash-q4", "llama-cpp", gib(8)),
option("m-dflash-q8", "llama-cpp", gib(14)),
option("m-dflash-f16", "llama-cpp", gib(48)),
base("m-q4", gib(2)),
}
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
AvailableMemory: gib(16),
EnginePreference: nvidia,
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-dflash-q8"))
})
It("does not read a feature token out of the middle of an unrelated word", func() {
// The guard on matching whole name segments rather than substrings.
// "smtp-assistant" is a mail model and carries no MTP heads, yet
// strings.Contains would rank it as an MTP build, and being the
// larger of the two it would then win. Entry names are
// author-supplied free text, unlike the closed engine vocabulary
// preferenceRank matches as substrings.
options := []gallery.VariantOption{
option("m-smtp-assistant-q8", "llama-cpp", gib(24)),
option("m-nvfp4-mtp", "llama-cpp", gib(14)),
base("m-q4", gib(6)),
}
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
AvailableMemory: gib(64),
EnginePreference: nvidia,
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-nvfp4-mtp"))
})
It("leaves an unfeatured build ranked last rather than dropping it", func() {
// Ranking never filters. With only plain builds on offer the axis
// scores them uniformly and selection degrades to size alone.
options := []gallery.VariantOption{
option("m-q8", "llama-cpp", gib(12)),
option("m-q4", "llama-cpp", gib(6)),
base("m-q2", gib(3)),
}
selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{
AvailableMemory: gib(32),
EnginePreference: nvidia,
ServingFeaturePreference: features,
}, "")
Expect(err).ToNot(HaveOccurred())
Expect(selection.Option.Variant.Model).To(Equal("m-q8"))
Expect(selection.Reasons).To(BeEmpty())
})
})
Describe("falling back to the base", func() {
It("selects the base when nothing else fits", func() {
options := []gallery.VariantOption{

25
gallery/index.yaml
View File

@@ -386,6 +386,7 @@
- model: deepseek-v4-flash-q2
- model: deepseek-v4-flash-q2-q4
- model: deepseek-v4-flash-q4-ssd
- model: deepseek-v4-flash-q2-mtp
url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
urls:
- https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF
@@ -508,6 +509,8 @@
sha256: e7a8eafdd8013443b6bcc4b6fb47b2d2025f772d359650b9ceb7d75971e22cad
uri: https://huggingface.co/unsloth/Qwen-AgentWorld-35B-A3B-GGUF/resolve/main/Qwen-AgentWorld-35B-A3B-UD-Q4_K_M.gguf
- name: "ornith-1.0-9b"
variants:
- model: ornith-1.0-9b-mtp
url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
urls:
- https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B-GGUF
@@ -1795,6 +1798,8 @@
sha256: ef269e294502d6ee3722cbf129681b2586c2e6ceb79d0507963c92146e058cd4
uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF/resolve/main/mmproj-F32.gguf
- name: "gemma-4-12b-it-qat-q4_0"
variants:
- model: gemma-4-12b-it-qat-mtp
url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
urls:
- https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf
@@ -1848,6 +1853,8 @@
sha256: e70b0e5cd80323d5d588b4ed06780356b7b1ba03995a4b8164c6ae9db0ff5989
uri: https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf/resolve/main/mmproj-gemma-4-12b-it-qat-q4_0.gguf
- name: "gemma-4-e2b-it-qat-q4_0"
variants:
- model: gemma-4-e2b-it-qat-mtp
url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
urls:
- https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf
@@ -1890,6 +1897,8 @@
sha256: 58c187648007cab392bd5678b87e862c3e8794017deb945feea2cf256195e96a
uri: https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf/resolve/main/gemma-4-E2B-it-mmproj.gguf
- name: "gemma-4-e4b-it-qat-q4_0"
variants:
- model: gemma-4-e4b-it-qat-mtp
url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
urls:
- https://huggingface.co/google/gemma-4-E4B-it-qat-q4_0-gguf
@@ -1975,6 +1984,8 @@
sha256: d8e2de16e17515d9061b23c9a002715f996f9e0c87b93a9354264611bfab9239
uri: https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf/resolve/main/gemma-4-26B-it-mmproj.gguf
- name: "gemma-4-31b-it-qat-q4_0"
variants:
- model: gemma-4-31b-it-qat-mtp
url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
urls:
- https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf
@@ -3151,6 +3162,9 @@
sha256: 13bd039f95c9ea46ef1d75905faa7be6ca4e47a5af9d4cf62e298a738a5b195f
uri: https://huggingface.co/KyleHessling1/Qwopus-GLM-18B-Merged-GGUF/resolve/main/Qwopus-GLM-18B-Healed-Q4_K_M.gguf
- name: qwen3.6-27b
variants:
- model: qwen3.6-27b-dflash
- model: qwen3.6-27b-nvfp4-mtp
url: github:mudler/LocalAI/gallery/virtual.yaml@master
urls:
- https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
@@ -3441,6 +3455,9 @@
sha256: b5aa0676be588bf6ef3bbdb89905d7d239b2a809637f0766a6ce23aed6c6b5b4
uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-Quality.gguf
- name: qwen3.6-35b-a3b
variants:
- model: qwen3.6-35b-a3b-dflash
- model: qwen3.6-35b-a3b-nvfp4-mtp
url: github:mudler/LocalAI/gallery/virtual.yaml@master
urls:
- https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF
@@ -3579,6 +3596,8 @@
sha256: 2aa99ffb47033ead4a3f1584fec5283905302c1c16fed59c99e0eec131c6dc53
uri: huggingface://ggml-org/gemma-4-26B-A4B-it-GGUF/mmproj-gemma-4-26B-A4B-it-bf16.gguf
- name: gemma-4-e2b-it
variants:
- model: "gemma-4-e2b-it:sglang-mtp"
url: github:mudler/LocalAI/gallery/virtual.yaml@master
urls:
- https://huggingface.co/google/gemma-4-E2B-it
@@ -3622,6 +3641,8 @@
sha256: e42083b71a9e31e0f722171d551f6d92b101544001c4dde040306a8f2160fe8c
uri: huggingface://ggml-org/gemma-4-E2B-it-GGUF/mmproj-gemma-4-E2B-it-bf16.gguf
- name: gemma-4-e4b-it
variants:
- model: "gemma-4-e4b-it:sglang-mtp"
url: github:mudler/LocalAI/gallery/virtual.yaml@master
urls:
- https://huggingface.co/google/gemma-4-E4B-it
@@ -4039,6 +4060,8 @@
sha256: 8221b6a48c714db6829a92760c31034d7ecd436f830c61624ccc92b461b4a1c4
uri: https://huggingface.co/mradermacher/Q3.5-BlueStar-27B-GGUF/resolve/main/Q3.5-BlueStar-27B.mmproj-f16.gguf
- name: qwen3.5-9b
variants:
- model: qwen3.5-9b-dflash
url: github:mudler/LocalAI/gallery/virtual.yaml@master
urls:
- https://huggingface.co/unsloth/Qwen3.5-9B-GGUF
@@ -12070,6 +12093,8 @@
sha256: 376902d50612ecfc5bd8b268f376c04d10ad7e480f99a1483b833f04344a549e
uri: huggingface://MaziyarPanahi/Qwen3-8B-GGUF/Qwen3-8B.Q4_K_M.gguf
- name: qwen3-4b
variants:
- model: qwen3-4b-dflash
url: github:mudler/LocalAI/gallery/qwen3.yaml@master
urls:
- https://huggingface.co/Qwen/Qwen3-4B

87
pkg/system/capabilities.go
View File

@@ -53,29 +53,43 @@ const (
engineSGLang = "sglang"
engineLlamaCpp = "llama-cpp"
engineMLX = "mlx"
// Serving feature names (private). A third vocabulary again: not a build
// tag and not an engine, but the inference-time strategy a published build
// enables. They are matched against a gallery ENTRY NAME, so they are
// spelled the way gallery authors spell them in an entry name segment.
servingFeatureDFlash = "dflash"
servingFeatureMTP = "mtp"
)
// There are TWO preference tables below and they speak DIFFERENT VOCABULARIES.
// Merging them looks tempting and silently breaks one of the two consumers,
// because a token that means something in one vocabulary means nothing in the
// other. Read this before editing either.
// There are THREE preference tables below and they speak DIFFERENT VOCABULARIES,
// matched against three different things. Merging any two looks tempting and
// silently breaks a consumer, because a token that means something in one
// vocabulary means nothing in the others. Read this before editing any of them.
//
// - backendBuildTagPreferenceRules holds BUILD TAGS ("cuda", "rocm", "metal").
// They are matched against INSTALLED BACKEND BUILD DIRECTORY NAMES such as
// "llama-cpp-cuda-12" or "cuda12-vllm". Consumer: alias resolution in
// ListSystemBackends (core/gallery/backends.go), which picks which installed
// build of one alias to run.
// They are matched as substrings of INSTALLED BACKEND BUILD DIRECTORY NAMES
// such as "llama-cpp-cuda-12" or "cuda12-vllm". Consumer: alias resolution
// in ListSystemBackends (core/gallery/backends.go), which picks which
// installed build of one alias to run.
//
// - engineNamePreferenceRules holds ENGINE NAMES ("vllm", "llama-cpp", "mlx").
// They are matched against a gallery entry's `backend:` value, which never
// carries a build tag: no entry in gallery/index.yaml contains "cuda",
// "rocm", "sycl" or "vulkan" anywhere in its backend name. Consumer: gallery
// variant auto-selection (core/gallery/resolve_variant.go), which picks
// which build of one model's weights to install.
// They are matched as substrings of a gallery entry's `backend:` value,
// which never carries a build tag: no entry in gallery/index.yaml contains
// "cuda", "rocm", "sycl" or "vulkan" anywhere in its backend name. Consumer:
// gallery variant auto-selection (core/gallery/resolve_variant.go), which
// picks which build of one model's weights to install.
//
// - servingFeaturePreferenceTokens holds SERVING FEATURES ("dflash", "mtp").
// They are matched against WHOLE SEGMENTS of a gallery ENTRY NAME, not
// against a backend or an engine, because no field on a gallery entry
// declares them. Same consumer as the engine table, applied one rank below
// it. See the token list for why segment matching rather than substring.
//
// Feeding build tags to the variant ranker matches nothing, which does not
// error: every candidate simply scores equal and size alone decides, so the
// preference silently stops existing. That is exactly the bug this split fixes.
// A serving feature fed to either of the other two tables fails the same way.
// backendPreferenceRule maps a detected capability to preferred tokens, best
// first. Both tables share this shape; only their vocabulary differs.
@@ -180,6 +194,53 @@ var engineNamePreferenceRules = []backendPreferenceRule{
// list as and is the behaviour that predates preference.
var defaultEnginePreferenceTokens = []string{}
// servingFeaturePreferenceTokens is the SERVING FEATURE table, best first. See
// the block comment above for the vocabulary contract.
//
// A serving feature is a way of running the same weights faster rather than a
// different set of weights: a DFlash build pairs the base GGUF with a drafter
// so the target accepts several speculated tokens per step, and an MTP build
// carries extra prediction heads to the same end. Both produce the same
// distribution as the plain build, so whenever one fits there is no reason to
// install the plain build instead.
//
// DFlash leads because it is the newer pairing and accepts more tokens per step
// in practice; MTP follows; a build naming neither is plain and ranks last.
// A build's extra weights make it strictly larger than the plain build, which
// is why this ranks BELOW fit: the size filter drops the pairing on a host too
// small for it before this list is ever consulted.
//
// Unlike the two tables above this one is NOT keyed by capability. A serving
// feature is a property of the published build, and no hardware prefers the
// plain build over an equivalent faster one, so there is no host-shaped
// ordering to express. Should one ever appear, this becomes a rule table like
// its neighbours without any consumer changing.
//
// Matching is against WHOLE SEGMENTS of the entry name, splitting on every
// non-alphanumeric run, rather than the substring matching the tables above
// use. The other two vocabularies are closed sets that LocalAI itself defines,
// whereas an entry name is author-supplied free text where a short token can
// turn up inside an unrelated word. Segment matching also keeps this honest
// about what it cannot know: "qwen3.6-27b-mtp-pi-tune" is a separate finetune
// and not a variant of anything, so it never reaches ranking at all, but if a
// future finetune were grouped, the name is all there is to go on either way.
//
// The name is the signal because nothing else is reliable. Tags would be the
// natural declaration, but they are optional and demonstrably inconsistent:
// "gemma-4-e2b-it:sglang-mtp" carries an "mtp" tag while "ornith-1.0-9b-mtp"
// and "qwen3.6-27b-nvfp4-mtp" carry none, so keying on tags would rank two of
// those three as plain builds.
var servingFeaturePreferenceTokens = []string{servingFeatureDFlash, servingFeatureMTP}
// ServingFeaturePreferenceTokens returns the serving features to prefer, best
// first, for ranking gallery model variants against each other.
//
// It is a package function rather than a SystemState method precisely because
// the answer does not depend on the host; see the table for why.
func ServingFeaturePreferenceTokens() []string {
return slices.Clone(servingFeaturePreferenceTokens)
}
var (
cuda13DirExists bool
cuda12DirExists bool

43
pkg/system/capabilities_test.go
View File

@@ -342,3 +342,46 @@ var _ = Describe("DetectedCapability", func() {
Expect(state.DetectedCapability()).NotTo(Equal("default"))
})
})
var _ = Describe("ServingFeaturePreferenceTokens", func() {
It("puts dflash ahead of mtp", func() {
// The rule the serving feature table exists to express. Both are ways
// of serving the same weights faster; only this order separates them.
Expect(ServingFeaturePreferenceTokens()).To(Equal([]string{"dflash", "mtp"}))
})
It("carries neither a build tag nor an engine name", func() {
// The third vocabulary, and the same merge hazard as the other two. A
// build tag or an engine name here would be matched against gallery
// entry names, where it means nothing.
Expect(ServingFeaturePreferenceTokens()).ToNot(ContainElements(
"cuda", "rocm", "hip", "sycl", "metal", "vulkan", "cpu", "darwin-x86",
"vllm", "sglang", "llama-cpp", "mlx",
))
})
It("does not depend on the host capability", func() {
// Deliberately not a SystemState method: no hardware prefers the plain
// build over an equivalent faster one, so there is no host-shaped
// ordering to express here.
previous := os.Getenv(capabilityEnv)
defer func() {
if previous != "" {
os.Setenv(capabilityEnv, previous)
} else {
os.Unsetenv(capabilityEnv)
}
}()
Expect(os.Setenv(capabilityEnv, "nvidia-cuda-12")).To(Succeed())
onNvidia := ServingFeaturePreferenceTokens()
Expect(os.Setenv(capabilityEnv, metal)).To(Succeed())
Expect(ServingFeaturePreferenceTokens()).To(Equal(onNvidia))
})
It("hands out a copy so one caller cannot corrupt another's lookup", func() {
first := ServingFeaturePreferenceTokens()
first[0] = "clobbered"
Expect(ServingFeaturePreferenceTokens()[0]).To(Equal("dflash"))
})
})