mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 18:09:05 -04:00
d32f45beca
Variant auto-selection ranked candidates with
SystemState.BackendPreferenceTokens, but that function and the variant
ranker speak different vocabularies.
BackendPreferenceTokens returns BUILD TAGS ("cuda", "rocm", "sycl",
"vulkan", "metal", "cpu"). It exists to match installed backend build
directory names like "llama-cpp-cuda-12" during alias resolution in
ListSystemBackends. Variant ranking instead matches a gallery entry's
`backend:` value, which is an ENGINE NAME: "llama-cpp", "vllm",
"vllm-omni", "sglang", "mlx" and the rest. No engine name in
gallery/index.yaml contains "cuda", "rocm", "sycl" or "vulkan".
preferenceRank matches by substring, so on an NVIDIA host the tokens
[cuda, vulkan, cpu] matched neither "llama-cpp" nor "vllm", every
candidate scored identically and size alone decided. The NVIDIA, AMD,
Intel, darwin-x86 and vulkan rules were all inert. Only metal appeared
to work, and only because the token "mlx" happens to equal an engine
name. The mismatch does not error, it silently deletes the feature.
Separate the two vocabularies. backendBuildTagPreferenceRules keeps the
build tags and its original output for every capability, including
metal, whose "mlx" token is removed again; its alias-resolution consumer
is byte-identical to before. engineNamePreferenceRules is new, holds
engine names, and is read by the new EnginePreferenceTokens, which
HostResolveEnv wires into the renamed ResolveEnv.EnginePreference. Both
tables sit adjacent under one block comment naming each vocabulary and
each consumer, and share one lookup helper so their semantics cannot
drift.
On NVIDIA the order is vLLM, then SGLang, then llama-cpp: vLLM is the
throughput engine and a model published with a vLLM build is published
that way because that build is the one worth running. AMD and Intel get
the same order, since rocm and intel builds of both serving engines
ship. Metal prefers mlx over llama-cpp. Vulkan prefers llama-cpp, the
only LLM engine with a Vulkan build. darwin-x86 and unknown
capabilities are deliberately absent rather than guessed at, degrading
to the size-only ordering that predates preference.
preferenceRank stays generic and names no engine and no capability, so
adding a runtime remains a one-line table edit.
Specs pin the NVIDIA and metal rules through the live table and the real
HostResolveEnv wiring, so emptying the engine table or wiring the build
tag source back in both go red. A regression table asserts
BackendPreferenceTokens' original output per capability, and mirrored
locks assert neither table carries the other's vocabulary.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>