mirror of
https://github.com/mudler/LocalAI.git
synced 2026-08-02 11:30:44 -04:00
d7eb059f44
Gallery entries could already carry a list of alternatives, but selection was
an authored, ordered, first-match policy: every candidate declared a
`capability` string and the VRAM floors had to descend in a hand-tuned order.
That pushed hardware knowledge onto whoever edits the gallery and made ordering
load-bearing, so a reordered list silently changed what users installed.
None of it was necessary. SystemState.IsBackendCompatible already derives
hardware support from a backend name alone: it knows MLX and metal are
Darwin-only, CUDA is NVIDIA-only, ROCm AMD-only, SYCL Intel-only. Selection can
read that instead of asking authors to restate it.
Authoring is now just a list of names:
- name: qwen3.6-27b
min_memory: 4GiB
variants:
- model: qwen3.6-27b-mlx-8bit
- model: qwen3.6-27b-gguf-q8
min_memory: 28GiB
and all the intelligence moved into the selector. Given a host it drops the
variants whose backend cannot run here, drops those whose known memory
requirement exceeds what the host has, and takes the LARGEST of what is left,
because a bigger footprint is a higher quality quantization of the same model.
A variant of unknown size is kept, since nothing proves it does not fit, but it
ranks last so a proven fit always beats a guess. An explicit pin still wins
outright, and if nothing survives the entry installs its own payload: the base
always installs, this never refuses.
Available memory is VRAM when a GPU was detected and system RAM otherwise, read
through xsysinfo so a cgroup limit is honored and a container gets its own
limit rather than the node's RAM.
Capability disappears entirely, from the types, the schema and the lint. VRAM
and RAM collapse into one `min_memory`, because a model's footprint is roughly
the same wherever it lives and one figure is compared against whichever applies.
The lint rules about ordering, the capability vocabulary and floor
relationships are deleted with the hazards they described; what remains is that
every variant names an entry that exists and does not itself declare variants,
plus that any memory figure actually parses.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
116 lines
3.9 KiB
JSON
116 lines
3.9 KiB
JSON
{
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"$id": "https://raw.githubusercontent.com/mudler/LocalAI/main/schemas/gallery.model.schema.json",
|
|
"title": "LocalAI Gallery Model Spec",
|
|
"description": "Schema for LocalAI gallery model YAML files",
|
|
"type": "object",
|
|
|
|
"properties": {
|
|
"name": {
|
|
"type": "string",
|
|
"description": "Model name"
|
|
},
|
|
"description": {
|
|
"type": "string",
|
|
"description": "Human-readable description of the model"
|
|
},
|
|
"icon": {
|
|
"type": "string",
|
|
"description": "Optional icon reference or URL"
|
|
},
|
|
"license": {
|
|
"type": "string",
|
|
"description": "Model license identifier or text"
|
|
},
|
|
"urls": {
|
|
"type": "array",
|
|
"description": "URLs pointing to remote model configuration",
|
|
"items": {
|
|
"type": "string",
|
|
"format": "uri"
|
|
}
|
|
},
|
|
"config_file": {
|
|
"type": "string",
|
|
"description": "Inline YAML configuration that will be written to the model config file"
|
|
},
|
|
"files": {
|
|
"type": "array",
|
|
"description": "Files to download and install for this model",
|
|
"items": {
|
|
"type": "object",
|
|
"required": ["filename", "uri"],
|
|
"properties": {
|
|
"filename": {
|
|
"type": "string"
|
|
},
|
|
"sha256": {
|
|
"type": "string",
|
|
"description": "Optional SHA256 checksum for file verification"
|
|
},
|
|
"uri": {
|
|
"type": "string",
|
|
"format": "uri"
|
|
}
|
|
},
|
|
"additionalProperties": false
|
|
}
|
|
},
|
|
"min_memory": {
|
|
"type": "string",
|
|
"description": "This entry's own memory requirement, for example 2GiB. One figure covers both VRAM and system RAM; the installer compares it against whichever the host will use. Advisory for the entry itself: falling short only warns."
|
|
},
|
|
"variants": {
|
|
"type": "array",
|
|
"description": "Other builds of the same model (other backends, other quantizations) the installer may pick instead of this entry's own payload. Order carries no meaning: the installer drops what this host cannot run or does not have the memory for, then takes the largest of what is left, installing it under this entry's own name. If nothing is left this entry installs itself. Clients that predate variants support drop this key and install the entry unchanged.",
|
|
"minItems": 1,
|
|
"items": {
|
|
"type": "object",
|
|
"required": ["model"],
|
|
"properties": {
|
|
"model": {
|
|
"type": "string",
|
|
"description": "Name of a gallery entry that declares no variants of its own"
|
|
},
|
|
"min_memory": {
|
|
"type": "string",
|
|
"description": "Authored memory requirement, for example 20GiB. Optional, and only needed to override an inferred estimate that is wrong. Authoritative: inference never overwrites it."
|
|
},
|
|
"backend": {
|
|
"type": "string",
|
|
"description": "Denormalized by the nightly job for display. Never authored by hand."
|
|
},
|
|
"quantization": {
|
|
"type": "string",
|
|
"description": "Denormalized by the nightly job for display. Never authored by hand."
|
|
},
|
|
"inferred_min_memory": {
|
|
"type": "string",
|
|
"description": "Denormalized by the nightly job as a fallback requirement. Never authored by hand; min_memory wins."
|
|
}
|
|
},
|
|
"additionalProperties": false
|
|
}
|
|
},
|
|
"prompt_templates": {
|
|
"type": "array",
|
|
"description": "Prompt templates written as .tmpl files",
|
|
"items": {
|
|
"type": "object",
|
|
"required": ["name", "content"],
|
|
"properties": {
|
|
"name": {
|
|
"type": "string"
|
|
},
|
|
"content": {
|
|
"type": "string"
|
|
}
|
|
},
|
|
"additionalProperties": false
|
|
}
|
|
}
|
|
},
|
|
|
|
"additionalProperties": false
|
|
}
|