mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-08-04 12:22:22 -04:00
Code Issues Packages Projects Releases Wiki Activity

perf(gallery): warm variant descriptions alongside VRAM estimates (#11297)

Browse Source 
Follow-up to #11288, which warmed the VRAM estimate caches at startup and left
the variant picker paying its own way.

Describing an entry's variants probes the weight files of every build it
offers, so the first time a model is opened costs 1.2-1.9s against a cold
cache. That is the same cost as an estimate wearing a different hat, and it
lands in the same caches underneath, so it belongs in the same pass rather than
in a second mechanism.

The warm-up now describes variants for the entries it walks. Entries that
declare none cost nothing: the call is gated on HasVariants rather than
attempted and discarded. The host resolve env is derived once for the run,
since it describes the machine rather than the entry.

Failure handling matches the estimate half. An entry whose variants cannot be
described is logged at debug and skipped, and the estimate for that same entry
is unaffected, because neither half is allowed to fail the other.

Measured against a live instance with 1,595 models, first ever call to
/api/models/variants/:id after a cold boot:

  before   1.2-1.9s
  after    2ms

The warm-up's own cost barely moves: 3m0s to 3m19s for 300 entries, of which
40 declared variants. It stays bounded by the same knobs, and
LOCALAI_VRAM_WARM_LIMIT=0 still turns the whole thing off.


Assisted-by: Claude:claude-opus-5 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler's LocalAI [bot]
2026-08-02 19:42:54 +02:00
committed by GitHub
parent 74b7ea2829
commit 1aa97381f3
3 changed files with 61 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
docs/content/advanced/vram-management.md
View File

@@ -463,13 +463,18 @@ takes a second or two the first time, and the gallery needs one per row. LocalAI
caches the result, and warms that cache in the background at startup so the
gallery reads instantly rather than filling in its own numbers while you watch.
The same warm-up also describes each entry's **variants** - the alternative
builds of the same weights that the picker offers - because that costs the same
kind of probe and lands in the same cache. Without it, the first model you open
pays for it again.
The warm-up is bounded, and covers the entries at the top of the gallery: the
ones you see first. Anything past it is estimated on first view and cached from
then on.
| Variable | Default | Meaning |
|---|---|---|
| `LOCALAI_VRAM_WARM_LIMIT` | `300` | How many gallery entries to warm at startup. Set to `0` to disable the warm-up entirely. |
| `LOCALAI_VRAM_WARM_LIMIT` | `300` | How many gallery entries to warm at startup, estimates and variants alike. Set to `0` to disable the warm-up entirely. |
| `LOCALAI_VRAM_WARM_CONCURRENCY` | `4` | How many estimates to run at once. |
```bash