mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-06 06:17:25 -04:00
dddd76e9b9
When a model is imported without an explicit context_size, the GGUF importer defaulted the model's context to its full trained window (n_ctx_train). For long-context models (128k / 256k / 1M) that KV cache cannot fit a consumer GPU, so the backend aborts on load (exitCode=-1) even though the model file is perfectly fine. Reproduced live: gemma-4-26b-a4b-it-qat-q4_0 defaulted to context=262144 and qwythos-9b-claude-mythos-5-1m to 1048576, both aborting on a 20 GB card. Instead of chasing the trained max, auto-derive a conservative default: min(trainedMax, DefaultAutoContextSize=8192). A small model keeps its trained window; a long-context model caps at 8k and users opt into more via context_size. This cap applies always, including CPU / unknown-VRAM hosts, so it never regresses those paths. Per-device VRAM is used only as a DOWNWARD safety: when a per-device ceiling is detected (xsysinfo.MinPerGPUVRAM) and even the 8k cap would not fit it with headroom, step down through candidate contexts to the largest that fits, floored at DefaultContextSize. When VRAM is unknown (0) or no GPU is detected we do NOT clamp — the bug is GPU OOM and the 8k cap is already safe, so detection gaps must not shrink the window. The footprint estimate reuses gpustack/gguf-parser-go's EstimateLLaMACppRun at a given context with all layers offloaded, taking the per-device NonUMA VRAM figure. The estimate and VRAM detection are package vars so tests inject deterministic values. Explicit context_size always wins (guessGGUFFromFile only acts when it is nil). Assisted-by: Claude:claude-opus-4-8 [golangci-lint go-test]