mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-08 07:18:31 -04:00
ed3b59baf1
When a model is imported without an explicit context_size, the GGUF importer defaulted the model's context to its full trained window (n_ctx_train). For long-context models (128k / 256k / 1M) that KV cache cannot fit a consumer GPU, so the backend aborts on load (exitCode=-1) even though the model file is perfectly fine. Reproduced live: gemma-4-26b-a4b-it-qat-q4_0 defaulted to context=262144 and qwythos-9b-claude-mythos-5-1m to 1048576, both aborting on a 20 GB card. Instead of chasing the trained max, auto-derive a conservative default: min(trainedMax, DefaultAutoContextSize=8192). A small model keeps its trained window; a long-context model caps at 8k and users opt into more via context_size. This cap applies always, including CPU / unknown-VRAM hosts, so it never regresses those paths. Per-device VRAM is used only as a DOWNWARD safety: when a per-device ceiling is detected (xsysinfo.MinPerGPUVRAM) and even the 8k cap would not fit it with headroom, step down through candidate contexts to the largest that fits, floored at DefaultContextSize. When VRAM is unknown (0) or no GPU is detected we do NOT clamp — the bug is GPU OOM and the 8k cap is already safe, so detection gaps must not shrink the window. The footprint estimate reuses gpustack/gguf-parser-go's EstimateLLaMACppRun at a given context with all layers offloaded, taking the per-device NonUMA VRAM figure. The estimate and VRAM detection are package vars so tests inject deterministic values. Explicit context_size always wins (guessGGUFFromFile only acts when it is nil). Assisted-by: Claude:claude-opus-4-8 [golangci-lint go-test] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
41 lines
2.1 KiB
Go
41 lines
2.1 KiB
Go
package config
|
|
|
|
// Canonical default values.
|
|
//
|
|
// These are owned here so the two layers that need them share a single source
|
|
// of truth: the config tiers (ApplyInference/Hardware/Serving/Generic — which
|
|
// *decide* defaults) and core/backend/options.go (which *translates* a
|
|
// ModelConfig to the backend wire format and supplies the same fallbacks
|
|
// defensively). Previously these were duplicated as literals across both
|
|
// packages and had drifted (e.g. n_gpu_layers 9999999 vs 99999999, two batch
|
|
// constants of 512). core/backend imports core/config, so backend references
|
|
// these; config never imports backend.
|
|
const (
|
|
// DefaultContextSize is the fallback context window when none is configured
|
|
// or estimable from the model. It is also the fallback for a GGUF whose
|
|
// metadata yields no usable estimate or that the parser cannot read at all
|
|
// (e.g. a quant type it does not know, such as NVFP4): a model-agnostic
|
|
// safe default beats a tiny, surprising window that truncates real prompts.
|
|
DefaultContextSize = 4096
|
|
|
|
// DefaultAutoContextSize caps the context we auto-derive from a GGUF when the
|
|
// user did not set context_size. The GGUF importer used to default a model's
|
|
// context to its full trained window (n_ctx_train). For long-context models
|
|
// (128k / 256k / 1M) that KV cache cannot fit a consumer GPU and the backend
|
|
// aborts on load (exitCode=-1) even though the model file is fine. So instead
|
|
// of shooting for the trained max, we keep a modest default: a small model
|
|
// (trained < this) keeps its trained window, while a long-context model caps
|
|
// here. Users who want the full window raise context_size explicitly. This is
|
|
// a conservative default, not a VRAM-maximizing one — VRAM is only used to
|
|
// step further DOWN when even this cap would not fit (see context_fit.go).
|
|
DefaultAutoContextSize = 8192
|
|
|
|
// DefaultNGPULayers means "offload all layers"; the backend (fit_params)
|
|
// clamps to what actually fits in device memory.
|
|
DefaultNGPULayers = 99999999
|
|
|
|
// DefaultFlashAttention is the flash-attention mode default; "auto" lets the
|
|
// backend enable it when the model + backend support it.
|
|
DefaultFlashAttention = "auto"
|
|
)
|