package config // Canonical default values. // // These are owned here so the two layers that need them share a single source // of truth: the config tiers (ApplyInference/Hardware/Serving/Generic — which // *decide* defaults) and core/backend/options.go (which *translates* a // ModelConfig to the backend wire format and supplies the same fallbacks // defensively). Previously these were duplicated as literals across both // packages and had drifted (e.g. n_gpu_layers 9999999 vs 99999999, two batch // constants of 512). core/backend imports core/config, so backend references // these; config never imports backend. const ( // DefaultContextSize is the fallback context window when none is configured // or estimable from the model. It is also the fallback for a GGUF whose // metadata yields no usable estimate or that the parser cannot read at all // (e.g. a quant type it does not know, such as NVFP4): a model-agnostic // safe default beats a tiny, surprising window that truncates real prompts. DefaultContextSize = 4096 // DefaultAutoContextSize caps the context we auto-derive from a GGUF when the // user did not set context_size. The GGUF importer used to default a model's // context to its full trained window (n_ctx_train). For long-context models // (128k / 256k / 1M) that KV cache cannot fit a consumer GPU and the backend // aborts on load (exitCode=-1) even though the model file is fine. So instead // of shooting for the trained max, we keep a modest default: a small model // (trained < this) keeps its trained window, while a long-context model caps // here. Users who want the full window raise context_size explicitly. This is // a conservative default, not a VRAM-maximizing one — VRAM is only used to // step further DOWN when even this cap would not fit (see context_fit.go). DefaultAutoContextSize = 8192 // DefaultNGPULayers means "offload all layers"; the backend (fit_params) // clamps to what actually fits in device memory. DefaultNGPULayers = 99999999 // DefaultFlashAttention is the flash-attention mode default; "auto" lets the // backend enable it when the model + backend support it. DefaultFlashAttention = "auto" )