mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 18:09:05 -04:00
6584db992f
* fix(nodes): never schedule a model onto a node that cannot store it
A worker whose models filesystem was 100% full kept advertising
`status: healthy`, stayed a scheduling candidate, was picked to host a
70 GB video model, accepted the staging request, transferred ~17 GB and
only then failed:
staging .../whisper-large-v3/model.fp32-00001-of-00002.safetensors:
upload to node b7bacbf4-... failed with status 500:
writing file: /models/longcat-video-avatar-1.5/...: no space left on device
The node was at 937G/937G/0-avail. Total elapsed before the truth
surfaced: 16 minutes, for a decision that could never have succeeded.
The worker health signal only ever proved liveness. `/readyz`
(WorkerReadiness/NATSReadiness) checks the NATS link; `status: healthy`
in the registry is driven by heartbeat recency. Node capacity carried
VRAM and RAM but no disk figure at all, and the router compared model
size against VRAM only — nothing anywhere looked at free space on the
filesystem that staging actually writes to.
Report it, then use it:
- Workers now measure the filesystem backing their MODELS directory
(not `/` -- staged weights land in the models path, and that mount is
very often separate) and report `total_disk`/`available_disk` on
registration and on every heartbeat. Free disk moves faster than VRAM
under staging traffic, so the per-heartbeat refresh matters.
- The SmartRouter drops nodes that cannot store the model before it
picks one. The requirement comes from `modelPayloadBytes` -- the same
local paths `stageModelFiles` uploads, already computed for the
size-derived load budget -- plus a 5% / 1 GiB margin, rather than a
fixed percentage of the node's disk. A percentage threshold would take
a small-but-usable node out of rotation for models it could hold, and
on a homogeneous cluster would strand every node at once.
- When no node fits, scheduling fails immediately with an error naming
the requirement and each node's free space, instead of picking one and
discovering it mid-transfer.
Two deliberate non-changes. Low disk does not mark a node `unhealthy`:
the check is per model, so a node too small for one model stays a valid
target for smaller ones. And `total_disk == 0` means "does not report
disk" (pre-upgrade worker, or a failed stat), not "full" -- such nodes
pass through untouched so a rolling upgrade never empties the candidate
pool. A genuinely full node is distinguishable: non-zero total, zero
available. Registry read failures are logged and scheduling continues
unfiltered; a database hiccup must not wedge a cluster.
Free space is surfaced on the node detail page next to VRAM, since the
incident's signature was a node that looked entirely healthy.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
* feat(nodes): make the disk-headroom check operator-controllable
The admission check added in the previous commit had no off switch. A
scheduler-side veto with no escape hatch is a liability: our size
estimate can be wrong (deduplicating or compressing filesystems, a
backend that fetches its own weights rather than loading the staged
copy), and an operator who hits that has no way out but a downgrade.
Add one knob with two surfaces that share a single source of truth:
- `--distributed-disk-headroom-check` / `LOCALAI_DISTRIBUTED_DISK_HEADROOM_CHECK`
(default true), following the `--distributed-prefix-cache` pattern for
a default-on distributed feature.
- `distributed_disk_headroom_check` in the runtime-settings registry, so
it can be flipped without a restart from `POST /api/settings` and from
Settings -> Distributed in the WebUI.
Both write `DistributedConfig.DiskHeadroomDisabled`, and the SmartRouter
reads that member LIVE on every scheduling decision through a closure
over the application config rather than a value snapshotted at
construction. Env/CLI sets the boot value, the runtime setting overrides
it live, last write wins, and there is exactly one member to read.
Snapshotting would have made the runtime toggle a no-op until restart.
Disabled means WARN, not SKIP. Selection goes back to ignoring free disk
-- byte for byte the pre-check behaviour -- but the check still runs, and
when it would have rejected every node it says so, naming the knob that
suppressed it. Going quiet when switched off would reproduce the exact
condition that made the original incident expensive: a cluster doing
something that could not work and saying nothing. Disabling is also
logged once at startup. Warning only on the total-rejection case keeps
it actionable rather than chatty on a heterogeneous cluster.
Also fixes a false positive in the check itself: shared-models mode
(LOCALAI_DISTRIBUTED_SHARED_MODELS) stages nothing at all -- every node
already mounts this models directory at this path -- so demanding the
full checkpoint size of free space per node would have rejected a
cluster that needs no new bytes. The check is skipped there entirely.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
114 lines
6.9 KiB
Go
114 lines
6.9 KiB
Go
package config
|
|
|
|
// RuntimeSettings represents runtime configuration that can be changed dynamically.
|
|
// This struct is used for:
|
|
// - API responses (GET /api/settings)
|
|
// - API requests (POST /api/settings)
|
|
// - Persisting to runtime_settings.json
|
|
// - Loading from runtime_settings.json on startup
|
|
//
|
|
// All fields are pointers to distinguish between "not set" and "set to zero/false value".
|
|
type RuntimeSettings struct {
|
|
// Watchdog settings
|
|
WatchdogEnabled *bool `json:"watchdog_enabled,omitempty"`
|
|
WatchdogIdleEnabled *bool `json:"watchdog_idle_enabled,omitempty"`
|
|
WatchdogBusyEnabled *bool `json:"watchdog_busy_enabled,omitempty"`
|
|
WatchdogIdleTimeout *string `json:"watchdog_idle_timeout,omitempty"`
|
|
WatchdogBusyTimeout *string `json:"watchdog_busy_timeout,omitempty"`
|
|
WatchdogInterval *string `json:"watchdog_interval,omitempty"` // Interval between watchdog checks (e.g., 2s, 30s)
|
|
|
|
// Backend management
|
|
SingleBackend *bool `json:"single_backend,omitempty"` // Deprecated: use MaxActiveBackends = 1 instead
|
|
MaxActiveBackends *int `json:"max_active_backends,omitempty"` // Maximum number of active backends (0 = unlimited, 1 = single backend mode)
|
|
AutoUpgradeBackends *bool `json:"auto_upgrade_backends,omitempty"` // Automatically upgrade backends when new versions are detected
|
|
PreferDevelopmentBackends *bool `json:"prefer_development_backends,omitempty"` // Prefer development backend versions by default in UI
|
|
// Memory Reclaimer settings (works with GPU if available, otherwise RAM)
|
|
MemoryReclaimerEnabled *bool `json:"memory_reclaimer_enabled,omitempty"` // Enable memory threshold monitoring
|
|
MemoryReclaimerThreshold *float64 `json:"memory_reclaimer_threshold,omitempty"` // Threshold 0.0-1.0 (e.g., 0.95 = 95%)
|
|
|
|
// Eviction settings
|
|
ForceEvictionWhenBusy *bool `json:"force_eviction_when_busy,omitempty"` // Force eviction even when models have active API calls (default: false for safety)
|
|
SizeAwareEviction *bool `json:"size_aware_eviction,omitempty"` // Evict largest models first rather than least-recently-used (default: false)
|
|
LRUEvictionMaxRetries *int `json:"lru_eviction_max_retries,omitempty"` // Maximum number of retries when waiting for busy models to become idle (default: 30)
|
|
LRUEvictionRetryInterval *string `json:"lru_eviction_retry_interval,omitempty"` // Interval between retries when waiting for busy models (e.g., 1s, 2s) (default: 1s)
|
|
|
|
// Performance settings
|
|
Threads *int `json:"threads,omitempty"`
|
|
ContextSize *int `json:"context_size,omitempty"`
|
|
VRAMBudget *string `json:"vram_budget,omitempty"` // Cap VRAM for allocation ("80%" or "12GB"; "" = no cap)
|
|
F16 *bool `json:"f16,omitempty"`
|
|
Debug *bool `json:"debug,omitempty"`
|
|
EnableTracing *bool `json:"enable_tracing,omitempty"`
|
|
TracingMaxItems *int `json:"tracing_max_items,omitempty"`
|
|
TracingMaxBodyBytes *int `json:"tracing_max_body_bytes,omitempty"` // Per-body cap in bytes; 0 disables the cap
|
|
EnableBackendLogging *bool `json:"enable_backend_logging,omitempty"`
|
|
|
|
// Security/CORS settings
|
|
CORS *bool `json:"cors,omitempty"`
|
|
CSRF *bool `json:"csrf,omitempty"`
|
|
CORSAllowOrigins *string `json:"cors_allow_origins,omitempty"`
|
|
|
|
// P2P settings
|
|
P2PToken *string `json:"p2p_token,omitempty"`
|
|
P2PNetworkID *string `json:"p2p_network_id,omitempty"`
|
|
Federated *bool `json:"federated,omitempty"`
|
|
|
|
// Gallery settings
|
|
Galleries *[]Gallery `json:"galleries,omitempty"`
|
|
BackendGalleries *[]Gallery `json:"backend_galleries,omitempty"`
|
|
AutoloadGalleries *bool `json:"autoload_galleries,omitempty"`
|
|
AutoloadBackendGalleries *bool `json:"autoload_backend_galleries,omitempty"`
|
|
|
|
// API keys - No omitempty as we need to save empty arrays to clear keys
|
|
ApiKeys *[]string `json:"api_keys"`
|
|
|
|
// Agent settings
|
|
AgentJobRetentionDays *int `json:"agent_job_retention_days,omitempty"`
|
|
|
|
// Open Responses settings
|
|
OpenResponsesStoreTTL *string `json:"open_responses_store_ttl,omitempty"` // TTL for stored responses (e.g., "1h", "30m", "0" = no expiration)
|
|
|
|
// Agent Pool settings
|
|
AgentPoolEnabled *bool `json:"agent_pool_enabled,omitempty"`
|
|
AgentPoolDefaultModel *string `json:"agent_pool_default_model,omitempty"`
|
|
AgentPoolEmbeddingModel *string `json:"agent_pool_embedding_model,omitempty"`
|
|
AgentPoolMaxChunkingSize *int `json:"agent_pool_max_chunking_size,omitempty"`
|
|
AgentPoolChunkOverlap *int `json:"agent_pool_chunk_overlap,omitempty"`
|
|
AgentPoolEnableLogs *bool `json:"agent_pool_enable_logs,omitempty"`
|
|
AgentPoolCollectionDBPath *string `json:"agent_pool_collection_db_path,omitempty"`
|
|
AgentPoolVectorEngine *string `json:"agent_pool_vector_engine,omitempty"` // chromem | postgres
|
|
AgentPoolDatabaseURL *string `json:"agent_pool_database_url,omitempty"` // PostgreSQL DSN when vector engine is postgres
|
|
AgentPoolAgentHubURL *string `json:"agent_pool_agent_hub_url,omitempty"` // override the agenthub.localai.io endpoint
|
|
|
|
// Distributed-mode settings. Read LIVE by the SmartRouter on each
|
|
// scheduling decision, so toggling takes effect on the next placement
|
|
// without a restart. Stored as the negation of
|
|
// DistributedConfig.DiskHeadroomDisabled for UI clarity.
|
|
DistributedDiskHeadroomCheck *bool `json:"distributed_disk_headroom_check,omitempty"` // Reject nodes without room to store the model (default: true)
|
|
|
|
// LocalAI Assistant settings — read live by the chat handler at request
|
|
// entry, so flipping the toggle takes effect on the next request.
|
|
LocalAIAssistantEnabled *bool `json:"localai_assistant_enabled,omitempty"` // negation of DisableLocalAIAssistant for UI clarity
|
|
|
|
// Branding / whitelabeling. Text fields are user-facing; *File fields hold
|
|
// just the basename of an uploaded asset under {DynamicConfigsDir}/branding/.
|
|
// All optional — empty values fall back to bundled LocalAI defaults.
|
|
InstanceName *string `json:"instance_name,omitempty"`
|
|
InstanceTagline *string `json:"instance_tagline,omitempty"`
|
|
LogoFile *string `json:"logo_file,omitempty"`
|
|
LogoHorizontalFile *string `json:"logo_horizontal_file,omitempty"`
|
|
FaviconFile *string `json:"favicon_file,omitempty"`
|
|
|
|
// Cloud-proxy MITM listener. MITMCADir is intentionally NOT
|
|
// exposed at runtime — the CA dir is a startup-only path and
|
|
// changing it after the CA has been generated would orphan
|
|
// trusted clients.
|
|
MITMListen *string `json:"mitm_listen,omitempty"`
|
|
|
|
// PIIDefaultDetectors are the token-classification detector models applied
|
|
// to any PII-enabled model that names no detectors of its own (so
|
|
// cloud-proxy/MITM redaction works without per-model config). No omitempty:
|
|
// an empty array must round-trip so the operator can clear it from the UI.
|
|
PIIDefaultDetectors *[]string `json:"pii_default_detectors"`
|
|
}
|