mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
mudler-agentandEttore Di Giacinto
0b5cab414a
* feat(config): add a limit for concurrent backend operations Add --max-concurrent-backend-operations (LOCALAI_MAX_CONCURRENT_BACKEND_OPERATIONS) and the runtime setting max_concurrent_backend_operations. It sets how many backend install, upgrade and delete operations run at the same time. The default is 1, which keeps the current behaviour. Values below 1 become 1. The runtime value is also kept in an atomic, so the scheduler that reads it never races with a settings change. The setting shows in the Settings page under Backends. This commit only adds the setting; the next one uses it. Assisted-by: Claude:claude-sonnet-5-5 go-vet Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(galleryop): run backend operations in a bounded pool One goroutine read both gallery channels and ran each operation inline, so one slow backend operation held every operation behind it, model operations included. In distributed mode the manager also visited the nodes one at a time and waited for each up to the install timeout. Backend operations now go through a scheduler that runs up to max_concurrent_backend_operations of them at once. Operations start in the order they arrive. Install, upgrade and delete of one backend share a key and never overlap; an operation that waits for its backend does not hold a slot, so independent operations pass it. Cancelling a queued operation drops it at once. The limit is read on every scheduling step, so a runtime change applies without a restart. With the default of 1 the behaviour is the old single worker. Model operations get their own loop and stay serial. In distributed mode the manager sends one operation to its nodes in parallel, up to the same limit, and keeps the result in node order. A per (node, backend) lock keeps a live operation and the reconciler from sending the same operation to a node together; the reconciler skips a row that is locked and checks that the row still exists before it sends. Timeouts are unchanged. The download progress timers were plain package variables. They are atomics now, because operations can run in parallel. Assisted-by: Claude:claude-sonnet-5-5 go-vet go-test Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
121 lines
7.5 KiB
Go
121 lines
7.5 KiB
Go
package config
|
|
|
|
// RuntimeSettings represents runtime configuration that can be changed dynamically.
|
|
// This struct is used for:
|
|
// - API responses (GET /api/settings)
|
|
// - API requests (POST /api/settings)
|
|
// - Persisting to runtime_settings.json
|
|
// - Loading from runtime_settings.json on startup
|
|
//
|
|
// All fields are pointers to distinguish between "not set" and "set to zero/false value".
|
|
type RuntimeSettings struct {
|
|
// Watchdog settings
|
|
WatchdogEnabled *bool `json:"watchdog_enabled,omitempty"`
|
|
WatchdogIdleEnabled *bool `json:"watchdog_idle_enabled,omitempty"`
|
|
WatchdogBusyEnabled *bool `json:"watchdog_busy_enabled,omitempty"`
|
|
WatchdogIdleTimeout *string `json:"watchdog_idle_timeout,omitempty"`
|
|
WatchdogBusyTimeout *string `json:"watchdog_busy_timeout,omitempty"`
|
|
WatchdogInterval *string `json:"watchdog_interval,omitempty"` // Interval between watchdog checks (e.g., 2s, 30s)
|
|
|
|
// Backend management
|
|
SingleBackend *bool `json:"single_backend,omitempty"` // Deprecated: use MaxActiveBackends = 1 instead
|
|
MaxActiveBackends *int `json:"max_active_backends,omitempty"` // Maximum number of active backends (0 = unlimited, 1 = single backend mode)
|
|
OfficialLibraries *bool `json:"official_libraries,omitempty"` // Read the shipped default library sources of add-ons, backends and models
|
|
AutoUpgradeBackends *bool `json:"auto_upgrade_backends,omitempty"` // Automatically upgrade backends when new versions are detected
|
|
AutoUpgradeAddons *bool `json:"auto_upgrade_addons,omitempty"` // Automatically upgrade installed add-ons when their library lists a newer version
|
|
PreferDevelopmentBackends *bool `json:"prefer_development_backends,omitempty"` // Display/download preference; installed alias resolver changes require restart
|
|
// Memory Reclaimer settings (works with GPU if available, otherwise RAM)
|
|
MemoryReclaimerEnabled *bool `json:"memory_reclaimer_enabled,omitempty"` // Enable memory threshold monitoring
|
|
MemoryReclaimerThreshold *float64 `json:"memory_reclaimer_threshold,omitempty"` // Threshold 0.0-1.0 (e.g., 0.95 = 95%)
|
|
|
|
// Eviction settings
|
|
ForceEvictionWhenBusy *bool `json:"force_eviction_when_busy,omitempty"` // Force eviction even when models have active API calls (default: false for safety)
|
|
SizeAwareEviction *bool `json:"size_aware_eviction,omitempty"` // Evict largest models first rather than least-recently-used (default: false)
|
|
LRUEvictionMaxRetries *int `json:"lru_eviction_max_retries,omitempty"` // Maximum number of retries when waiting for busy models to become idle (default: 30)
|
|
LRUEvictionRetryInterval *string `json:"lru_eviction_retry_interval,omitempty"` // Interval between retries when waiting for busy models (e.g., 1s, 2s) (default: 1s)
|
|
|
|
// Performance settings
|
|
Threads *int `json:"threads,omitempty"`
|
|
ContextSize *int `json:"context_size,omitempty"`
|
|
ArtifactDownloadConcurrency *int `json:"artifact_download_concurrency,omitempty"`
|
|
|
|
MaxConcurrentBackendOperations *int `json:"max_concurrent_backend_operations,omitempty"`
|
|
|
|
VRAMBudget *string `json:"vram_budget,omitempty"` // Cap VRAM for allocation ("80%" or "12GB"; "" = no cap)
|
|
F16 *bool `json:"f16,omitempty"`
|
|
Debug *bool `json:"debug,omitempty"`
|
|
EnableTracing *bool `json:"enable_tracing,omitempty"`
|
|
TracingMaxItems *int `json:"tracing_max_items,omitempty"`
|
|
TracingMaxBodyBytes *int `json:"tracing_max_body_bytes,omitempty"` // Per-body cap in bytes; 0 disables the cap
|
|
EnableBackendLogging *bool `json:"enable_backend_logging,omitempty"`
|
|
|
|
// Security/CORS settings
|
|
CORS *bool `json:"cors,omitempty"`
|
|
CSRF *bool `json:"csrf,omitempty"`
|
|
CORSAllowOrigins *string `json:"cors_allow_origins,omitempty"`
|
|
|
|
// P2P settings
|
|
P2PToken *string `json:"p2p_token,omitempty"`
|
|
P2PNetworkID *string `json:"p2p_network_id,omitempty"`
|
|
Federated *bool `json:"federated,omitempty"`
|
|
|
|
// Gallery settings
|
|
Galleries *[]Gallery `json:"galleries,omitempty"`
|
|
BackendGalleries *[]Gallery `json:"backend_galleries,omitempty"`
|
|
AutoloadGalleries *bool `json:"autoload_galleries,omitempty"`
|
|
AutoloadBackendGalleries *bool `json:"autoload_backend_galleries,omitempty"`
|
|
VRAMPersistentCache *bool `json:"vram_persistent_cache,omitempty"`
|
|
|
|
// API keys - No omitempty as we need to save empty arrays to clear keys
|
|
ApiKeys *[]string `json:"api_keys"`
|
|
|
|
// Agent settings
|
|
AgentJobRetentionDays *int `json:"agent_job_retention_days,omitempty"`
|
|
|
|
// Open Responses settings
|
|
OpenResponsesStoreTTL *string `json:"open_responses_store_ttl,omitempty"` // TTL for stored responses (e.g., "1h", "30m", "0" = no expiration)
|
|
|
|
// Agent Pool settings
|
|
AgentPoolEnabled *bool `json:"agent_pool_enabled,omitempty"`
|
|
AgentPoolDefaultModel *string `json:"agent_pool_default_model,omitempty"`
|
|
AgentPoolEmbeddingModel *string `json:"agent_pool_embedding_model,omitempty"`
|
|
AgentPoolMaxChunkingSize *int `json:"agent_pool_max_chunking_size,omitempty"`
|
|
AgentPoolChunkOverlap *int `json:"agent_pool_chunk_overlap,omitempty"`
|
|
AgentPoolEnableLogs *bool `json:"agent_pool_enable_logs,omitempty"`
|
|
AgentPoolCollectionDBPath *string `json:"agent_pool_collection_db_path,omitempty"`
|
|
AgentPoolVectorEngine *string `json:"agent_pool_vector_engine,omitempty"` // chromem | postgres
|
|
AgentPoolDatabaseURL *string `json:"agent_pool_database_url,omitempty"` // PostgreSQL DSN when vector engine is postgres
|
|
AgentPoolAgentHubURL *string `json:"agent_pool_agent_hub_url,omitempty"` // override the agenthub.localai.io endpoint
|
|
|
|
// Distributed-mode settings. Read LIVE by the SmartRouter on each
|
|
// scheduling decision, so toggling takes effect on the next placement
|
|
// without a restart. Stored as the negation of
|
|
// DistributedConfig.DiskHeadroomDisabled for UI clarity.
|
|
DistributedDiskHeadroomCheck *bool `json:"distributed_disk_headroom_check,omitempty"` // Reject nodes without room to store the model (default: true)
|
|
|
|
// LocalAI Assistant settings — read live by the chat handler at request
|
|
// entry, so flipping the toggle takes effect on the next request.
|
|
LocalAIAssistantEnabled *bool `json:"localai_assistant_enabled,omitempty"` // negation of DisableLocalAIAssistant for UI clarity
|
|
|
|
// Branding / whitelabeling. Text fields are user-facing; *File fields hold
|
|
// just the basename of an uploaded asset under {DynamicConfigsDir}/branding/.
|
|
// All optional — empty values fall back to bundled LocalAI defaults.
|
|
InstanceName *string `json:"instance_name,omitempty"`
|
|
InstanceTagline *string `json:"instance_tagline,omitempty"`
|
|
LogoFile *string `json:"logo_file,omitempty"`
|
|
LogoHorizontalFile *string `json:"logo_horizontal_file,omitempty"`
|
|
FaviconFile *string `json:"favicon_file,omitempty"`
|
|
|
|
// Cloud-proxy MITM listener. MITMCADir is intentionally NOT
|
|
// exposed at runtime — the CA dir is a startup-only path and
|
|
// changing it after the CA has been generated would orphan
|
|
// trusted clients.
|
|
MITMListen *string `json:"mitm_listen,omitempty"`
|
|
|
|
// PIIDefaultDetectors are the token-classification detector models applied
|
|
// to any PII-enabled model that names no detectors of its own (so
|
|
// cloud-proxy/MITM redaction works without per-model config). No omitempty:
|
|
// an empty array must round-trip so the operator can clear it from the UI.
|
|
PIIDefaultDetectors *[]string `json:"pii_default_detectors"`
|
|
}
|