mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 01:48:06 -04:00
Code Issues Packages Projects Releases Wiki Activity

refactor(settings): single declarative registry for runtime settings (fixes the #10845 bug class) (#10864)

Browse Source 
* feat(settings): add declarative runtime-settings field registry

One fieldSpec row per RuntimeSettings field, with a reflection
completeness spec so a field added without a registry row is a red
test instead of a silently-dropped setting (the #10845 bug class).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5

* refactor(settings): drive ToRuntimeSettings/ApplyRuntimeSettings from the field registry

Behavior-preserving: ~350 hand-written per-field lines become two loops
over runtimeSettingsFields, gated by a To->Apply->To round-trip spec.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5

* feat(settings): baseline-driven startup merge for persisted runtime settings

ApplyRuntimeSettingsAtStartup compares the live config against
DefaultRuntimeBaseline (option-less-run defaults incl. kong-injected
flag defaults) instead of per-field == 0 guards. Fixes persisted
lru_eviction_max_retries, tracing_max_items, agent_job_retention_days,
memory_reclaimer_threshold, galleries and autoload flags being
silently ignored at boot.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5

* fix(settings): registry-driven startup merge, applied before consumers

loadRuntimeSettingsFromFile becomes a thin wrapper over
ApplyRuntimeSettingsAtStartup and runs at the top of New(), before
model configs capture app-level defaults. WithThreads stops eagerly
resolving 0 so a persisted thread count survives restart while
LOCALAI_THREADS still wins (#10845); the physical-core fallback moves
after the merge.

Also: run.go now injects the memory-reclaimer threshold unconditionally
so the option-less boot matches DefaultRuntimeBaseline and a UI-saved
threshold survives restart.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5

* refactor(settings): file watcher delegates to the registry merge; shared API-key merge

Manual edits to runtime_settings.json now behave like a boot-time load
(env still wins) instead of the inverted diverged-from-startup guard
that ignored most manual edits. MergeAPIKeys dedups env keys in one
place for the endpoint and the watcher.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5

* docs(settings): document unified runtime-settings precedence

Document the single env/CLI > runtime_settings.json > defaults rule,
applied identically at boot, on POST /api/settings, and on manual file
edits, plus the two known limitations (default-valued env vars are
indistinguishable from unset; API-changed fields hot-apply on the next
restart only). Also add a completion debug log when the watcher applies
runtime_settings.json.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5

* test(settings): reset the global VRAM cap leaked by the round-trip spec

The round-trip spec applies vram_budget=12GiB, whose post-loop hook
installs a process-global default cap; without a reset every spec
ordered after it runs under that phantom budget. Also drop a stale
enumeration in the ApplyRuntimeSettings doc comment.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
LocalAI [bot]
2026-07-16 22:39:59 +02:00
committed by GitHub
parent 06b4a29387
commit c1a891662c
13 changed files with 1031 additions and 977 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
cmd/local-ai/main.go
View File

@@ -7,6 +7,7 @@ import (
"github.com/alecthomas/kong"
"github.com/joho/godotenv"
"github.com/mudler/LocalAI/core/cli"
"github.com/mudler/LocalAI/core/config"
"github.com/mudler/LocalAI/internal"
"github.com/mudler/xlog"
@@ -65,8 +66,8 @@ For documentation and support:
// denied". See cli.DefaultGeneratedContentPath.
"generatedcontentpath": cli.DefaultGeneratedContentPath(),
"uploadpath": cli.DefaultUploadPath(),
"galleries": `[{"name":"localai", "url":"github:mudler/LocalAI/gallery/index.yaml@master"}]`,
"backends": `[{"name":"localai", "url":"github:mudler/LocalAI/backend/index.yaml@master"}]`,
"galleries": config.DefaultGalleriesJSON,
"backends": config.DefaultBackendGalleriesJSON,
"version": internal.PrintableVersion(),
},
)

214
core/application/config_file_watcher.go
View File

@@ -6,7 +6,6 @@ import (
"os"
"path"
"path/filepath"
"slices"
"time"
"dario.cat/mergo"
@@ -186,202 +185,27 @@ func readExternalBackendsJson(startupAppConfig config.ApplicationConfig) fileHan
}
func readRuntimeSettingsJson(startupAppConfig config.ApplicationConfig) fileHandler {
handler := func(fileContent []byte, appConfig *config.ApplicationConfig) error {
return func(fileContent []byte, appConfig *config.ApplicationConfig) error {
xlog.Debug("processing runtime_settings.json")
// Determine if settings came from env vars by comparing with startup config
// startupAppConfig contains the original values set from env vars at startup.
// If current values match startup values, they came from env vars (or defaults).
// We apply file settings only if current values match startup values (meaning not from env vars).
envWatchdogIdle := appConfig.WatchDogIdle == startupAppConfig.WatchDogIdle
envWatchdogBusy := appConfig.WatchDogBusy == startupAppConfig.WatchDogBusy
envWatchdogIdleTimeout := appConfig.WatchDogIdleTimeout == startupAppConfig.WatchDogIdleTimeout
envWatchdogBusyTimeout := appConfig.WatchDogBusyTimeout == startupAppConfig.WatchDogBusyTimeout
envWatchdogInterval := appConfig.WatchDogInterval == startupAppConfig.WatchDogInterval
envSingleBackend := appConfig.SingleBackend == startupAppConfig.SingleBackend
envMaxActiveBackends := appConfig.MaxActiveBackends == startupAppConfig.MaxActiveBackends
envMemoryReclaimerEnabled := appConfig.MemoryReclaimerEnabled == startupAppConfig.MemoryReclaimerEnabled
envMemoryReclaimerThreshold := appConfig.MemoryReclaimerThreshold == startupAppConfig.MemoryReclaimerThreshold
envThreads := appConfig.Threads == startupAppConfig.Threads
envContextSize := appConfig.ContextSize == startupAppConfig.ContextSize
envF16 := appConfig.F16 == startupAppConfig.F16
envDebug := appConfig.Debug == startupAppConfig.Debug
envCORS := appConfig.CORS == startupAppConfig.CORS
envCSRF := appConfig.DisableCSRF == startupAppConfig.DisableCSRF
envCORSAllowOrigins := appConfig.CORSAllowOrigins == startupAppConfig.CORSAllowOrigins
envP2PToken := appConfig.P2PToken == startupAppConfig.P2PToken
envP2PNetworkID := appConfig.P2PNetworkID == startupAppConfig.P2PNetworkID
envFederated := appConfig.Federated == startupAppConfig.Federated
envGalleries := slices.Equal(appConfig.Galleries, startupAppConfig.Galleries)
envBackendGalleries := slices.Equal(appConfig.BackendGalleries, startupAppConfig.BackendGalleries)
envAutoloadGalleries := appConfig.AutoloadGalleries == startupAppConfig.AutoloadGalleries
envAutoloadBackendGalleries := appConfig.AutoloadBackendGalleries == startupAppConfig.AutoloadBackendGalleries
envPIIDefaultDetectors := slices.Equal(appConfig.PIIDefaultDetectors, startupAppConfig.PIIDefaultDetectors)
envAgentJobRetentionDays := appConfig.AgentJobRetentionDays == startupAppConfig.AgentJobRetentionDays
envForceEvictionWhenBusy := appConfig.ForceEvictionWhenBusy == startupAppConfig.ForceEvictionWhenBusy
envLRUEvictionMaxRetries := appConfig.LRUEvictionMaxRetries == startupAppConfig.LRUEvictionMaxRetries
envLRUEvictionRetryInterval := appConfig.LRUEvictionRetryInterval == startupAppConfig.LRUEvictionRetryInterval
if len(fileContent) > 0 {
var settings config.RuntimeSettings
err := json.Unmarshal(fileContent, &settings)
if err != nil {
return err
}
// Apply file settings only if they don't match startup values (i.e., not from env vars)
if settings.WatchdogIdleEnabled != nil && !envWatchdogIdle {
appConfig.WatchDogIdle = *settings.WatchdogIdleEnabled
if appConfig.WatchDogIdle {
appConfig.WatchDog = true
}
}
if settings.WatchdogBusyEnabled != nil && !envWatchdogBusy {
appConfig.WatchDogBusy = *settings.WatchdogBusyEnabled
if appConfig.WatchDogBusy {
appConfig.WatchDog = true
}
}
if settings.WatchdogIdleTimeout != nil && !envWatchdogIdleTimeout {
dur, err := time.ParseDuration(*settings.WatchdogIdleTimeout)
if err == nil {
appConfig.WatchDogIdleTimeout = dur
} else {
xlog.Warn("invalid watchdog idle timeout in runtime_settings.json", "error", err, "timeout", *settings.WatchdogIdleTimeout)
}
}
if settings.WatchdogBusyTimeout != nil && !envWatchdogBusyTimeout {
dur, err := time.ParseDuration(*settings.WatchdogBusyTimeout)
if err == nil {
appConfig.WatchDogBusyTimeout = dur
} else {
xlog.Warn("invalid watchdog busy timeout in runtime_settings.json", "error", err, "timeout", *settings.WatchdogBusyTimeout)
}
}
if settings.WatchdogInterval != nil && !envWatchdogInterval {
dur, err := time.ParseDuration(*settings.WatchdogInterval)
if err == nil {
appConfig.WatchDogInterval = dur
} else {
xlog.Warn("invalid watchdog interval in runtime_settings.json", "error", err, "interval", *settings.WatchdogInterval)
}
}
// Handle MaxActiveBackends (new) and SingleBackend (deprecated)
if settings.MaxActiveBackends != nil && !envMaxActiveBackends {
appConfig.MaxActiveBackends = *settings.MaxActiveBackends
// For backward compatibility, also set SingleBackend if MaxActiveBackends == 1
appConfig.SingleBackend = (*settings.MaxActiveBackends == 1)
} else if settings.SingleBackend != nil && !envSingleBackend {
// Legacy: SingleBackend maps to MaxActiveBackends = 1
appConfig.SingleBackend = *settings.SingleBackend
if *settings.SingleBackend {
appConfig.MaxActiveBackends = 1
} else {
appConfig.MaxActiveBackends = 0
}
}
if settings.MemoryReclaimerEnabled != nil && !envMemoryReclaimerEnabled {
appConfig.MemoryReclaimerEnabled = *settings.MemoryReclaimerEnabled
if appConfig.MemoryReclaimerEnabled {
appConfig.WatchDog = true // Memory reclaimer requires watchdog
}
}
if settings.MemoryReclaimerThreshold != nil && !envMemoryReclaimerThreshold {
appConfig.MemoryReclaimerThreshold = *settings.MemoryReclaimerThreshold
}
if settings.ForceEvictionWhenBusy != nil && !envForceEvictionWhenBusy {
appConfig.ForceEvictionWhenBusy = *settings.ForceEvictionWhenBusy
}
if settings.LRUEvictionMaxRetries != nil && !envLRUEvictionMaxRetries {
appConfig.LRUEvictionMaxRetries = *settings.LRUEvictionMaxRetries
}
if settings.LRUEvictionRetryInterval != nil && !envLRUEvictionRetryInterval {
dur, err := time.ParseDuration(*settings.LRUEvictionRetryInterval)
if err == nil {
appConfig.LRUEvictionRetryInterval = dur
} else {
xlog.Warn("invalid LRU eviction retry interval in runtime_settings.json", "error", err, "interval", *settings.LRUEvictionRetryInterval)
}
}
if settings.Threads != nil && !envThreads {
appConfig.Threads = *settings.Threads
}
if settings.ContextSize != nil && !envContextSize {
appConfig.ContextSize = *settings.ContextSize
}
if settings.F16 != nil && !envF16 {
appConfig.F16 = *settings.F16
}
if settings.Debug != nil && !envDebug {
appConfig.Debug = *settings.Debug
}
if settings.CORS != nil && !envCORS {
appConfig.CORS = *settings.CORS
}
if settings.CSRF != nil && !envCSRF {
appConfig.DisableCSRF = *settings.CSRF
}
if settings.CORSAllowOrigins != nil && !envCORSAllowOrigins {
appConfig.CORSAllowOrigins = *settings.CORSAllowOrigins
}
if settings.P2PToken != nil && !envP2PToken {
appConfig.P2PToken = *settings.P2PToken
}
if settings.P2PNetworkID != nil && !envP2PNetworkID {
appConfig.P2PNetworkID = *settings.P2PNetworkID
}
if settings.Federated != nil && !envFederated {
appConfig.Federated = *settings.Federated
}
if settings.Galleries != nil && !envGalleries {
appConfig.Galleries = *settings.Galleries
}
if settings.BackendGalleries != nil && !envBackendGalleries {
appConfig.BackendGalleries = *settings.BackendGalleries
}
if settings.AutoloadGalleries != nil && !envAutoloadGalleries {
appConfig.AutoloadGalleries = *settings.AutoloadGalleries
}
if settings.AutoloadBackendGalleries != nil && !envAutoloadBackendGalleries {
appConfig.AutoloadBackendGalleries = *settings.AutoloadBackendGalleries
}
if settings.PIIDefaultDetectors != nil && !envPIIDefaultDetectors {
// Request-side default redaction reads this live via
// ResolvePIIPolicy, so a file edit takes effect on the next chat
// request. The MITM listener resolves its per-host detector map
// once at start, so a raw file edit reaches cloud-proxy traffic
// only after a restart or a POST /api/settings (which rebuilds
// the listener) — the admin UI uses the latter.
appConfig.PIIDefaultDetectors = append([]string(nil), (*settings.PIIDefaultDetectors)...)
}
if settings.AutoUpgradeBackends != nil {
appConfig.AutoUpgradeBackends = *settings.AutoUpgradeBackends
}
if settings.PreferDevelopmentBackends != nil {
appConfig.PreferDevelopmentBackends = *settings.PreferDevelopmentBackends
}
if settings.ApiKeys != nil {
// API keys from env vars (startup) should be kept, runtime settings keys replace all runtime keys
// If runtime_settings.json specifies ApiKeys (even if empty), it replaces all runtime keys
// Start with env keys, then add runtime_settings.json keys (which may be empty to clear them)
envKeys := startupAppConfig.ApiKeys
runtimeKeys := *settings.ApiKeys
// Replace all runtime keys with what's in runtime_settings.json
appConfig.ApiKeys = append(envKeys, runtimeKeys...)
}
if settings.AgentJobRetentionDays != nil && !envAgentJobRetentionDays {
appConfig.AgentJobRetentionDays = *settings.AgentJobRetentionDays
}
// If watchdog is enabled via file but not via env, ensure WatchDog flag is set
if !envWatchdogIdle && !envWatchdogBusy {
if settings.WatchdogEnabled != nil && *settings.WatchdogEnabled {
appConfig.WatchDog = true
}
}
if len(fileContent) == 0 {
return nil
}
xlog.Debug("runtime settings loaded from runtime_settings.json")
var settings config.RuntimeSettings
if err := json.Unmarshal(fileContent, &settings); err != nil {
return err
}
// Same merge semantics as boot: env/CLI-claimed fields win, the
// file supplies the rest. This replaces the old inverted guard
// (apply only when live != startup snapshot) which skipped genuine
// manual edits of any field still at its boot value. Trade-off: a
// field previously changed via the API looks env-set to the
// baseline comparison, so a manual file edit of that field lands
// on the next restart instead of hot-applying.
appConfig.ApplyRuntimeSettingsAtStartup(&settings)
if settings.ApiKeys != nil {
appConfig.ApiKeys = config.MergeAPIKeys(startupAppConfig.ApiKeys, *settings.ApiKeys)
}
xlog.Debug("runtime settings applied from runtime_settings.json")
return nil
}
return handler
}

99
core/application/runtime_settings_branding_test.go
View File

@@ -157,20 +157,32 @@ var _ = Describe("loadRuntimeSettingsFromFile", func() {
})
})
// The live file watcher applies pii_default_detectors on a runtime change
// the same way it handles galleries/threads/etc.: env-set values (current
// == startup snapshot) are left alone, otherwise the file value is applied
// to the live config so request-side default redaction picks it up without
// a restart.
// The live file watcher delegates to ApplyRuntimeSettingsAtStartup, so a
// manual edit of runtime_settings.json behaves exactly like a boot-time
// load: env/CLI-claimed values win, the file supplies the rest, and the
// applied value reaches request-side default redaction without a restart.
Describe("file watcher: pii_default_detectors", func() {
It("applies a changed file value to the live config", func() {
It("applies a file value when the live config is still at its default", func() {
startup := config.ApplicationConfig{} // no env baseline
live := &config.ApplicationConfig{PIIDefaultDetectors: []string{"old"}}
live := config.NewApplicationConfig() // detectors at default (empty)
handler := readRuntimeSettingsJson(startup)
Expect(handler([]byte(`{"pii_default_detectors":["new-a","new-b"]}`), live)).To(Succeed())
Expect(live.PIIDefaultDetectors).To(Equal([]string{"new-a", "new-b"}))
})
It("defers a file edit of an already-customized value to the next restart", func() {
// The baseline comparison cannot tell an API-set value from an
// env-set one, so the watcher leaves it alone; the boot loader
// applies it on restart. Documented trade-off of the unified
// merge semantics (design doc 2026-07-16).
startup := config.ApplicationConfig{}
live := config.NewApplicationConfig()
live.PIIDefaultDetectors = []string{"old"}
handler := readRuntimeSettingsJson(startup)
Expect(handler([]byte(`{"pii_default_detectors":["from-file"]}`), live)).To(Succeed())
Expect(live.PIIDefaultDetectors).To(Equal([]string{"old"}))
})
It("leaves an env-controlled value untouched", func() {
startup := config.ApplicationConfig{PIIDefaultDetectors: []string{"from-env"}}
live := &config.ApplicationConfig{PIIDefaultDetectors: []string{"from-env"}}
@@ -262,6 +274,79 @@ var _ = Describe("loadRuntimeSettingsFromFile", func() {
})
})
// Issue #10845 root cause class: fields with non-zero boot defaults
// whose == 0 guards never fired, so the persisted value was ignored on
// every restart. The registry's baseline comparison fixes them; these
// specs pin that.
Describe("fields with non-zero defaults (silent restart loss)", func() {
It("loads a persisted lru_eviction_max_retries over the default 30", func() {
cfg := config.NewApplicationConfig()
cfg.DynamicConfigsDir = seedSettings(`{"lru_eviction_max_retries": 99}`)
loadRuntimeSettingsFromFile(cfg)
Expect(cfg.LRUEvictionMaxRetries).To(Equal(99))
})
It("loads a persisted tracing_max_items over the default 1024", func() {
cfg := config.NewApplicationConfig()
cfg.DynamicConfigsDir = seedSettings(`{"tracing_max_items": 4096}`)
loadRuntimeSettingsFromFile(cfg)
Expect(cfg.TracingMaxItems).To(Equal(4096))
})
It("loads a persisted agent_job_retention_days over the default 30", func() {
cfg := config.NewApplicationConfig()
cfg.DynamicConfigsDir = seedSettings(`{"agent_job_retention_days": 7}`)
loadRuntimeSettingsFromFile(cfg)
Expect(cfg.AgentJobRetentionDays).To(Equal(7))
})
})
// #10845: threads saved via the UI must survive a restart, while
// LOCALAI_THREADS/CLI still wins. WithThreads no longer eagerly
// resolves 0 to the physical-core count; application.New() does that
// after this loader has run.
Describe("performance fields", func() {
It("loads persisted threads when env/CLI left them unset", func() {
cfg := config.NewApplicationConfig(config.WithThreads(0))
cfg.DynamicConfigsDir = seedSettings(`{"threads": 4}`)
loadRuntimeSettingsFromFile(cfg)
Expect(cfg.Threads).To(Equal(4))
})
It("keeps an env/CLI thread count over the file", func() {
cfg := config.NewApplicationConfig(config.WithThreads(8))
cfg.DynamicConfigsDir = seedSettings(`{"threads": 4}`)
loadRuntimeSettingsFromFile(cfg)
Expect(cfg.Threads).To(Equal(8))
})
It("loads persisted context_size and f16", func() {
cfg := config.NewApplicationConfig()
cfg.DynamicConfigsDir = seedSettings(`{"context_size": 4096, "f16": true}`)
loadRuntimeSettingsFromFile(cfg)
Expect(cfg.ContextSize).To(Equal(4096))
Expect(cfg.F16).To(BeTrue())
})
})
// Option-less boot contract with core/cli/run.go: the kong threshold
// default (0.95) is injected unconditionally even when the reclaimer is
// disabled, so DefaultRuntimeBaseline's 0.95 overlay matches reality and
// a UI-persisted threshold is not mistaken for an env-set one. Both
// persisted fields must apply, and enabling the reclaimer must force the
// watchdog master flag (startup invariant).
Describe("memory reclaimer", func() {
It("applies a persisted enable + threshold on an option-less boot", func() {
cfg := config.NewApplicationConfig()
cfg.MemoryReclaimerThreshold = 0.95 // what run.go injects when no flag is passed
cfg.DynamicConfigsDir = seedSettings(`{"memory_reclaimer_enabled": true, "memory_reclaimer_threshold": 0.5}`)
loadRuntimeSettingsFromFile(cfg)
Expect(cfg.MemoryReclaimerEnabled).To(BeTrue())
Expect(cfg.MemoryReclaimerThreshold).To(Equal(0.5))
Expect(cfg.WatchDog).To(BeTrue(), "an enabled reclaimer must force the watchdog on")
})
})
// Backend logging capture. Worker/distributed mode force-enables it
// (core/services/worker.SetBackendLoggingEnabled(true)); single mode used
// to leave it off by default with no CLI flag, so the UI "Backend Logs"

407
core/application/startup.go
View File

@@ -3,7 +3,6 @@ package application
import (
"crypto/rand"
"encoding/hex"
"encoding/json"
"fmt"
"os"
"path/filepath"
@@ -39,9 +38,27 @@ import (
func New(opts ...config.AppOption) (*Application, error) {
options := config.NewApplicationConfig(opts...)
// Store a copy of the startup config (from env vars, before file loading)
// This is used to determine if settings came from env vars vs file
// Store a copy of the startup config (env/CLI only, before file
// loading): the settings endpoint uses it to tell env-provided API
// keys apart from runtime-managed ones.
startupConfigCopy := *options
// Merge persisted runtime settings BEFORE anything consumes options:
// model-config defaults (ToConfigLoaderOptions), gallery services, the
// watchdog, and the MITM listener are all configured from options
// further down. Loading late (the old call site, after model configs
// were read) meant boot-loaded models saw pre-file defaults for one
// full restart. Env/CLI values win over the file (see
// ApplyRuntimeSettingsAtStartup).
loadRuntimeSettingsFromFile(options)
// WithThreads no longer eagerly resolves 0 (so the settings merge can
// tell "unset" from "env-set"); resolve the physical-core default now
// that env, CLI, and file have all had their say.
if options.Threads == 0 {
options.Threads = xsysinfo.CPUPhysicalCores()
}
application := newApplication(options)
application.startupConfig = &startupConfigCopy
@@ -436,13 +453,6 @@ func New(opts ...config.AppOption) (*Application, error) {
}
}
// Load runtime settings from file if DynamicConfigsDir is set
// This applies file settings with env var precedence (env vars take priority)
// Note: startupConfigCopy was already created above, so it has the original env var values
if options.DynamicConfigsDir != "" {
loadRuntimeSettingsFromFile(options)
}
// Wire the cloudproxy MITM listener. Opt-in: empty MITMListen
// means "no MITM" — operators must explicitly choose to start
// it because clients have to install the generated CA cert.
@@ -515,372 +525,27 @@ func startWatcher(options *config.ApplicationConfig) {
}
}
// loadRuntimeSettingsFromFile loads settings from runtime_settings.json with env var precedence
// This function is called at startup, before env vars are applied via AppOptions.
// Since env vars are applied via AppOptions in run.go, we need to check if they're set.
// We do this by checking if the current options values differ from defaults, which would
// indicate they were set from env vars. However, a simpler approach is to just apply
// file settings here, and let the AppOptions (which are applied after this) override them.
// But actually, this is called AFTER AppOptions are applied in New(), so we need to check env vars.
// The cleanest solution: Store original values before applying file, or check if values match
// what would be set from env vars. For now, we'll apply file settings and they'll be
// overridden by AppOptions if env vars were set (but AppOptions are already applied).
// Actually, this function is called in New() before AppOptions are fully processed for watchdog.
// Let's check the call order: New() -> loadRuntimeSettingsFromFile() -> initializeWatchdog()
// But AppOptions are applied in NewApplicationConfig() which is called first.
// So at this point, options already has values from env vars. We should compare against
// defaults to see if env vars were set. But we don't have defaults stored.
// Simplest: Just apply file settings. If env vars were set, they're already in options.
// The file watcher handler will handle runtime changes properly by comparing with startupAppConfig.
// loadRuntimeSettingsFromFile merges runtime_settings.json into options
// with env-over-file precedence. Field coverage and the precedence rules
// live in the config registry (ApplyRuntimeSettingsAtStartup); this wrapper
// only handles the file read. No-op when DynamicConfigsDir is unset.
func loadRuntimeSettingsFromFile(options *config.ApplicationConfig) {
settingsFile := filepath.Join(options.DynamicConfigsDir, "runtime_settings.json")
fileContent, err := os.ReadFile(settingsFile)
if options.DynamicConfigsDir == "" {
return
}
// ReadPersistedSettings treats a missing file as zero settings, so probe
// for it here only to keep the log honest: "loaded" must mean a file was
// actually read, not that we merged an all-nil struct.
if _, err := os.Stat(filepath.Join(options.DynamicConfigsDir, "runtime_settings.json")); os.IsNotExist(err) {
xlog.Debug("runtime_settings.json not found, using defaults")
return
}
settings, err := options.ReadPersistedSettings()
if err != nil {
if os.IsNotExist(err) {
xlog.Debug("runtime_settings.json not found, using defaults")
return
}
xlog.Warn("failed to read runtime_settings.json", "error", err)
return
}
var settings config.RuntimeSettings
if err := json.Unmarshal(fileContent, &settings); err != nil {
xlog.Warn("failed to parse runtime_settings.json", "error", err)
return
}
// At this point, options already has values from env vars (via AppOptions in run.go).
// To avoid env var duplication, we determine if env vars were set by checking if
// current values differ from defaults. Defaults are: false for bools, 0 for durations.
// If current value is at default, it likely wasn't set from env var, so we can apply file.
// If current value is non-default, it was likely set from env var, so we preserve it.
// Note: This means env vars explicitly setting to false/0 won't be distinguishable from defaults,
// but that's an acceptable limitation to avoid env var duplication.
if settings.WatchdogIdleEnabled != nil {
// Only apply if current value is default (false), suggesting it wasn't set from env var
if !options.WatchDogIdle {
options.WatchDogIdle = *settings.WatchdogIdleEnabled
if options.WatchDogIdle {
options.WatchDog = true
}
}
}
if settings.WatchdogBusyEnabled != nil {
if !options.WatchDogBusy {
options.WatchDogBusy = *settings.WatchdogBusyEnabled
if options.WatchDogBusy {
options.WatchDog = true
}
}
}
if settings.WatchdogIdleTimeout != nil {
// Only apply if current value is default (0), suggesting it wasn't set from env var
if options.WatchDogIdleTimeout == 0 {
dur, err := time.ParseDuration(*settings.WatchdogIdleTimeout)
if err == nil {
options.WatchDogIdleTimeout = dur
} else {
xlog.Warn("invalid watchdog idle timeout in runtime_settings.json", "error", err, "timeout", *settings.WatchdogIdleTimeout)
}
}
}
if settings.WatchdogBusyTimeout != nil {
if options.WatchDogBusyTimeout == 0 {
dur, err := time.ParseDuration(*settings.WatchdogBusyTimeout)
if err == nil {
options.WatchDogBusyTimeout = dur
} else {
xlog.Warn("invalid watchdog busy timeout in runtime_settings.json", "error", err, "timeout", *settings.WatchdogBusyTimeout)
}
}
}
if settings.WatchdogInterval != nil {
if options.WatchDogInterval == 0 {
dur, err := time.ParseDuration(*settings.WatchdogInterval)
if err == nil {
options.WatchDogInterval = dur
} else {
xlog.Warn("invalid watchdog interval in runtime_settings.json", "error", err, "interval", *settings.WatchdogInterval)
options.WatchDogInterval = model.DefaultWatchdogInterval
}
}
}
// Handle MaxActiveBackends (new) and SingleBackend (deprecated)
if settings.MaxActiveBackends != nil {
// Only apply if current value is default (0), suggesting it wasn't set from env var
if options.MaxActiveBackends == 0 {
options.MaxActiveBackends = *settings.MaxActiveBackends
// For backward compatibility, also set SingleBackend if MaxActiveBackends == 1
options.SingleBackend = (*settings.MaxActiveBackends == 1)
}
} else if settings.SingleBackend != nil {
// Legacy: SingleBackend maps to MaxActiveBackends = 1
if !options.SingleBackend {
options.SingleBackend = *settings.SingleBackend
if *settings.SingleBackend {
options.MaxActiveBackends = 1
}
}
}
if settings.MemoryReclaimerEnabled != nil {
// Only apply if current value is default (false), suggesting it wasn't set from env var
if !options.MemoryReclaimerEnabled {
options.MemoryReclaimerEnabled = *settings.MemoryReclaimerEnabled
if options.MemoryReclaimerEnabled {
options.WatchDog = true // Memory reclaimer requires watchdog
}
}
}
if settings.MemoryReclaimerThreshold != nil {
// Only apply if current value is default (0), suggesting it wasn't set from env var
if options.MemoryReclaimerThreshold == 0 {
options.MemoryReclaimerThreshold = *settings.MemoryReclaimerThreshold
}
}
if settings.ForceEvictionWhenBusy != nil {
// Only apply if current value is default (false), suggesting it wasn't set from env var
if !options.ForceEvictionWhenBusy {
options.ForceEvictionWhenBusy = *settings.ForceEvictionWhenBusy
}
}
if settings.SizeAwareEviction != nil {
// Only apply if current value is default (false), suggesting it wasn't set from env var
if !options.SizeAwareEviction {
options.SizeAwareEviction = *settings.SizeAwareEviction
}
}
if settings.LRUEvictionMaxRetries != nil {
// Only apply if current value is default (30), suggesting it wasn't set from env var
if options.LRUEvictionMaxRetries == 0 {
options.LRUEvictionMaxRetries = *settings.LRUEvictionMaxRetries
}
}
if settings.LRUEvictionRetryInterval != nil {
// Only apply if current value is default (1s), suggesting it wasn't set from env var
if options.LRUEvictionRetryInterval == 0 {
dur, err := time.ParseDuration(*settings.LRUEvictionRetryInterval)
if err == nil {
options.LRUEvictionRetryInterval = dur
} else {
xlog.Warn("invalid LRU eviction retry interval in runtime_settings.json", "error", err, "interval", *settings.LRUEvictionRetryInterval)
}
}
}
if settings.AgentJobRetentionDays != nil {
// Only apply if current value is default (0), suggesting it wasn't set from env var
if options.AgentJobRetentionDays == 0 {
options.AgentJobRetentionDays = *settings.AgentJobRetentionDays
}
}
// Performance settings (threads / context size / f16). ApplyRuntimeSettings
// already persists these on the live /api/settings path, but the startup
// loader dropped them, so a value saved via the Middleware UI was silently
// ignored on restart (the model booted with the CLI/physical-core default).
//
// Threads is special: unlike ContextSize/F16, WithThreads eagerly resolves
// an unset (0) value to xsysinfo.CPUPhysicalCores() at option-apply time
// (see application_config.go), so options.Threads is never 0 here and the
// usual "== default" heuristic can't distinguish an env/CLI value from the
// physical-core fallback. Detect the env/CLI explicitly so
// LOCALAI_THREADS/THREADS still win over the persisted file value.
if settings.Threads != nil {
if os.Getenv("LOCALAI_THREADS") == "" && os.Getenv("THREADS") == "" { //nolint:forbidigo // deliberate env probe, see comment above
options.Threads = *settings.Threads
}
}
if settings.ContextSize != nil {
// Only apply if current value is default (0), suggesting it wasn't set from env var
if options.ContextSize == 0 {
options.ContextSize = *settings.ContextSize
}
}
if settings.F16 != nil {
// Only apply if current value is default (false), suggesting it wasn't set from env var
if !options.F16 {
options.F16 = *settings.F16
}
}
if !options.WatchDogIdle && !options.WatchDogBusy {
if settings.WatchdogEnabled != nil && *settings.WatchdogEnabled {
options.WatchDog = true
}
}
// P2P settings
if settings.P2PToken != nil {
if options.P2PToken == "" {
options.P2PToken = *settings.P2PToken
}
}
if settings.P2PNetworkID != nil {
if options.P2PNetworkID == "" {
options.P2PNetworkID = *settings.P2PNetworkID
}
}
if settings.Federated != nil {
if !options.Federated {
options.Federated = *settings.Federated
}
}
// Backend logging defaults on in single mode (NewApplicationConfig), so
// the usual "only flip false->true" merge would ignore a persisted false
// and revert the UI toggle-off on every restart. There is no env var/CLI
// flag for it, so an explicit persisted value is authoritative here.
if settings.EnableBackendLogging != nil {
options.EnableBackendLogging = *settings.EnableBackendLogging
}
// Tracing settings
if settings.EnableTracing != nil {
if !options.EnableTracing {
options.EnableTracing = *settings.EnableTracing
}
}
if settings.TracingMaxItems != nil {
if options.TracingMaxItems == 0 {
options.TracingMaxItems = *settings.TracingMaxItems
}
}
if settings.TracingMaxBodyBytes != nil {
// Allow the on-disk setting to override the CLI/env default. The
// startup default is non-zero (see NewApplicationConfig), so a plain
// `== 0` guard like the others would never trigger; we instead respect
// any value the file specifies. 0 in the file means "uncapped".
options.TracingMaxBodyBytes = *settings.TracingMaxBodyBytes
}
// Branding / whitelabeling. There are no env vars for these — the file is
// the only source — so apply unconditionally. Without this block a server
// restart silently drops the configured instance name, tagline, and asset
// filenames.
if settings.InstanceName != nil {
options.Branding.InstanceName = *settings.InstanceName
}
if settings.InstanceTagline != nil {
options.Branding.InstanceTagline = *settings.InstanceTagline
}
if settings.LogoFile != nil {
options.Branding.LogoFile = *settings.LogoFile
}
if settings.LogoHorizontalFile != nil {
options.Branding.LogoHorizontalFile = *settings.LogoHorizontalFile
}
if settings.FaviconFile != nil {
options.Branding.FaviconFile = *settings.FaviconFile
}
// MITM listener address. The CLI flag WithMITMListen populates
// options at startup; if the user configured MITM via /api/settings
// after the fact, only the file holds the value. Apply when the
// CLI flag did not already set it. (Intercept hosts now live in
// model YAML mitm.hosts: rather than runtime_settings.json.)
if settings.MITMListen != nil && options.MITMListen == "" {
options.MITMListen = *settings.MITMListen
}
// Instance-wide default PII detectors. LOCALAI_PII_DEFAULT_DETECTORS (via
// WithPIIDefaultDetectors) wins when set; otherwise the file is the source
// — apply it only when the env/CLI left the value empty, mirroring the
// "env > file" precedence used for the other fields. This must land before
// startMITMIfConfigured (called right after this loader): the cloud-proxy
// listener resolves each intercept host's detectors once at start via
// ResolvePIIPolicy, and a MITM model that names no detectors of its own
// falls back to these defaults. Without it the listener (and request-side
// default redaction) starts with an empty detector set and forwards
// traffic unredacted even though pii_default_detectors is on disk.
if settings.PIIDefaultDetectors != nil && len(options.PIIDefaultDetectors) == 0 {
options.PIIDefaultDetectors = append([]string(nil), (*settings.PIIDefaultDetectors)...)
}
// Backend upgrade flags
if settings.AutoUpgradeBackends != nil {
if !options.AutoUpgradeBackends {
options.AutoUpgradeBackends = *settings.AutoUpgradeBackends
}
}
if settings.PreferDevelopmentBackends != nil {
if !options.PreferDevelopmentBackends {
options.PreferDevelopmentBackends = *settings.PreferDevelopmentBackends
}
}
// LocalAI Assistant — file-stored as the negation (LocalAIAssistantEnabled).
// Default is enabled (DisableLocalAIAssistant=false). Apply the file value
// unless env explicitly disabled the assistant (DisableLocalAIAssistant=true).
if settings.LocalAIAssistantEnabled != nil {
if !options.DisableLocalAIAssistant {
options.DisableLocalAIAssistant = !*settings.LocalAIAssistantEnabled
}
}
// Open Responses TTL. Default is 0 (no expiration). Treat the on-disk
// "0"/empty as "no expiration" — a no-op since options is already 0 —
// and parse anything else as a duration.
if settings.OpenResponsesStoreTTL != nil && options.OpenResponsesStoreTTL == 0 {
v := *settings.OpenResponsesStoreTTL
if v != "0" && v != "" {
if dur, err := time.ParseDuration(v); err == nil {
options.OpenResponsesStoreTTL = dur
} else {
xlog.Warn("invalid open_responses_store_ttl in runtime_settings.json", "error", err, "ttl", v)
}
}
}
// Agent Pool. NewApplicationConfig seeds non-zero defaults for some of
// these fields (Enabled=true, EmbeddingModel="granite-embedding-107m-
// multilingual", MaxChunkingSize=400). The "if at default, apply file"
// gate uses each field's actual default literal so file values can
// override the bootstrap default while still letting an env-set value
// (e.g. WithAgentPoolEmbeddingModel from a flag) win.
if settings.AgentPoolEnabled != nil && options.AgentPool.Enabled {
options.AgentPool.Enabled = *settings.AgentPoolEnabled
}
if settings.AgentPoolDefaultModel != nil && options.AgentPool.DefaultModel == "" {
options.AgentPool.DefaultModel = *settings.AgentPoolDefaultModel
}
if settings.AgentPoolEmbeddingModel != nil {
if options.AgentPool.EmbeddingModel == "" || options.AgentPool.EmbeddingModel == "granite-embedding-107m-multilingual" {
options.AgentPool.EmbeddingModel = *settings.AgentPoolEmbeddingModel
}
}
if settings.AgentPoolMaxChunkingSize != nil {
if options.AgentPool.MaxChunkingSize == 0 || options.AgentPool.MaxChunkingSize == 400 {
options.AgentPool.MaxChunkingSize = *settings.AgentPoolMaxChunkingSize
}
}
if settings.AgentPoolChunkOverlap != nil && options.AgentPool.ChunkOverlap == 0 {
options.AgentPool.ChunkOverlap = *settings.AgentPoolChunkOverlap
}
if settings.AgentPoolEnableLogs != nil && !options.AgentPool.EnableLogs {
options.AgentPool.EnableLogs = *settings.AgentPoolEnableLogs
}
if settings.AgentPoolCollectionDBPath != nil && options.AgentPool.CollectionDBPath == "" {
options.AgentPool.CollectionDBPath = *settings.AgentPoolCollectionDBPath
}
if settings.AgentPoolVectorEngine != nil {
// Default is "chromem"; treat both that and empty as "not env-set".
if options.AgentPool.VectorEngine == "" || options.AgentPool.VectorEngine == "chromem" {
options.AgentPool.VectorEngine = *settings.AgentPoolVectorEngine
}
}
if settings.AgentPoolDatabaseURL != nil && options.AgentPool.DatabaseURL == "" {
options.AgentPool.DatabaseURL = *settings.AgentPoolDatabaseURL
}
if settings.AgentPoolAgentHubURL != nil {
// Default is "https://agenthub.localai.io"; treat both that and empty
// as "not env-set".
if options.AgentPool.AgentHubURL == "" || options.AgentPool.AgentHubURL == "https://agenthub.localai.io" {
options.AgentPool.AgentHubURL = *settings.AgentPoolAgentHubURL
}
}
options.ApplyRuntimeSettingsAtStartup(&settings)
xlog.Debug("Runtime settings loaded from runtime_settings.json")
}

12
core/cli/run.go
View File

@@ -595,10 +595,14 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
}
}
// Handle memory reclaimer (uses GPU VRAM if available, otherwise RAM)
if r.EnableMemoryReclaimer {
opts = append(opts, config.WithMemoryReclaimer(true, r.MemoryReclaimerThreshold))
}
// Memory reclaimer (GPU VRAM if available, otherwise RAM). Injected
// unconditionally so the kong threshold default always reaches the
// config: DefaultRuntimeBaseline models an option-less boot with
// threshold 0.95, and the settings loader treats a live value equal
// to the baseline as "not env-set" (file may apply). Gating this on
// EnableMemoryReclaimer left the threshold at 0 and made a UI-saved
// threshold look env-set at boot.
opts = append(opts, config.WithMemoryReclaimer(r.EnableMemoryReclaimer, r.MemoryReclaimerThreshold))
// Handle max active backends (LRU eviction)
// MaxActiveBackends takes precedence over SingleActiveBackend

428
core/config/application_config.go
View File

@@ -6,7 +6,6 @@ import (
"regexp"
"time"
"github.com/mudler/LocalAI/pkg/model"
"github.com/mudler/LocalAI/pkg/modelartifacts"
"github.com/mudler/LocalAI/pkg/system"
"github.com/mudler/LocalAI/pkg/vrambudget"
@@ -701,11 +700,12 @@ func WithUploadLimitMB(limit int) AppOption {
}
}
// WithThreads sets the thread-count hint. 0 means "unset": the effective
// physical-core default is resolved in application.New() only after
// persisted runtime settings merge, so a file-saved value is not masked by
// an eager fallback (#10845).
func WithThreads(threads int) AppOption {
return func(o *ApplicationConfig) {
if threads == 0 { // 0 is not allowed
threads = xsysinfo.CPUPhysicalCores()
}
o.Threads = threads
}
}
@@ -1119,410 +1119,60 @@ func (o *ApplicationConfig) ToConfigLoaderOptions() []ConfigLoaderOption {
}
}
// ToRuntimeSettings converts ApplicationConfig to RuntimeSettings for API responses and JSON serialization.
// This provides a single source of truth - ApplicationConfig holds the live values,
// and this method creates a RuntimeSettings snapshot for external consumption.
// ToRuntimeSettings converts ApplicationConfig to RuntimeSettings for API
// responses and JSON serialization. Field coverage comes entirely from the
// runtimeSettingsFields registry (runtime_settings_registry.go).
func (o *ApplicationConfig) ToRuntimeSettings() RuntimeSettings {
// Create local copies for pointer fields
watchdogEnabled := o.WatchDog
watchdogIdle := o.WatchDogIdle
watchdogBusy := o.WatchDogBusy
singleBackend := o.SingleBackend
maxActiveBackends := o.MaxActiveBackends
memoryReclaimerEnabled := o.MemoryReclaimerEnabled
memoryReclaimerThreshold := o.MemoryReclaimerThreshold
forceEvictionWhenBusy := o.ForceEvictionWhenBusy
sizeAwareEviction := o.SizeAwareEviction
lruEvictionMaxRetries := o.LRUEvictionMaxRetries
threads := o.Threads
contextSize := o.ContextSize
vramBudget := o.VRAMBudget
f16 := o.F16
debug := o.Debug
tracingMaxItems := o.TracingMaxItems
tracingMaxBodyBytes := o.TracingMaxBodyBytes
enableTracing := o.EnableTracing
enableBackendLogging := o.EnableBackendLogging
cors := o.CORS
csrf := o.DisableCSRF
corsAllowOrigins := o.CORSAllowOrigins
p2pToken := o.P2PToken
p2pNetworkID := o.P2PNetworkID
federated := o.Federated
galleries := o.Galleries
backendGalleries := o.BackendGalleries
autoloadGalleries := o.AutoloadGalleries
autoloadBackendGalleries := o.AutoloadBackendGalleries
autoUpgradeBackends := o.AutoUpgradeBackends
preferDevelopmentBackends := o.PreferDevelopmentBackends
apiKeys := o.ApiKeys
agentJobRetentionDays := o.AgentJobRetentionDays
// Format timeouts as strings
var idleTimeout, busyTimeout, watchdogInterval string
if o.WatchDogIdleTimeout > 0 {
idleTimeout = o.WatchDogIdleTimeout.String()
} else {
idleTimeout = "15m" // default
}
if o.WatchDogBusyTimeout > 0 {
busyTimeout = o.WatchDogBusyTimeout.String()
} else {
busyTimeout = "5m" // default
}
if o.WatchDogInterval > 0 {
watchdogInterval = o.WatchDogInterval.String()
} else {
watchdogInterval = model.DefaultWatchdogInterval.String() // default: 500ms
}
var lruEvictionRetryInterval string
if o.LRUEvictionRetryInterval > 0 {
lruEvictionRetryInterval = o.LRUEvictionRetryInterval.String()
} else {
lruEvictionRetryInterval = "1s" // default
}
var openResponsesStoreTTL string
if o.OpenResponsesStoreTTL > 0 {
openResponsesStoreTTL = o.OpenResponsesStoreTTL.String()
} else {
openResponsesStoreTTL = "0" // default: no expiration
}
// Agent Pool settings
agentPoolEnabled := o.AgentPool.Enabled
agentPoolDefaultModel := o.AgentPool.DefaultModel
agentPoolEmbeddingModel := o.AgentPool.EmbeddingModel
agentPoolMaxChunkingSize := o.AgentPool.MaxChunkingSize
agentPoolChunkOverlap := o.AgentPool.ChunkOverlap
agentPoolEnableLogs := o.AgentPool.EnableLogs
agentPoolCollectionDBPath := o.AgentPool.CollectionDBPath
agentPoolVectorEngine := o.AgentPool.VectorEngine
agentPoolDatabaseURL := o.AgentPool.DatabaseURL
agentPoolAgentHubURL := o.AgentPool.AgentHubURL
// LocalAI Assistant settings
localAIAssistantEnabled := !o.DisableLocalAIAssistant
// Branding settings
instanceName := o.Branding.InstanceName
instanceTagline := o.Branding.InstanceTagline
logoFile := o.Branding.LogoFile
logoHorizontalFile := o.Branding.LogoHorizontalFile
faviconFile := o.Branding.FaviconFile
mitmListen := o.MITMListen
piiDefaultDetectors := append([]string(nil), o.PIIDefaultDetectors...)
return RuntimeSettings{
WatchdogEnabled: &watchdogEnabled,
WatchdogIdleEnabled: &watchdogIdle,
WatchdogBusyEnabled: &watchdogBusy,
WatchdogIdleTimeout: &idleTimeout,
WatchdogBusyTimeout: &busyTimeout,
WatchdogInterval: &watchdogInterval,
SingleBackend: &singleBackend,
MaxActiveBackends: &maxActiveBackends,
MemoryReclaimerEnabled: &memoryReclaimerEnabled,
MemoryReclaimerThreshold: &memoryReclaimerThreshold,
ForceEvictionWhenBusy: &forceEvictionWhenBusy,
SizeAwareEviction: &sizeAwareEviction,
LRUEvictionMaxRetries: &lruEvictionMaxRetries,
LRUEvictionRetryInterval: &lruEvictionRetryInterval,
Threads: &threads,
ContextSize: &contextSize,
VRAMBudget: &vramBudget,
F16: &f16,
Debug: &debug,
TracingMaxItems: &tracingMaxItems,
TracingMaxBodyBytes: &tracingMaxBodyBytes,
EnableTracing: &enableTracing,
EnableBackendLogging: &enableBackendLogging,
CORS: &cors,
CSRF: &csrf,
CORSAllowOrigins: &corsAllowOrigins,
P2PToken: &p2pToken,
P2PNetworkID: &p2pNetworkID,
Federated: &federated,
Galleries: &galleries,
BackendGalleries: &backendGalleries,
AutoloadGalleries: &autoloadGalleries,
AutoloadBackendGalleries: &autoloadBackendGalleries,
AutoUpgradeBackends: &autoUpgradeBackends,
PreferDevelopmentBackends: &preferDevelopmentBackends,
ApiKeys: &apiKeys,
AgentJobRetentionDays: &agentJobRetentionDays,
OpenResponsesStoreTTL: &openResponsesStoreTTL,
AgentPoolEnabled: &agentPoolEnabled,
AgentPoolDefaultModel: &agentPoolDefaultModel,
AgentPoolEmbeddingModel: &agentPoolEmbeddingModel,
AgentPoolMaxChunkingSize: &agentPoolMaxChunkingSize,
AgentPoolChunkOverlap: &agentPoolChunkOverlap,
AgentPoolEnableLogs: &agentPoolEnableLogs,
AgentPoolCollectionDBPath: &agentPoolCollectionDBPath,
AgentPoolVectorEngine: &agentPoolVectorEngine,
AgentPoolDatabaseURL: &agentPoolDatabaseURL,
AgentPoolAgentHubURL: &agentPoolAgentHubURL,
LocalAIAssistantEnabled: &localAIAssistantEnabled,
InstanceName: &instanceName,
InstanceTagline: &instanceTagline,
LogoFile: &logoFile,
LogoHorizontalFile: &logoHorizontalFile,
FaviconFile: &faviconFile,
MITMListen: &mitmListen,
PIIDefaultDetectors: &piiDefaultDetectors,
var s RuntimeSettings
for _, f := range runtimeSettingsFields {
f.snapshot(o, &s)
}
return s
}
// ApplyRuntimeSettings applies RuntimeSettings to ApplicationConfig.
// Only non-nil fields in RuntimeSettings are applied.
// Returns true if watchdog-related settings changed (requiring restart).
// ApplyRuntimeSettings applies RuntimeSettings to ApplicationConfig (the
// live POST /api/settings path). Only non-nil fields are applied. Returns
// true if restart-requiring settings changed (see each row's
// restartRequired flag in runtimeSettingsFields).
func (o *ApplicationConfig) ApplyRuntimeSettings(settings *RuntimeSettings) (requireRestart bool) {
if settings == nil {
return false
}
if settings.WatchdogEnabled != nil {
o.WatchDog = *settings.WatchdogEnabled
requireRestart = true
}
if settings.WatchdogIdleEnabled != nil {
o.WatchDogIdle = *settings.WatchdogIdleEnabled
requireRestart = true
}
if settings.WatchdogBusyEnabled != nil {
o.WatchDogBusy = *settings.WatchdogBusyEnabled
requireRestart = true
for _, f := range runtimeSettingsFields {
if f.snapshotOnly || !f.isSet(settings) {
continue
}
if f.apply(o, settings) && f.requiresRestart {
requireRestart = true
}
}
// The React Settings "Enable Watchdog" master toggle manages only the
// idle/busy checks — watchdog_enabled is vestigial in that UI. Whenever
// either idle/busy field is present in the body, derive the run-state from
// idle||busy so a cold enable starts the watchdog and a full disable stops
// it, instead of trusting the stale watchdog_enabled the UI never updates.
// This mirrors the startup invariant in startup.go. An API client posting
// only watchdog_enabled (idle/busy absent) keeps its explicit value.
// idle/busy checks - watchdog_enabled is vestigial in that UI. Whenever
// either idle/busy field is present in the body, derive the run-state
// from idle||busy so a cold enable starts the watchdog and a full
// disable stops it, instead of trusting the stale watchdog_enabled the
// UI never updates. An API client posting only watchdog_enabled
// (idle/busy absent) keeps its explicit value. (#9125)
if settings.WatchdogIdleEnabled != nil || settings.WatchdogBusyEnabled != nil {
o.WatchDog = o.WatchDogIdle || o.WatchDogBusy
}
if settings.WatchdogIdleTimeout != nil {
if dur, err := time.ParseDuration(*settings.WatchdogIdleTimeout); err == nil {
o.WatchDogIdleTimeout = dur
requireRestart = true
}
}
if settings.WatchdogBusyTimeout != nil {
if dur, err := time.ParseDuration(*settings.WatchdogBusyTimeout); err == nil {
o.WatchDogBusyTimeout = dur
requireRestart = true
}
}
if settings.WatchdogInterval != nil {
if dur, err := time.ParseDuration(*settings.WatchdogInterval); err == nil {
o.WatchDogInterval = dur
requireRestart = true
}
}
if settings.MaxActiveBackends != nil {
o.MaxActiveBackends = *settings.MaxActiveBackends
o.SingleBackend = (*settings.MaxActiveBackends == 1)
requireRestart = true
} else if settings.SingleBackend != nil {
o.SingleBackend = *settings.SingleBackend
if *settings.SingleBackend {
o.MaxActiveBackends = 1
} else {
o.MaxActiveBackends = 0
}
requireRestart = true
}
if settings.MemoryReclaimerEnabled != nil {
o.MemoryReclaimerEnabled = *settings.MemoryReclaimerEnabled
if *settings.MemoryReclaimerEnabled {
o.WatchDog = true
}
requireRestart = true
}
if settings.MemoryReclaimerThreshold != nil {
if *settings.MemoryReclaimerThreshold > 0 && *settings.MemoryReclaimerThreshold <= 1.0 {
o.MemoryReclaimerThreshold = *settings.MemoryReclaimerThreshold
requireRestart = true
}
}
if settings.ForceEvictionWhenBusy != nil {
o.ForceEvictionWhenBusy = *settings.ForceEvictionWhenBusy
// This setting doesn't require restart, can be updated dynamically
}
if settings.SizeAwareEviction != nil {
o.SizeAwareEviction = *settings.SizeAwareEviction
// This setting doesn't require restart, can be updated dynamically
}
if settings.LRUEvictionMaxRetries != nil {
o.LRUEvictionMaxRetries = *settings.LRUEvictionMaxRetries
// This setting doesn't require restart, can be updated dynamically
}
if settings.LRUEvictionRetryInterval != nil {
if dur, err := time.ParseDuration(*settings.LRUEvictionRetryInterval); err == nil {
o.LRUEvictionRetryInterval = dur
// This setting doesn't require restart, can be updated dynamically
}
}
if settings.Threads != nil {
o.Threads = *settings.Threads
}
if settings.ContextSize != nil {
o.ContextSize = *settings.ContextSize
// The memory reclaimer needs the watchdog run loop.
if settings.MemoryReclaimerEnabled != nil && *settings.MemoryReclaimerEnabled {
o.WatchDog = true
}
// VRAM budget live-apply: a cross-cutting process-global side effect
// (xsysinfo), kept out of the registry row like the watchdog/reclaimer
// invariants above - rows only own config members. An empty string
// clears the cap; a malformed value is rejected by the settings
// endpoint, but stay fail-open here too so a bad persisted value
// cannot wedge apply.
if settings.VRAMBudget != nil {
o.VRAMBudget = *settings.VRAMBudget
// Live-apply so the cap takes effect without a restart. An empty string
// clears the cap; a malformed value is rejected by the settings endpoint,
// but stay fail-open here too so a bad persisted value cannot wedge apply.
if b, err := vrambudget.Parse(o.VRAMBudget); err == nil {
xsysinfo.SetDefaultVRAMBudget(b)
}
}
if settings.F16 != nil {
o.F16 = *settings.F16
}
if settings.Debug != nil {
o.Debug = *settings.Debug
}
if settings.EnableTracing != nil {
o.EnableTracing = *settings.EnableTracing
}
if settings.TracingMaxItems != nil {
o.TracingMaxItems = *settings.TracingMaxItems
}
if settings.TracingMaxBodyBytes != nil {
o.TracingMaxBodyBytes = *settings.TracingMaxBodyBytes
}
if settings.EnableBackendLogging != nil {
o.EnableBackendLogging = *settings.EnableBackendLogging
}
if settings.CORS != nil {
o.CORS = *settings.CORS
}
if settings.CSRF != nil {
o.DisableCSRF = *settings.CSRF
}
if settings.CORSAllowOrigins != nil {
o.CORSAllowOrigins = *settings.CORSAllowOrigins
}
if settings.P2PToken != nil {
o.P2PToken = *settings.P2PToken
}
if settings.P2PNetworkID != nil {
o.P2PNetworkID = *settings.P2PNetworkID
}
if settings.Federated != nil {
o.Federated = *settings.Federated
}
if settings.Galleries != nil {
o.Galleries = *settings.Galleries
}
if settings.BackendGalleries != nil {
o.BackendGalleries = *settings.BackendGalleries
}
if settings.AutoloadGalleries != nil {
o.AutoloadGalleries = *settings.AutoloadGalleries
}
if settings.AutoloadBackendGalleries != nil {
o.AutoloadBackendGalleries = *settings.AutoloadBackendGalleries
}
if settings.AutoUpgradeBackends != nil {
o.AutoUpgradeBackends = *settings.AutoUpgradeBackends
}
if settings.PreferDevelopmentBackends != nil {
o.PreferDevelopmentBackends = *settings.PreferDevelopmentBackends
}
if settings.AgentJobRetentionDays != nil {
o.AgentJobRetentionDays = *settings.AgentJobRetentionDays
}
if settings.OpenResponsesStoreTTL != nil {
if *settings.OpenResponsesStoreTTL == "0" || *settings.OpenResponsesStoreTTL == "" {
o.OpenResponsesStoreTTL = 0 // No expiration
} else if dur, err := time.ParseDuration(*settings.OpenResponsesStoreTTL); err == nil {
o.OpenResponsesStoreTTL = dur
}
// This setting doesn't require restart, can be updated dynamically
}
// Agent Pool settings
if settings.AgentPoolEnabled != nil {
o.AgentPool.Enabled = *settings.AgentPoolEnabled
requireRestart = true
}
if settings.AgentPoolDefaultModel != nil {
o.AgentPool.DefaultModel = *settings.AgentPoolDefaultModel
requireRestart = true
}
if settings.AgentPoolEmbeddingModel != nil {
o.AgentPool.EmbeddingModel = *settings.AgentPoolEmbeddingModel
requireRestart = true
}
if settings.AgentPoolMaxChunkingSize != nil {
o.AgentPool.MaxChunkingSize = *settings.AgentPoolMaxChunkingSize
requireRestart = true
}
if settings.AgentPoolChunkOverlap != nil {
o.AgentPool.ChunkOverlap = *settings.AgentPoolChunkOverlap
requireRestart = true
}
if settings.AgentPoolEnableLogs != nil {
o.AgentPool.EnableLogs = *settings.AgentPoolEnableLogs
requireRestart = true
}
if settings.AgentPoolCollectionDBPath != nil {
o.AgentPool.CollectionDBPath = *settings.AgentPoolCollectionDBPath
requireRestart = true
}
if settings.AgentPoolVectorEngine != nil {
o.AgentPool.VectorEngine = *settings.AgentPoolVectorEngine
requireRestart = true
}
if settings.AgentPoolDatabaseURL != nil {
o.AgentPool.DatabaseURL = *settings.AgentPoolDatabaseURL
requireRestart = true
}
if settings.AgentPoolAgentHubURL != nil {
o.AgentPool.AgentHubURL = *settings.AgentPoolAgentHubURL
requireRestart = true
}
// LocalAI Assistant: read live at request entry by the chat handler, so
// flipping the disable flag takes effect on the next request without a
// restart.
if settings.LocalAIAssistantEnabled != nil {
o.DisableLocalAIAssistant = !*settings.LocalAIAssistantEnabled
}
// Branding: read live by the public /api/branding endpoint and asset
// server, so changes apply on the next request without a restart.
if settings.InstanceName != nil {
o.Branding.InstanceName = *settings.InstanceName
}
if settings.InstanceTagline != nil {
o.Branding.InstanceTagline = *settings.InstanceTagline
}
if settings.LogoFile != nil {
o.Branding.LogoFile = *settings.LogoFile
}
if settings.LogoHorizontalFile != nil {
o.Branding.LogoHorizontalFile = *settings.LogoHorizontalFile
}
if settings.FaviconFile != nil {
o.Branding.FaviconFile = *settings.FaviconFile
}
if settings.MITMListen != nil {
o.MITMListen = *settings.MITMListen
}
if settings.PIIDefaultDetectors != nil {
o.PIIDefaultDetectors = append([]string(nil), (*settings.PIIDefaultDetectors)...)
}
// Note: ApiKeys requires special handling (merging with startup keys) - handled in caller
// Note: ApiKeys need env-merge handling (MergeAPIKeys) - done by the
// caller, because the env-provided keys live on the startup config.
return requireRestart
}

19
core/config/runtime_settings_persist.go
View File

@@ -63,6 +63,25 @@ func (s *RuntimeSettings) MergeNonNil(overlay RuntimeSettings) {
}
}
// MergeAPIKeys returns the env/CLI-provided keys followed by the
// runtime-managed (settings) keys, dropping runtime entries that duplicate
// an env key so repeated saves cannot stack the env keys (#9071). Env keys
// always survive: the settings file owns only the runtime tail.
func MergeAPIKeys(envKeys, runtimeKeys []string) []string {
merged := append([]string(nil), envKeys...)
seen := make(map[string]struct{}, len(envKeys))
for _, k := range envKeys {
seen[k] = struct{}{}
}
for _, k := range runtimeKeys {
if _, dup := seen[k]; dup {
continue
}
merged = append(merged, k)
}
return merged
}
// WritePersistedSettings serialises the given RuntimeSettings to
// runtime_settings.json with restricted permissions (it may carry API
// keys and P2P tokens).

27
core/config/runtime_settings_persist_test.go
View File

@@ -14,7 +14,7 @@ import (
)
func strPtr(s string) *string { return &s }
func boolPtr(b bool) *bool { return &b }
func boolPtr(b bool) *bool { return &b }
var _ = Describe("RuntimeSettings persistence helpers", func() {
var (
@@ -197,4 +197,29 @@ var _ = Describe("RuntimeSettings persistence helpers", func() {
Expect(*ondisk.InstanceName).To(Equal("Acme AI"))
})
})
// MergeAPIKeys is the single env+runtime key merge shared by the
// settings endpoint and the runtime_settings.json file watcher. Env/CLI
// keys must always survive, and runtime entries that duplicate an env
// key must be dropped so repeated saves cannot stack them (#9071).
Describe("MergeAPIKeys", func() {
It("keeps env keys first and appends runtime keys", func() {
Expect(config.MergeAPIKeys([]string{"env1"}, []string{"rt1", "rt2"})).
To(Equal([]string{"env1", "rt1", "rt2"}))
})
It("drops runtime entries that duplicate an env key (#9071)", func() {
Expect(config.MergeAPIKeys([]string{"env1"}, []string{"env1", "rt1"})).
To(Equal([]string{"env1", "rt1"}))
})
It("clears runtime keys when the runtime list is empty", func() {
Expect(config.MergeAPIKeys([]string{"env1"}, []string{})).
To(Equal([]string{"env1"}))
})
It("handles no env keys", func() {
Expect(config.MergeAPIKeys(nil, []string{"rt1"})).To(Equal([]string{"rt1"}))
})
})
})

468
core/config/runtime_settings_registry.go Normal file
View File

@@ -0,0 +1,468 @@
package config
import (
"slices"
"time"
"github.com/mudler/LocalAI/pkg/model"
"github.com/mudler/xlog"
)
// fieldSpec is one runtime setting: how to snapshot it out of
// ApplicationConfig, how to apply it back, and how to decide at load time
// whether env/CLI already claimed the value. Every RuntimeSettings field is
// owned by exactly one spec (the registry completeness spec pins this), so
// GET /api/settings, POST /api/settings, boot, and the file watcher can
// never drift apart per-field again (the bug class behind #10845).
type fieldSpec struct {
// jsonNames this spec owns; a single entry except for composite rows
// (max_active_backends + its deprecated alias single_backend).
jsonNames []string
// requiresRestart marks fields whose live change makes
// ApplyRuntimeSettings report a restart to the settings endpoint.
requiresRestart bool
// fileAuthoritative: at startup/watch time the persisted value applies
// unconditionally - either no env/CLI source exists (branding) or the
// file is defined as the owner (enable_backend_logging,
// tracing_max_body_bytes).
fileAuthoritative bool
// snapshotOnly: echoed by ToRuntimeSettings but never applied by the
// loops (api_keys: the endpoint and file watcher own the env merge).
snapshotOnly bool
snapshot func(o *ApplicationConfig, s *RuntimeSettings)
isSet func(s *RuntimeSettings) bool
// apply writes the (non-nil) settings value into o and reports whether
// it was accepted (validation may reject it, e.g. a bad duration).
apply func(o *ApplicationConfig, s *RuntimeSettings) bool
// envSet reports whether env/CLI already set this field: the live value
// differs from the option-less-run baseline (DefaultRuntimeBaseline).
envSet func(current, baseline *ApplicationConfig) bool
}
type fieldOpt func(*fieldSpec)
func restartRequired() fieldOpt { return func(f *fieldSpec) { f.requiresRestart = true } }
func fromFileAlways() fieldOpt { return func(f *fieldSpec) { f.fileAuthoritative = true } }
func field[T comparable](name string, sel func(*RuntimeSettings) **T, get func(*ApplicationConfig) T, set func(*ApplicationConfig, T), opts ...fieldOpt) fieldSpec {
return fieldEq(name, sel, get, set, func(a, b T) bool { return a == b }, opts...)
}
func fieldEq[T any](name string, sel func(*RuntimeSettings) **T, get func(*ApplicationConfig) T, set func(*ApplicationConfig, T), eq func(a, b T) bool, opts ...fieldOpt) fieldSpec {
f := fieldSpec{
jsonNames: []string{name},
snapshot: func(o *ApplicationConfig, s *RuntimeSettings) { v := get(o); *sel(s) = &v },
isSet: func(s *RuntimeSettings) bool { return *sel(s) != nil },
apply: func(o *ApplicationConfig, s *RuntimeSettings) bool { set(o, **sel(s)); return true },
envSet: func(cur, base *ApplicationConfig) bool { return !eq(get(cur), get(base)) },
}
for _, opt := range opts {
opt(&f)
}
return f
}
// durationField maps a time.Duration config member to a duration-string
// settings field. fallback is the presentation default ToRuntimeSettings
// reports while the member is unset (0). envSet compares the raw durations
// rather than the formatted strings so the fallback text cannot mask an
// env-set value that happens to equal it.
func durationField(name string, sel func(*RuntimeSettings) **string, dur func(*ApplicationConfig) *time.Duration, fallback string, opts ...fieldOpt) fieldSpec {
f := fieldSpec{
jsonNames: []string{name},
snapshot: func(o *ApplicationConfig, s *RuntimeSettings) {
v := fallback
if d := *dur(o); d > 0 {
v = d.String()
}
*sel(s) = &v
},
isSet: func(s *RuntimeSettings) bool { return *sel(s) != nil },
apply: func(o *ApplicationConfig, s *RuntimeSettings) bool {
d, err := time.ParseDuration(**sel(s))
if err != nil {
xlog.Warn("invalid duration in runtime settings", "field", name, "value", **sel(s), "error", err)
return false
}
*dur(o) = d
return true
},
envSet: func(cur, base *ApplicationConfig) bool { return *dur(cur) != *dur(base) },
}
for _, opt := range opts {
opt(&f)
}
return f
}
// runtimeSettingsFields is THE per-field description of every runtime
// setting. ToRuntimeSettings, ApplyRuntimeSettings, and
// ApplyRuntimeSettingsAtStartup are loops over this table; adding a setting
// means adding a RuntimeSettings struct field plus one row here (the
// registry completeness spec fails until both exist).
var runtimeSettingsFields = []fieldSpec{
// Watchdog. The live master flag is post-processed by the apply loops
// (WatchDog follows idle||busy; see ApplyRuntimeSettings).
field("watchdog_enabled",
func(s *RuntimeSettings) **bool { return &s.WatchdogEnabled },
func(o *ApplicationConfig) bool { return o.WatchDog },
func(o *ApplicationConfig, v bool) { o.WatchDog = v },
restartRequired()),
field("watchdog_idle_enabled",
func(s *RuntimeSettings) **bool { return &s.WatchdogIdleEnabled },
func(o *ApplicationConfig) bool { return o.WatchDogIdle },
func(o *ApplicationConfig, v bool) { o.WatchDogIdle = v },
restartRequired()),
field("watchdog_busy_enabled",
func(s *RuntimeSettings) **bool { return &s.WatchdogBusyEnabled },
func(o *ApplicationConfig) bool { return o.WatchDogBusy },
func(o *ApplicationConfig, v bool) { o.WatchDogBusy = v },
restartRequired()),
durationField("watchdog_idle_timeout",
func(s *RuntimeSettings) **string { return &s.WatchdogIdleTimeout },
func(o *ApplicationConfig) *time.Duration { return &o.WatchDogIdleTimeout },
"15m", restartRequired()),
durationField("watchdog_busy_timeout",
func(s *RuntimeSettings) **string { return &s.WatchdogBusyTimeout },
func(o *ApplicationConfig) *time.Duration { return &o.WatchDogBusyTimeout },
"5m", restartRequired()),
durationField("watchdog_interval",
func(s *RuntimeSettings) **string { return &s.WatchdogInterval },
func(o *ApplicationConfig) *time.Duration { return &o.WatchDogInterval },
model.DefaultWatchdogInterval.String(), restartRequired()),
// Backend management. max_active_backends and its deprecated alias
// single_backend are one composite row: they must stay mutually
// consistent and max_active_backends wins when both are posted.
{
jsonNames: []string{"max_active_backends", "single_backend"},
requiresRestart: true,
snapshot: func(o *ApplicationConfig, s *RuntimeSettings) {
mab := o.MaxActiveBackends
sb := o.SingleBackend
s.MaxActiveBackends = &mab
s.SingleBackend = &sb
},
isSet: func(s *RuntimeSettings) bool { return s.MaxActiveBackends != nil || s.SingleBackend != nil },
apply: func(o *ApplicationConfig, s *RuntimeSettings) bool {
if s.MaxActiveBackends != nil {
o.MaxActiveBackends = *s.MaxActiveBackends
o.SingleBackend = *s.MaxActiveBackends == 1
return true
}
o.SingleBackend = *s.SingleBackend
if *s.SingleBackend {
o.MaxActiveBackends = 1
} else {
o.MaxActiveBackends = 0
}
return true
},
envSet: func(cur, base *ApplicationConfig) bool {
return cur.MaxActiveBackends != base.MaxActiveBackends || cur.SingleBackend != base.SingleBackend
},
},
field("auto_upgrade_backends",
func(s *RuntimeSettings) **bool { return &s.AutoUpgradeBackends },
func(o *ApplicationConfig) bool { return o.AutoUpgradeBackends },
func(o *ApplicationConfig, v bool) { o.AutoUpgradeBackends = v }),
field("prefer_development_backends",
func(s *RuntimeSettings) **bool { return &s.PreferDevelopmentBackends },
func(o *ApplicationConfig) bool { return o.PreferDevelopmentBackends },
func(o *ApplicationConfig, v bool) { o.PreferDevelopmentBackends = v }),
// Memory reclaimer. Enabling it forces the watchdog master flag - that
// cross-field invariant lives in the apply loops, not here.
field("memory_reclaimer_enabled",
func(s *RuntimeSettings) **bool { return &s.MemoryReclaimerEnabled },
func(o *ApplicationConfig) bool { return o.MemoryReclaimerEnabled },
func(o *ApplicationConfig, v bool) { o.MemoryReclaimerEnabled = v },
restartRequired()),
{
jsonNames: []string{"memory_reclaimer_threshold"},
requiresRestart: true,
snapshot: func(o *ApplicationConfig, s *RuntimeSettings) {
v := o.MemoryReclaimerThreshold
s.MemoryReclaimerThreshold = &v
},
isSet: func(s *RuntimeSettings) bool { return s.MemoryReclaimerThreshold != nil },
apply: func(o *ApplicationConfig, s *RuntimeSettings) bool {
v := *s.MemoryReclaimerThreshold
if v <= 0 || v > 1.0 {
xlog.Warn("memory_reclaimer_threshold out of range (0,1], ignoring", "value", v)
return false
}
o.MemoryReclaimerThreshold = v
return true
},
envSet: func(cur, base *ApplicationConfig) bool {
return cur.MemoryReclaimerThreshold != base.MemoryReclaimerThreshold
},
},
// Eviction.
field("force_eviction_when_busy",
func(s *RuntimeSettings) **bool { return &s.ForceEvictionWhenBusy },
func(o *ApplicationConfig) bool { return o.ForceEvictionWhenBusy },
func(o *ApplicationConfig, v bool) { o.ForceEvictionWhenBusy = v }),
field("size_aware_eviction",
func(s *RuntimeSettings) **bool { return &s.SizeAwareEviction },
func(o *ApplicationConfig) bool { return o.SizeAwareEviction },
func(o *ApplicationConfig, v bool) { o.SizeAwareEviction = v }),
field("lru_eviction_max_retries",
func(s *RuntimeSettings) **int { return &s.LRUEvictionMaxRetries },
func(o *ApplicationConfig) int { return o.LRUEvictionMaxRetries },
func(o *ApplicationConfig, v int) { o.LRUEvictionMaxRetries = v }),
durationField("lru_eviction_retry_interval",
func(s *RuntimeSettings) **string { return &s.LRUEvictionRetryInterval },
func(o *ApplicationConfig) *time.Duration { return &o.LRUEvictionRetryInterval },
"1s"),
// Performance.
field("threads",
func(s *RuntimeSettings) **int { return &s.Threads },
func(o *ApplicationConfig) int { return o.Threads },
func(o *ApplicationConfig, v int) { o.Threads = v }),
field("context_size",
func(s *RuntimeSettings) **int { return &s.ContextSize },
func(o *ApplicationConfig) int { return o.ContextSize },
func(o *ApplicationConfig, v int) { o.ContextSize = v }),
// VRAM budget: the cap string ("80%"/"12GB"/"" = uncapped). The live
// side effect (xsysinfo.SetDefaultVRAMBudget) is post-processing in the
// apply loop, not here - the row only owns the config member, matching
// how the memory-reclaimer/watchdog cross-field effects are handled.
field("vram_budget",
func(s *RuntimeSettings) **string { return &s.VRAMBudget },
func(o *ApplicationConfig) string { return o.VRAMBudget },
func(o *ApplicationConfig, v string) { o.VRAMBudget = v }),
field("f16",
func(s *RuntimeSettings) **bool { return &s.F16 },
func(o *ApplicationConfig) bool { return o.F16 },
func(o *ApplicationConfig, v bool) { o.F16 = v }),
field("debug",
func(s *RuntimeSettings) **bool { return &s.Debug },
func(o *ApplicationConfig) bool { return o.Debug },
func(o *ApplicationConfig, v bool) { o.Debug = v }),
field("enable_tracing",
func(s *RuntimeSettings) **bool { return &s.EnableTracing },
func(o *ApplicationConfig) bool { return o.EnableTracing },
func(o *ApplicationConfig, v bool) { o.EnableTracing = v }),
field("tracing_max_items",
func(s *RuntimeSettings) **int { return &s.TracingMaxItems },
func(o *ApplicationConfig) int { return o.TracingMaxItems },
func(o *ApplicationConfig, v int) { o.TracingMaxItems = v }),
// The on-disk setting overrides the CLI/env default by design: 0 in the
// file means "uncapped" and must not be mistaken for "unset".
field("tracing_max_body_bytes",
func(s *RuntimeSettings) **int { return &s.TracingMaxBodyBytes },
func(o *ApplicationConfig) int { return o.TracingMaxBodyBytes },
func(o *ApplicationConfig, v int) { o.TracingMaxBodyBytes = v },
fromFileAlways()),
// Backend logging defaults on in single mode; a persisted false is the
// UI toggle-off and must win over that default on restart. No env/CLI
// source exists, so the file is authoritative.
field("enable_backend_logging",
func(s *RuntimeSettings) **bool { return &s.EnableBackendLogging },
func(o *ApplicationConfig) bool { return o.EnableBackendLogging },
func(o *ApplicationConfig, v bool) { o.EnableBackendLogging = v },
fromFileAlways()),
// Security / CORS. The "csrf" wire field carries DisableCSRF (historic
// naming kept for API compatibility).
field("cors",
func(s *RuntimeSettings) **bool { return &s.CORS },
func(o *ApplicationConfig) bool { return o.CORS },
func(o *ApplicationConfig, v bool) { o.CORS = v }),
field("csrf",
func(s *RuntimeSettings) **bool { return &s.CSRF },
func(o *ApplicationConfig) bool { return o.DisableCSRF },
func(o *ApplicationConfig, v bool) { o.DisableCSRF = v }),
field("cors_allow_origins",
func(s *RuntimeSettings) **string { return &s.CORSAllowOrigins },
func(o *ApplicationConfig) string { return o.CORSAllowOrigins },
func(o *ApplicationConfig, v string) { o.CORSAllowOrigins = v }),
// P2P.
field("p2p_token",
func(s *RuntimeSettings) **string { return &s.P2PToken },
func(o *ApplicationConfig) string { return o.P2PToken },
func(o *ApplicationConfig, v string) { o.P2PToken = v }),
field("p2p_network_id",
func(s *RuntimeSettings) **string { return &s.P2PNetworkID },
func(o *ApplicationConfig) string { return o.P2PNetworkID },
func(o *ApplicationConfig, v string) { o.P2PNetworkID = v }),
field("federated",
func(s *RuntimeSettings) **bool { return &s.Federated },
func(o *ApplicationConfig) bool { return o.Federated },
func(o *ApplicationConfig, v bool) { o.Federated = v }),
// Galleries. Gallery is comparable (string fields + a pointer), so
// slices.Equal gives element-wise comparison against the baseline's
// default gallery list.
fieldEq("galleries",
func(s *RuntimeSettings) **[]Gallery { return &s.Galleries },
func(o *ApplicationConfig) []Gallery { return o.Galleries },
func(o *ApplicationConfig, v []Gallery) { o.Galleries = v },
slices.Equal),
fieldEq("backend_galleries",
func(s *RuntimeSettings) **[]Gallery { return &s.BackendGalleries },
func(o *ApplicationConfig) []Gallery { return o.BackendGalleries },
func(o *ApplicationConfig, v []Gallery) { o.BackendGalleries = v },
slices.Equal),
field("autoload_galleries",
func(s *RuntimeSettings) **bool { return &s.AutoloadGalleries },
func(o *ApplicationConfig) bool { return o.AutoloadGalleries },
func(o *ApplicationConfig, v bool) { o.AutoloadGalleries = v }),
field("autoload_backend_galleries",
func(s *RuntimeSettings) **bool { return &s.AutoloadBackendGalleries },
func(o *ApplicationConfig) bool { return o.AutoloadBackendGalleries },
func(o *ApplicationConfig, v bool) { o.AutoloadBackendGalleries = v }),
// API keys: echoed for the UI, but the apply loops never touch them.
// The settings endpoint and the file watcher own the env+runtime merge
// (MergeAPIKeys) because env keys must always survive.
{
jsonNames: []string{"api_keys"},
snapshotOnly: true,
snapshot: func(o *ApplicationConfig, s *RuntimeSettings) {
keys := o.ApiKeys
s.ApiKeys = &keys
},
isSet: func(s *RuntimeSettings) bool { return s.ApiKeys != nil },
},
field("agent_job_retention_days",
func(s *RuntimeSettings) **int { return &s.AgentJobRetentionDays },
func(o *ApplicationConfig) int { return o.AgentJobRetentionDays },
func(o *ApplicationConfig, v int) { o.AgentJobRetentionDays = v }),
// Open Responses TTL: "0" or "" mean "no expiration" on the wire.
{
jsonNames: []string{"open_responses_store_ttl"},
snapshot: func(o *ApplicationConfig, s *RuntimeSettings) {
v := "0"
if o.OpenResponsesStoreTTL > 0 {
v = o.OpenResponsesStoreTTL.String()
}
s.OpenResponsesStoreTTL = &v
},
isSet: func(s *RuntimeSettings) bool { return s.OpenResponsesStoreTTL != nil },
apply: func(o *ApplicationConfig, s *RuntimeSettings) bool {
v := *s.OpenResponsesStoreTTL
if v == "0" || v == "" {
o.OpenResponsesStoreTTL = 0
return true
}
d, err := time.ParseDuration(v)
if err != nil {
xlog.Warn("invalid open_responses_store_ttl in runtime settings", "value", v, "error", err)
return false
}
o.OpenResponsesStoreTTL = d
return true
},
envSet: func(cur, base *ApplicationConfig) bool {
return cur.OpenResponsesStoreTTL != base.OpenResponsesStoreTTL
},
},
// Agent Pool.
field("agent_pool_enabled",
func(s *RuntimeSettings) **bool { return &s.AgentPoolEnabled },
func(o *ApplicationConfig) bool { return o.AgentPool.Enabled },
func(o *ApplicationConfig, v bool) { o.AgentPool.Enabled = v },
restartRequired()),
field("agent_pool_default_model",
func(s *RuntimeSettings) **string { return &s.AgentPoolDefaultModel },
func(o *ApplicationConfig) string { return o.AgentPool.DefaultModel },
func(o *ApplicationConfig, v string) { o.AgentPool.DefaultModel = v },
restartRequired()),
field("agent_pool_embedding_model",
func(s *RuntimeSettings) **string { return &s.AgentPoolEmbeddingModel },
func(o *ApplicationConfig) string { return o.AgentPool.EmbeddingModel },
func(o *ApplicationConfig, v string) { o.AgentPool.EmbeddingModel = v },
restartRequired()),
field("agent_pool_max_chunking_size",
func(s *RuntimeSettings) **int { return &s.AgentPoolMaxChunkingSize },
func(o *ApplicationConfig) int { return o.AgentPool.MaxChunkingSize },
func(o *ApplicationConfig, v int) { o.AgentPool.MaxChunkingSize = v },
restartRequired()),
field("agent_pool_chunk_overlap",
func(s *RuntimeSettings) **int { return &s.AgentPoolChunkOverlap },
func(o *ApplicationConfig) int { return o.AgentPool.ChunkOverlap },
func(o *ApplicationConfig, v int) { o.AgentPool.ChunkOverlap = v },
restartRequired()),
field("agent_pool_enable_logs",
func(s *RuntimeSettings) **bool { return &s.AgentPoolEnableLogs },
func(o *ApplicationConfig) bool { return o.AgentPool.EnableLogs },
func(o *ApplicationConfig, v bool) { o.AgentPool.EnableLogs = v },
restartRequired()),
field("agent_pool_collection_db_path",
func(s *RuntimeSettings) **string { return &s.AgentPoolCollectionDBPath },
func(o *ApplicationConfig) string { return o.AgentPool.CollectionDBPath },
func(o *ApplicationConfig, v string) { o.AgentPool.CollectionDBPath = v },
restartRequired()),
field("agent_pool_vector_engine",
func(s *RuntimeSettings) **string { return &s.AgentPoolVectorEngine },
func(o *ApplicationConfig) string { return o.AgentPool.VectorEngine },
func(o *ApplicationConfig, v string) { o.AgentPool.VectorEngine = v },
restartRequired()),
field("agent_pool_database_url",
func(s *RuntimeSettings) **string { return &s.AgentPoolDatabaseURL },
func(o *ApplicationConfig) string { return o.AgentPool.DatabaseURL },
func(o *ApplicationConfig, v string) { o.AgentPool.DatabaseURL = v },
restartRequired()),
field("agent_pool_agent_hub_url",
func(s *RuntimeSettings) **string { return &s.AgentPoolAgentHubURL },
func(o *ApplicationConfig) string { return o.AgentPool.AgentHubURL },
func(o *ApplicationConfig, v string) { o.AgentPool.AgentHubURL = v },
restartRequired()),
// LocalAI Assistant: stored as the negation for UI clarity.
field("localai_assistant_enabled",
func(s *RuntimeSettings) **bool { return &s.LocalAIAssistantEnabled },
func(o *ApplicationConfig) bool { return !o.DisableLocalAIAssistant },
func(o *ApplicationConfig, v bool) { o.DisableLocalAIAssistant = !v }),
// Branding: the file is the only source (no env/CLI), so it is always
// authoritative. Without fromFileAlways a restart silently drops the
// configured instance name and asset filenames.
field("instance_name",
func(s *RuntimeSettings) **string { return &s.InstanceName },
func(o *ApplicationConfig) string { return o.Branding.InstanceName },
func(o *ApplicationConfig, v string) { o.Branding.InstanceName = v },
fromFileAlways()),
field("instance_tagline",
func(s *RuntimeSettings) **string { return &s.InstanceTagline },
func(o *ApplicationConfig) string { return o.Branding.InstanceTagline },
func(o *ApplicationConfig, v string) { o.Branding.InstanceTagline = v },
fromFileAlways()),
field("logo_file",
func(s *RuntimeSettings) **string { return &s.LogoFile },
func(o *ApplicationConfig) string { return o.Branding.LogoFile },
func(o *ApplicationConfig, v string) { o.Branding.LogoFile = v },
fromFileAlways()),
field("logo_horizontal_file",
func(s *RuntimeSettings) **string { return &s.LogoHorizontalFile },
func(o *ApplicationConfig) string { return o.Branding.LogoHorizontalFile },
func(o *ApplicationConfig, v string) { o.Branding.LogoHorizontalFile = v },
fromFileAlways()),
field("favicon_file",
func(s *RuntimeSettings) **string { return &s.FaviconFile },
func(o *ApplicationConfig) string { return o.Branding.FaviconFile },
func(o *ApplicationConfig, v string) { o.Branding.FaviconFile = v },
fromFileAlways()),
field("mitm_listen",
func(s *RuntimeSettings) **string { return &s.MITMListen },
func(o *ApplicationConfig) string { return o.MITMListen },
func(o *ApplicationConfig, v string) { o.MITMListen = v }),
// PII default detectors: cloned on both directions so callers can't
// alias the live config's slice.
fieldEq("pii_default_detectors",
func(s *RuntimeSettings) **[]string { return &s.PIIDefaultDetectors },
func(o *ApplicationConfig) []string { return append([]string(nil), o.PIIDefaultDetectors...) },
func(o *ApplicationConfig, v []string) { o.PIIDefaultDetectors = append([]string(nil), v...) },
slices.Equal),
}

208
core/config/runtime_settings_registry_internal_test.go Normal file
View File

@@ -0,0 +1,208 @@
package config
import (
"reflect"
"strings"
"time"
"github.com/mudler/LocalAI/pkg/vrambudget"
"github.com/mudler/LocalAI/pkg/xsysinfo"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
var _ = Describe("runtime settings registry", func() {
// The registry is the single description of every runtime setting. If a
// RuntimeSettings field has no registry row, the setting will be echoed,
// persisted, or re-applied inconsistently - the exact bug class behind
// #10845 (threads/context_size/f16 saved but ignored on restart). This
// spec makes that a red test instead of a silent runtime regression.
It("owns every RuntimeSettings field exactly once", func() {
structFields := map[string]bool{}
t := reflect.TypeOf(RuntimeSettings{})
for i := 0; i < t.NumField(); i++ {
name, _, _ := strings.Cut(t.Field(i).Tag.Get("json"), ",")
Expect(name).NotTo(BeEmpty(), "RuntimeSettings.%s needs a json tag", t.Field(i).Name)
structFields[name] = true
}
owned := map[string]int{}
for _, f := range runtimeSettingsFields {
for _, n := range f.jsonNames {
owned[n]++
}
}
for name := range structFields {
Expect(owned[name]).To(Equal(1),
"RuntimeSettings field %q must be owned by exactly one registry row - add it to runtimeSettingsFields in runtime_settings_registry.go", name)
}
for name := range owned {
Expect(structFields[name]).To(BeTrue(),
"registry row %q owns no RuntimeSettings field - remove or fix the row", name)
}
})
// Catches a registry row wired to the wrong ApplicationConfig member:
// a config with a distinct value in every settings-backed member must
// survive To -> Apply -> To unchanged.
It("round-trips every field through Apply and To", func() {
// Applying vram_budget installs a process-global default cap via
// xsysinfo.SetDefaultVRAMBudget; reset it so later specs don't
// inherit a phantom 12GiB budget.
DeferCleanup(func() {
xsysinfo.SetDefaultVRAMBudget(vrambudget.Budget{})
})
src := NewApplicationConfig()
src.WatchDog = true
src.WatchDogIdle = true
src.WatchDogBusy = true
src.WatchDogIdleTimeout = 21 * time.Minute
src.WatchDogBusyTimeout = 6 * time.Minute
src.WatchDogInterval = 2 * time.Second
src.MaxActiveBackends = 3
src.SingleBackend = false
src.AutoUpgradeBackends = true
src.PreferDevelopmentBackends = true
src.MemoryReclaimerEnabled = true
src.MemoryReclaimerThreshold = 0.5
src.ForceEvictionWhenBusy = true
src.SizeAwareEviction = true
src.LRUEvictionMaxRetries = 42
src.LRUEvictionRetryInterval = 3 * time.Second
src.Threads = 7
src.ContextSize = 8192
src.VRAMBudget = "12GiB"
src.F16 = true
src.Debug = true
src.EnableTracing = true
src.TracingMaxItems = 2048
src.TracingMaxBodyBytes = 1234
src.EnableBackendLogging = false
src.CORS = true
src.DisableCSRF = true
src.CORSAllowOrigins = "https://example.com"
src.P2PToken = "tok"
src.P2PNetworkID = "netid"
src.Federated = true
src.Galleries = []Gallery{{Name: "g1", URL: "https://g1"}}
src.BackendGalleries = []Gallery{{Name: "b1", URL: "https://b1"}}
src.AutoloadGalleries = true
src.AutoloadBackendGalleries = true
src.AgentJobRetentionDays = 9
src.OpenResponsesStoreTTL = time.Hour
src.AgentPool.Enabled = false
src.AgentPool.DefaultModel = "dm"
src.AgentPool.EmbeddingModel = "em"
src.AgentPool.MaxChunkingSize = 123
src.AgentPool.ChunkOverlap = 12
src.AgentPool.EnableLogs = true
src.AgentPool.CollectionDBPath = "/tmp/c.db"
src.AgentPool.VectorEngine = "postgres"
src.AgentPool.DatabaseURL = "postgres://x"
src.AgentPool.AgentHubURL = "https://hub"
src.DisableLocalAIAssistant = true
src.Branding = BrandingConfig{
InstanceName: "n", InstanceTagline: "t",
LogoFile: "l.png", LogoHorizontalFile: "lh.png", FaviconFile: "f.ico",
}
src.MITMListen = ":8443"
src.PIIDefaultDetectors = []string{"d1"}
s1 := src.ToRuntimeSettings()
dst := NewApplicationConfig()
dst.ApplyRuntimeSettings(&s1)
s2 := dst.ToRuntimeSettings()
Expect(s2).To(Equal(s1))
})
})
var _ = Describe("ApplyRuntimeSettingsAtStartup", func() {
// DefaultRuntimeBaseline simulates the config an option-less
// `local-ai run` boots with (NewApplicationConfig + the kong flag
// defaults run.go always injects). A live value equal to the baseline
// means env/CLI did not touch it, so the file may apply.
It("applies persisted values whose non-zero defaults broke the old == 0 guards", func() {
// Regression pins for the silent startup losses on master:
// lru_eviction_max_retries (default 30), tracing_max_items (1024),
// agent_job_retention_days (30), memory_reclaimer_threshold (0.95).
o := DefaultRuntimeBaseline()
retries, items, days, thr := 99, 4096, 7, 0.5
o.ApplyRuntimeSettingsAtStartup(&RuntimeSettings{
LRUEvictionMaxRetries: &retries,
TracingMaxItems: &items,
AgentJobRetentionDays: &days,
MemoryReclaimerThreshold: &thr,
})
Expect(o.LRUEvictionMaxRetries).To(Equal(99))
Expect(o.TracingMaxItems).To(Equal(4096))
Expect(o.AgentJobRetentionDays).To(Equal(7))
Expect(o.MemoryReclaimerThreshold).To(Equal(0.5))
})
It("lets an env/CLI-set value win over the file", func() {
o := DefaultRuntimeBaseline()
o.Threads = 8 // simulate LOCALAI_THREADS=8
fileThreads := 4
o.ApplyRuntimeSettingsAtStartup(&RuntimeSettings{Threads: &fileThreads})
Expect(o.Threads).To(Equal(8), "env value must win over the persisted file value")
})
It("applies persisted threads when env/CLI left them unset", func() {
// Relies on WithThreads no longer eagerly resolving 0 (Task 5).
o := DefaultRuntimeBaseline()
fileThreads := 4
o.ApplyRuntimeSettingsAtStartup(&RuntimeSettings{Threads: &fileThreads})
Expect(o.Threads).To(Equal(4))
})
It("applies persisted galleries over the kong default list", func() {
o := DefaultRuntimeBaseline()
saved := []Gallery{{Name: "mine", URL: "https://example.com/index.yaml"}}
o.ApplyRuntimeSettingsAtStartup(&RuntimeSettings{Galleries: &saved})
Expect(o.Galleries).To(Equal(saved))
})
It("keeps env-configured galleries over the file", func() {
o := DefaultRuntimeBaseline()
o.Galleries = []Gallery{{Name: "env", URL: "https://env.example/index.yaml"}}
saved := []Gallery{{Name: "file", URL: "https://file.example/index.yaml"}}
o.ApplyRuntimeSettingsAtStartup(&RuntimeSettings{Galleries: &saved})
Expect(o.Galleries[0].Name).To(Equal("env"))
})
It("always applies file-authoritative fields (backend logging toggle-off)", func() {
o := DefaultRuntimeBaseline() // EnableBackendLogging defaults true
off := false
o.ApplyRuntimeSettingsAtStartup(&RuntimeSettings{EnableBackendLogging: &off})
Expect(o.EnableBackendLogging).To(BeFalse())
})
It("raises the watchdog master flag when the file enables idle checks", func() {
o := DefaultRuntimeBaseline()
on := true
o.ApplyRuntimeSettingsAtStartup(&RuntimeSettings{WatchdogIdleEnabled: &on})
Expect(o.WatchDogIdle).To(BeTrue())
Expect(o.WatchDog).To(BeTrue())
})
It("applies a persisted vram_budget when env/CLI left it unset", func() {
// The vram_budget CLI flag has no kong default (empty string means
// uncapped), so the baseline needs no overlay for it. Assert only
// the config member: the xsysinfo default-budget side effect is
// process-global and asserting it here would make sibling specs
// order-dependent. Reset the cap afterwards for the same reason
// (mirrors the persist suite's AfterEach).
DeferCleanup(func() {
xsysinfo.SetDefaultVRAMBudget(vrambudget.Budget{})
})
o := DefaultRuntimeBaseline()
budget := "80%"
o.ApplyRuntimeSettingsAtStartup(&RuntimeSettings{VRAMBudget: &budget})
Expect(o.VRAMBudget).To(Equal("80%"))
})
It("is nil-safe", func() {
o := DefaultRuntimeBaseline()
o.ApplyRuntimeSettingsAtStartup(nil) // must not panic
})
})

96
core/config/runtime_settings_startup.go Normal file
View File

@@ -0,0 +1,96 @@
package config
import (
"encoding/json"
"github.com/mudler/LocalAI/pkg/vrambudget"
"github.com/mudler/LocalAI/pkg/xsysinfo"
)
// DefaultGalleriesJSON / DefaultBackendGalleriesJSON are the gallery lists
// an option-less `local-ai run` gets. cmd/local-ai feeds them to kong as
// the "${galleries}" / "${backends}" vars, and DefaultRuntimeBaseline uses
// them to tell "kong default" apart from "user-configured" at settings-load
// time - they must stay the single source for both.
const DefaultGalleriesJSON = `[{"name":"localai", "url":"github:mudler/LocalAI/gallery/index.yaml@master"}]`
const DefaultBackendGalleriesJSON = `[{"name":"localai", "url":"github:mudler/LocalAI/backend/index.yaml@master"}]`
func mustGalleries(jsonList string) []Gallery {
var g []Gallery
if err := json.Unmarshal([]byte(jsonList), &g); err != nil {
// Both inputs are compile-time constants above; failing loudly at
// init beats silently mis-detecting every gallery as env-set.
panic("invalid built-in gallery JSON: " + err.Error())
}
return g
}
// DefaultRuntimeBaseline is the ApplicationConfig an option-less `local-ai
// run` produces: NewApplicationConfig plus the flag defaults kong injects
// even when the user passes nothing (see the default:"..." tags in
// core/cli/run.go). ApplyRuntimeSettingsAtStartup compares the live config
// against it to decide whether env/CLI claimed a field (env > file).
func DefaultRuntimeBaseline() *ApplicationConfig {
o := NewApplicationConfig()
// WithDebug(log level) is always applied by run.go; the default level
// is not debug, so an option-less run boots with Debug=false even
// though the NewApplicationConfig literal says true.
o.Debug = false
o.Galleries = mustGalleries(DefaultGalleriesJSON)
o.BackendGalleries = mustGalleries(DefaultBackendGalleriesJSON)
o.AutoloadGalleries = true
o.AutoloadBackendGalleries = true
// core/cli/run.go injects WithMemoryReclaimer(enabled, threshold)
// unconditionally, so the kong threshold default (0.95) reaches the
// config even when the reclaimer flag is off - this overlay must match
// that contract or a UI-persisted threshold looks env-set and is
// skipped at boot.
o.MemoryReclaimerThreshold = 0.95
return o
}
// ApplyRuntimeSettingsAtStartup merges persisted settings into o with
// env-over-file precedence: a field is taken from the file only when the
// live value still equals the option-less baseline (env/CLI did not touch
// it) or when the file is that field's only source (fileAuthoritative).
// Used at boot (application.New) and by the runtime_settings.json file
// watcher, so a manual file edit behaves exactly like a boot-time load.
//
// Known limitation (accepted): an env var explicitly set to its default
// value is indistinguishable from "not set", so the file wins there; and a
// field previously changed via the API looks env-set to the watcher, so a
// manual file edit of that field lands on the next restart.
func (o *ApplicationConfig) ApplyRuntimeSettingsAtStartup(settings *RuntimeSettings) {
if settings == nil {
return
}
baseline := DefaultRuntimeBaseline()
for _, f := range runtimeSettingsFields {
if f.snapshotOnly || !f.isSet(settings) {
continue
}
if !f.fileAuthoritative && f.envSet(o, baseline) {
continue
}
f.apply(o, settings)
}
// Startup invariant, mirroring ApplyRuntimeSettings and the gating in
// startWatchdog: enabled idle/busy checks or the memory reclaimer imply
// the watchdog master flag. Never forced off here - an explicit
// watchdog_enabled=false row already applied above when unclaimed.
if o.WatchDogIdle || o.WatchDogBusy || o.MemoryReclaimerEnabled {
o.WatchDog = true
}
// VRAM budget post-processing, mirroring ApplyRuntimeSettings: at boot
// only run.go's env path installs the process-wide cap, so a
// file-persisted budget applied by the loop above would set
// o.VRAMBudget without ever capping allocations. When env set the
// budget the loop skipped the file value and this re-installs the env
// value - idempotent. Fail-open on a malformed persisted value so it
// cannot wedge startup.
if settings.VRAMBudget != nil {
if b, err := vrambudget.Parse(o.VRAMBudget); err == nil {
xsysinfo.SetDefaultVRAMBudget(b)
}
}
}

4
core/http/endpoints/localai/settings.go
View File

@@ -188,9 +188,7 @@ func UpdateSettingsEndpoint(app *application.Application) echo.HandlerFunc {
// Handle API keys specially (merge with startup keys)
if settings.ApiKeys != nil {
envKeys := startupConfig.ApiKeys
runtimeKeys := *settings.ApiKeys
appConfig.ApiKeys = append(envKeys, runtimeKeys...)
appConfig.ApiKeys = config.MergeAPIKeys(startupConfig.ApiKeys, *settings.ApiKeys)
}
// Update backend logging dynamically

21
docs/content/features/runtime-settings.md
View File

@@ -50,6 +50,7 @@ You can configure these settings via the web UI or through environment variables
- **Threads**: Number of threads used for parallel computation (recommended: number of physical cores)
- **Context Size**: Default context size for models (default: `512`)
- **F16**: Enable GPU acceleration using 16-bit floating point
- **VRAM Budget**: Cap on VRAM used for model allocation (for example `80%` or `12GB`; empty means no cap). See [VRAM Management]({{%relref "advanced/vram-management" %}})
### Debug and Logging
@@ -101,15 +102,25 @@ Configure the built-in agent platform (see [Agents]({{%relref "features/agents"
All settings are automatically saved to `runtime_settings.json` in the `LOCALAI_CONFIG_DIR` directory (default: `BASEPATH/configuration`). This file is watched for changes, so modifications made directly to the file will also be applied at runtime.
## Environment Variable Precedence
## Settings Precedence
Environment variables take precedence over settings configured via the web UI or configuration files. If a setting is controlled by an environment variable, it cannot be modified through the web interface. The settings page will indicate when a setting is controlled by an environment variable.
Every runtime setting follows a single precedence rule:
The precedence order is:
1. **Environment variables** (highest priority)
1. **Environment variables and CLI flags** (highest priority)
2. **Configuration files** (`runtime_settings.json`, `api_keys.json`)
3. **Default values** (lowest priority)
The same rule is applied identically in all three places where settings can change:
- **At boot**: values persisted in `runtime_settings.json` fill in every setting that was not explicitly set via an environment variable or CLI flag.
- **On `POST /api/settings`** (the web UI Settings page): if a setting is controlled by an environment variable, it cannot be modified through the web interface. The settings page will indicate when a setting is controlled by an environment variable.
- **On manual file edits**: the configuration watcher hot-applies edits to `runtime_settings.json` with the same env-over-file semantics as boot, so editing the file by hand behaves like restarting with that file. For example, hand-editing `vram_budget` installs the new VRAM allocation cap live, without a restart.
### Known Limitations
- An environment variable explicitly set to its default value is indistinguishable from an unset one, so the value from `runtime_settings.json` wins in that case. In particular, an explicit `LOCALAI_THREADS=0` behaves like an unset `LOCALAI_THREADS`.
- A field previously changed via the API (or the web UI) looks environment-set to the file watcher, so a manual file edit of that same field is not hot-applied: it takes effect on the next restart.
## Example Configuration
The `runtime_settings.json` file follows this structure:
@@ -171,7 +182,7 @@ The runtime settings system supports dynamic configuration file watching. When `
- `api_keys.json` - API keys (for backward compatibility)
- `external_backends.json` - External backend configurations
Changes to these files are automatically detected and applied without requiring a restart.
Changes to these files are automatically detected and applied without requiring a restart, using the same precedence rule described in [Settings Precedence](#settings-precedence).
## Best Practices