mirror of
https://github.com/mudler/LocalAI.git
synced 2026-05-30 03:25:42 -04:00
* feat(distributed): add per-request node ID context holder Introduce pkg/distributedhdr, a leaf package carrying a per-request *atomic.Value holder for the picked worker node ID from the SmartRouter (core/services/nodes) up to the HTTP response writer wrapper (core/http/middleware). Avoids the import cycle that a shared key in either consumer would create. Exposes NewHolder, WithHolder, Holder, Stamp, Load, Inherit. The holder is atomic.Value so cross-goroutine publish from the router to the response writer wrapper is race-clean. Assisted-by: Claude:claude-opus-4-7[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): add ExposeNodeHeader middleware + response writer wrapper New ApplicationConfig.ExposeNodeHeader bool + --expose-node-header CLI flag / LOCALAI_EXPOSE_NODE_HEADER env var (default off; the node ID reveals internal topology and is opt-in). The middleware creates a per-request *atomic.Value holder, attaches it to c.Request().Context() via distributedhdr.WithHolder, and wraps c.Response().Writer with a custom http.ResponseWriter that sets the X-LocalAI-Node header on first Write / WriteHeader / Flush by reading the holder. Implements http.Flusher, http.Hijacker, Unwrap so it composes cleanly with Echo and http.NewResponseController. request.go propagates the holder onto derived contexts via distributedhdr.Inherit so the holder survives the correlation-ID context replacement. Unit + race-clean concurrency + integration specs. Assisted-by: Claude:claude-opus-4-7[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): stamp node ID in router and wire middleware to inference routes ModelRouterAdapter.Route stamps the picked node ID into the per-request holder via distributedhdr.Stamp(ctx, result.Node.ID) right after replica selection. Wire ExposeNodeHeader middleware to: - OpenAI chat/completion/embeddings + audio transcriptions/speech + image generations/inpainting - Anthropic /v1/messages - Ollama /api/chat, /api/generate, /api/embed, /api/embeddings - Jina /v1/rerank - LocalAI /v1/vad The middleware's wrapper reads the holder on first byte and sets the X-LocalAI-Node response header before delegating to the underlying writer. Per-request scope means no race under concurrent multi-replica routing. Assisted-by: Claude:claude-opus-4-7[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): thread request context through backend Load + cover ctx propagation Five non-OpenAI backend helpers were silently using app.Context instead of the request context for the gRPC backend call: transcription, TTS, image generation, rerank, VAD. Effect: distributedhdr.Stamp in the router callback was a silent no-op for these paths, AND client cancellation didn't propagate to in-flight inference. Thread c.Request().Context() (or the equivalent input.Context after the request middleware has installed the correlation-ID derived context) through each helper and into ModelOptions via model.WithContext(ctx). ImageGeneration's signature gains a leading ctx parameter; in-tree callers (openai image, openai inpainting, openai inpainting_test) are updated to match. ModelEmbedding gains a leading ctx parameter for the same reason; the openai and ollama embedding handlers pass the request context through. chat_stream_workers.go defers the initial role=assistant chunk emission until the first token callback so the wrapper's lazy X-LocalAI-Node lookup against the loader runs AFTER ml.Load has stamped the per-modelID node ID; semantically identical for clients (role still arrives before any text). Regression test core/backend/ctx_propagation_test.go pins ctx propagation for all five helpers. Docs updated to enumerate the full endpoint coverage of the --expose-node-header flag. Assisted-by: Claude:claude-opus-4-7[1m] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
1404 lines
42 KiB
Go
1404 lines
42 KiB
Go
package config
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"regexp"
|
|
"time"
|
|
|
|
"github.com/mudler/LocalAI/pkg/system"
|
|
"github.com/mudler/LocalAI/pkg/xsysinfo"
|
|
"github.com/mudler/xlog"
|
|
)
|
|
|
|
type ApplicationConfig struct {
|
|
Context context.Context
|
|
ConfigFile string
|
|
SystemState *system.SystemState
|
|
ExternalBackends []string
|
|
UploadLimitMB, Threads, ContextSize int
|
|
F16 bool
|
|
Debug bool
|
|
EnableTracing bool
|
|
TracingMaxItems int
|
|
TracingMaxBodyBytes int // Per-body cap for captured request/response bodies; 0 disables the cap
|
|
EnableBackendLogging bool
|
|
GeneratedContentDir string
|
|
|
|
UploadDir string
|
|
DataPath string // Persistent data directory for collectiondb, agents, etc.
|
|
|
|
DynamicConfigsDir string
|
|
DynamicConfigsDirPollInterval time.Duration
|
|
CORS bool
|
|
DisableCSRF bool
|
|
PreloadJSONModels string
|
|
PreloadModelsFromPath string
|
|
CORSAllowOrigins string
|
|
ApiKeys []string
|
|
P2PToken string
|
|
P2PNetworkID string
|
|
Federated bool
|
|
|
|
// DisableStats turns off per-request token tracking. By default the
|
|
// routing module's billing recorder runs in every mode (including
|
|
// no-auth single-user) so dashboards and `/api/usage` are immediately
|
|
// useful; set this to opt out of that, e.g., for ephemeral CI runs
|
|
// or privacy-strict deployments where no token-count history should
|
|
// touch disk or memory.
|
|
DisableStats bool
|
|
|
|
// PIIConfigPath points to an optional YAML file describing the PII
|
|
// pattern set. When empty, the routing/pii module's DefaultPatterns()
|
|
// (email, phone, SSN, credit card, IPv4, API key prefixes) are
|
|
// loaded with their default actions. Each entry overrides the
|
|
// matching default by ID:
|
|
//
|
|
// patterns:
|
|
// - id: email
|
|
// action: route_local # downgrade default mask -> route_local
|
|
// - id: ssn
|
|
// action: block # upgrade default mask -> block
|
|
//
|
|
// Unknown ids are rejected with a clear error at startup.
|
|
PIIConfigPath string
|
|
|
|
// DisablePII turns the regex PII filter off entirely. Default
|
|
// (false) enables it on the OpenAI chat completions route.
|
|
DisablePII bool
|
|
|
|
// MITMListen is the address (host:port) the cloudproxy MITM
|
|
// listener binds on. Empty disables the MITM proxy entirely.
|
|
// Use case: redacting PII from Claude Code / Codex CLI traffic
|
|
// without LocalAI holding the upstream API key. Clients set
|
|
// HTTPS_PROXY=http://localai:port and trust the CA cert
|
|
// LocalAI exposes at /api/middleware/proxy-ca.crt.
|
|
MITMListen string
|
|
|
|
// MITMCADir holds the persisted MITM proxy CA cert and private
|
|
// key. The CA is generated on first start; subsequent starts
|
|
// reload it so clients keep trusting the same root. The key
|
|
// file is mode 0600.
|
|
MITMCADir string
|
|
|
|
|
|
// PIIPatternOverrides applies persisted per-id deltas (action,
|
|
// disabled) to the live redactor at startup. Loaded from
|
|
// runtime_settings.json and applied right after pii.NewRedactor.
|
|
// nil/empty leaves the YAML defaults in place.
|
|
PIIPatternOverrides map[string]PIIPatternRuntimeOverride
|
|
|
|
DisableWebUI bool
|
|
OllamaAPIRootEndpoint bool
|
|
EnforcePredownloadScans bool
|
|
OpaqueErrors bool
|
|
UseSubtleKeyComparison bool
|
|
DisableApiKeyRequirementForHttpGet bool
|
|
DisableMetrics bool
|
|
HttpGetExemptedEndpoints []*regexp.Regexp
|
|
DisableGalleryEndpoint bool
|
|
DisableMCP bool
|
|
LoadToMemory []string
|
|
|
|
Galleries []Gallery
|
|
BackendGalleries []Gallery
|
|
|
|
ExternalGRPCBackends map[string]string
|
|
|
|
AutoloadGalleries, AutoloadBackendGalleries bool
|
|
AutoUpgradeBackends bool
|
|
PreferDevelopmentBackends bool
|
|
|
|
// RequireBackendIntegrity promotes a missing SHA256 (tarball/HTTP URIs)
|
|
// or missing verification policy (OCI URIs) from a warning to a hard
|
|
// failure during backend install/upgrade. Off by default to keep
|
|
// upgrades non-breaking; operators opt in explicitly via
|
|
// --require-backend-integrity / LOCALAI_REQUIRE_BACKEND_INTEGRITY.
|
|
RequireBackendIntegrity bool
|
|
|
|
SingleBackend bool // Deprecated: use MaxActiveBackends = 1 instead
|
|
MaxActiveBackends int // Maximum number of active backends (0 = unlimited, 1 = single backend mode)
|
|
WatchDogIdle bool
|
|
WatchDogBusy bool
|
|
WatchDog bool
|
|
|
|
// Memory Reclaimer settings (works with GPU if available, otherwise RAM)
|
|
MemoryReclaimerEnabled bool // Enable memory threshold monitoring
|
|
MemoryReclaimerThreshold float64 // Threshold 0.0-1.0 (e.g., 0.95 = 95%)
|
|
|
|
// Eviction settings
|
|
ForceEvictionWhenBusy bool // Force eviction even when models have active API calls (default: false for safety)
|
|
LRUEvictionMaxRetries int // Maximum number of retries when waiting for busy models to become idle (default: 30)
|
|
LRUEvictionRetryInterval time.Duration // Interval between retries when waiting for busy models (default: 1s)
|
|
|
|
ModelsURL []string
|
|
|
|
WatchDogBusyTimeout, WatchDogIdleTimeout time.Duration
|
|
WatchDogInterval time.Duration // Interval between watchdog checks
|
|
|
|
MachineTag string
|
|
|
|
APIAddress string
|
|
|
|
LlamaCPPTunnelCallback func(tunnels []string)
|
|
MLXTunnelCallback func(tunnels []string)
|
|
|
|
DisableRuntimeSettings bool
|
|
|
|
AgentJobRetentionDays int // Default: 30 days
|
|
|
|
OpenResponsesStoreTTL time.Duration // TTL for Open Responses store (0 = no expiration)
|
|
|
|
PathWithoutAuth []string
|
|
|
|
// Agent Pool (LocalAGI integration)
|
|
AgentPool AgentPoolConfig
|
|
|
|
// Authentication & Authorization
|
|
Auth AuthConfig
|
|
|
|
// Distributed / Horizontal Scaling
|
|
Distributed DistributedConfig
|
|
|
|
// ExposeNodeHeader, when true, activates middleware.ExposeNodeHeader on
|
|
// the inference routes (OpenAI chat/completions/embeddings, Anthropic
|
|
// /v1/messages, Ollama /api/chat,/api/generate,/api/embed). The
|
|
// middleware wraps the response writer and attaches an "X-LocalAI-Node"
|
|
// response header carrying the ID of the distributed-mode worker node
|
|
// that served the request. Off by default because the node ID is
|
|
// internal topology that can aid attacker reconnaissance if surfaced on
|
|
// a public endpoint; operators opt in explicitly via
|
|
// --expose-node-header / LOCALAI_EXPOSE_NODE_HEADER for debugging,
|
|
// observability and load-balancer attribution.
|
|
ExposeNodeHeader bool
|
|
|
|
// LocalAI Assistant chat modality. Hard-disable the in-process admin MCP
|
|
// server with this flag; runtime-toggleable via /api/settings.
|
|
DisableLocalAIAssistant bool
|
|
|
|
// Branding / whitelabeling — runtime-mutable via /api/settings (text) and
|
|
// /api/branding/asset/:kind (binary uploads). All values optional; empty
|
|
// strings fall back to bundled LocalAI defaults.
|
|
Branding BrandingConfig
|
|
}
|
|
|
|
// BrandingConfig holds the whitelabel/branding configuration of the instance.
|
|
// Text fields are exposed via the public GET /api/branding endpoint so the
|
|
// login page can read them before authentication. Binary asset filenames
|
|
// (logo, horizontal logo, favicon) are stored as basenames; the actual files
|
|
// live under {DynamicConfigsDir}/branding/.
|
|
type BrandingConfig struct {
|
|
InstanceName string
|
|
InstanceTagline string
|
|
LogoFile string
|
|
LogoHorizontalFile string
|
|
FaviconFile string
|
|
}
|
|
|
|
// AuthConfig holds configuration for user authentication and authorization.
|
|
type AuthConfig struct {
|
|
Enabled bool
|
|
DatabaseURL string // "postgres://..." or file path for SQLite
|
|
GitHubClientID string
|
|
GitHubClientSecret string
|
|
OIDCIssuer string // OIDC issuer URL for auto-discovery (e.g. https://accounts.google.com)
|
|
OIDCClientID string
|
|
OIDCClientSecret string
|
|
BaseURL string // for OAuth callback URLs (e.g. "http://localhost:8080")
|
|
AdminEmail string // auto-promote to admin on login
|
|
RegistrationMode string // "open", "approval" (default when empty), "invite"
|
|
DisableLocalAuth bool // disable local email/password registration and login
|
|
APIKeyHMACSecret string // HMAC secret for API key hashing; auto-generated if empty
|
|
DefaultAPIKeyExpiry string // default expiry duration for API keys (e.g. "90d"); empty = no expiry
|
|
}
|
|
|
|
// AgentPoolConfig holds configuration for the LocalAGI agent pool integration.
|
|
type AgentPoolConfig struct {
|
|
Enabled bool // default: true (disabled by LOCALAI_DISABLE_AGENTS=true)
|
|
StateDir string // default: DynamicConfigsDir (LocalAI configuration folder)
|
|
APIURL string // default: self-referencing LocalAI (http://127.0.0.1:<port>)
|
|
APIKey string // default: first API key from LocalAI config
|
|
DefaultModel string
|
|
MultimodalModel string
|
|
TranscriptionModel string
|
|
TranscriptionLanguage string
|
|
TTSModel string
|
|
Timeout string // default: "5m"
|
|
EnableSkills bool
|
|
EnableLogs bool
|
|
CustomActionsDir string
|
|
CollectionDBPath string
|
|
VectorEngine string // default: "chromem"
|
|
EmbeddingModel string // default: "granite-embedding-107m-multilingual"
|
|
MaxChunkingSize int // default: 400
|
|
ChunkOverlap int // default: 0
|
|
DatabaseURL string
|
|
AgentHubURL string // default: "https://agenthub.localai.io"
|
|
}
|
|
|
|
type AppOption func(*ApplicationConfig)
|
|
|
|
func NewApplicationConfig(o ...AppOption) *ApplicationConfig {
|
|
opt := &ApplicationConfig{
|
|
Context: context.Background(),
|
|
UploadLimitMB: 15,
|
|
Debug: true,
|
|
AgentJobRetentionDays: 30, // Default: 30 days
|
|
LRUEvictionMaxRetries: 30, // Default: 30 retries
|
|
LRUEvictionRetryInterval: 1 * time.Second, // Default: 1 second
|
|
WatchDogInterval: 500 * time.Millisecond, // Default: 500ms
|
|
TracingMaxItems: 1024,
|
|
TracingMaxBodyBytes: 64 * 1024, // 64 KiB - caps each request/response body in the trace buffer
|
|
AgentPool: AgentPoolConfig{
|
|
Enabled: true,
|
|
Timeout: "5m",
|
|
VectorEngine: "chromem",
|
|
EmbeddingModel: "granite-embedding-107m-multilingual",
|
|
MaxChunkingSize: 400,
|
|
AgentHubURL: "https://agenthub.localai.io",
|
|
},
|
|
PathWithoutAuth: []string{
|
|
"/static/",
|
|
"/generated-audio/",
|
|
"/generated-images/",
|
|
"/generated-videos/",
|
|
"/favicon.svg",
|
|
"/readyz",
|
|
"/healthz",
|
|
"/api/auth/",
|
|
"/assets/",
|
|
// Branding read endpoint + public asset server. The login
|
|
// screen renders before authentication completes, so it has
|
|
// to be able to GET /api/branding and the configured logo.
|
|
//
|
|
// IMPORTANT: PathWithoutAuth uses a prefix match (see
|
|
// auth.isExemptPath). The "/api/branding" entry therefore
|
|
// also exempts POST/DELETE /api/branding/asset/:kind from
|
|
// the *global* auth middleware. Those routes are still
|
|
// admin-gated because they are registered with the
|
|
// route-level adminMiddleware (auth.RequireAdmin) in
|
|
// core/http/routes/ui_api.go — that's what keeps anonymous
|
|
// uploads/deletes returning 401. Any new admin-only sub-route
|
|
// added under /api/branding/* MUST also carry adminMiddleware
|
|
// at the route registration site, otherwise it ships
|
|
// unauthenticated. The TestBrandingRoutes_AdminGatingHolds
|
|
// integration test in core/http/auth pins this contract.
|
|
"/api/branding",
|
|
"/branding/",
|
|
},
|
|
}
|
|
for _, oo := range o {
|
|
oo(opt)
|
|
}
|
|
return opt
|
|
}
|
|
|
|
func WithModelsURL(urls ...string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.ModelsURL = urls
|
|
}
|
|
}
|
|
|
|
func WithSystemState(state *system.SystemState) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.SystemState = state
|
|
}
|
|
}
|
|
|
|
func WithExternalBackends(backends ...string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.ExternalBackends = backends
|
|
}
|
|
}
|
|
|
|
func WithMachineTag(tag string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.MachineTag = tag
|
|
}
|
|
}
|
|
|
|
func WithCors(b bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.CORS = b
|
|
}
|
|
}
|
|
|
|
func WithP2PNetworkID(s string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.P2PNetworkID = s
|
|
}
|
|
}
|
|
|
|
func WithDisableCSRF(b bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.DisableCSRF = b
|
|
}
|
|
}
|
|
|
|
func WithP2PToken(s string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.P2PToken = s
|
|
}
|
|
}
|
|
|
|
var EnableWatchDog = func(o *ApplicationConfig) {
|
|
o.WatchDog = true
|
|
}
|
|
|
|
var EnableTracing = func(o *ApplicationConfig) {
|
|
o.EnableTracing = true
|
|
}
|
|
|
|
var EnableBackendLogging = func(o *ApplicationConfig) {
|
|
o.EnableBackendLogging = true
|
|
}
|
|
|
|
var EnableWatchDogIdleCheck = func(o *ApplicationConfig) {
|
|
o.WatchDog = true
|
|
o.WatchDogIdle = true
|
|
}
|
|
|
|
var DisableGalleryEndpoint = func(o *ApplicationConfig) {
|
|
o.DisableGalleryEndpoint = true
|
|
}
|
|
|
|
var DisableMCP = func(o *ApplicationConfig) {
|
|
o.DisableMCP = true
|
|
}
|
|
|
|
var EnableWatchDogBusyCheck = func(o *ApplicationConfig) {
|
|
o.WatchDog = true
|
|
o.WatchDogBusy = true
|
|
}
|
|
|
|
var DisableWebUI = func(o *ApplicationConfig) {
|
|
o.DisableWebUI = true
|
|
}
|
|
|
|
var EnableOllamaAPIRootEndpoint = func(o *ApplicationConfig) {
|
|
o.OllamaAPIRootEndpoint = true
|
|
}
|
|
|
|
var DisableRuntimeSettings = func(o *ApplicationConfig) {
|
|
o.DisableRuntimeSettings = true
|
|
}
|
|
|
|
func SetWatchDogBusyTimeout(t time.Duration) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.WatchDogBusyTimeout = t
|
|
}
|
|
}
|
|
|
|
func SetWatchDogIdleTimeout(t time.Duration) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.WatchDogIdleTimeout = t
|
|
}
|
|
}
|
|
|
|
func SetWatchDogInterval(t time.Duration) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.WatchDogInterval = t
|
|
}
|
|
}
|
|
|
|
// EnableMemoryReclaimer enables memory threshold monitoring.
|
|
// When enabled, the watchdog will evict backends if memory usage exceeds the threshold.
|
|
// Works with GPU VRAM if available, otherwise uses system RAM.
|
|
var EnableMemoryReclaimer = func(o *ApplicationConfig) {
|
|
o.MemoryReclaimerEnabled = true
|
|
o.WatchDog = true // Memory reclaimer requires watchdog infrastructure
|
|
}
|
|
|
|
// SetMemoryReclaimerThreshold sets the memory usage threshold (0.0-1.0).
|
|
// When memory usage exceeds this threshold, backends will be evicted using LRU strategy.
|
|
func SetMemoryReclaimerThreshold(threshold float64) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
if threshold > 0 && threshold <= 1.0 {
|
|
o.MemoryReclaimerThreshold = threshold
|
|
o.MemoryReclaimerEnabled = true
|
|
o.WatchDog = true // Memory reclaimer requires watchdog infrastructure
|
|
}
|
|
}
|
|
}
|
|
|
|
// WithMemoryReclaimer configures the memory reclaimer with the given settings
|
|
func WithMemoryReclaimer(enabled bool, threshold float64) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.MemoryReclaimerEnabled = enabled
|
|
if threshold > 0 && threshold <= 1.0 {
|
|
o.MemoryReclaimerThreshold = threshold
|
|
}
|
|
if enabled {
|
|
o.WatchDog = true // Memory reclaimer requires watchdog infrastructure
|
|
}
|
|
}
|
|
}
|
|
|
|
// EnableSingleBackend is deprecated: use SetMaxActiveBackends(1) instead.
|
|
// This is kept for backward compatibility.
|
|
var EnableSingleBackend = func(o *ApplicationConfig) {
|
|
o.SingleBackend = true
|
|
o.MaxActiveBackends = 1
|
|
}
|
|
|
|
// SetMaxActiveBackends sets the maximum number of active backends.
|
|
// 0 = unlimited, 1 = single backend mode (replaces EnableSingleBackend)
|
|
func SetMaxActiveBackends(n int) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.MaxActiveBackends = n
|
|
// For backward compatibility, also set SingleBackend if n == 1
|
|
if n == 1 {
|
|
o.SingleBackend = true
|
|
}
|
|
}
|
|
}
|
|
|
|
// GetEffectiveMaxActiveBackends returns the effective max active backends limit.
|
|
// It considers both MaxActiveBackends and the deprecated SingleBackend setting.
|
|
// If MaxActiveBackends is set (> 0), it takes precedence.
|
|
// If SingleBackend is true and MaxActiveBackends is 0, returns 1.
|
|
// Otherwise returns 0 (unlimited).
|
|
func (o *ApplicationConfig) GetEffectiveMaxActiveBackends() int {
|
|
if o.MaxActiveBackends > 0 {
|
|
return o.MaxActiveBackends
|
|
}
|
|
if o.SingleBackend {
|
|
return 1
|
|
}
|
|
return 0
|
|
}
|
|
|
|
// WithForceEvictionWhenBusy sets whether to force eviction even when models have active API calls
|
|
func WithForceEvictionWhenBusy(enabled bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.ForceEvictionWhenBusy = enabled
|
|
}
|
|
}
|
|
|
|
// WithLRUEvictionMaxRetries sets the maximum number of retries when waiting for busy models to become idle
|
|
func WithLRUEvictionMaxRetries(maxRetries int) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
if maxRetries > 0 {
|
|
o.LRUEvictionMaxRetries = maxRetries
|
|
}
|
|
}
|
|
}
|
|
|
|
// WithLRUEvictionRetryInterval sets the interval between retries when waiting for busy models
|
|
func WithLRUEvictionRetryInterval(interval time.Duration) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
if interval > 0 {
|
|
o.LRUEvictionRetryInterval = interval
|
|
}
|
|
}
|
|
}
|
|
|
|
var EnableGalleriesAutoload = func(o *ApplicationConfig) {
|
|
o.AutoloadGalleries = true
|
|
}
|
|
|
|
var EnableBackendGalleriesAutoload = func(o *ApplicationConfig) {
|
|
o.AutoloadBackendGalleries = true
|
|
}
|
|
|
|
func WithAutoUpgradeBackends(v bool) AppOption {
|
|
return func(o *ApplicationConfig) { o.AutoUpgradeBackends = v }
|
|
}
|
|
|
|
func WithRequireBackendIntegrity(v bool) AppOption {
|
|
return func(o *ApplicationConfig) { o.RequireBackendIntegrity = v }
|
|
}
|
|
|
|
func WithPreferDevelopmentBackends(v bool) AppOption {
|
|
return func(o *ApplicationConfig) { o.PreferDevelopmentBackends = v }
|
|
}
|
|
|
|
var EnableFederated = func(o *ApplicationConfig) {
|
|
o.Federated = true
|
|
}
|
|
|
|
func WithExternalBackend(name string, uri string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
if o.ExternalGRPCBackends == nil {
|
|
o.ExternalGRPCBackends = make(map[string]string)
|
|
}
|
|
o.ExternalGRPCBackends[name] = uri
|
|
}
|
|
}
|
|
|
|
func WithCorsAllowOrigins(b string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.CORSAllowOrigins = b
|
|
}
|
|
}
|
|
|
|
func WithStringGalleries(galls string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
if galls == "" {
|
|
o.Galleries = []Gallery{}
|
|
return
|
|
}
|
|
var galleries []Gallery
|
|
if err := json.Unmarshal([]byte(galls), &galleries); err != nil {
|
|
xlog.Error("failed loading galleries", "error", err)
|
|
}
|
|
o.Galleries = append(o.Galleries, galleries...)
|
|
}
|
|
}
|
|
|
|
func WithBackendGalleries(galls string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
if galls == "" {
|
|
o.BackendGalleries = []Gallery{}
|
|
return
|
|
}
|
|
var galleries []Gallery
|
|
if err := json.Unmarshal([]byte(galls), &galleries); err != nil {
|
|
xlog.Error("failed loading galleries", "error", err)
|
|
}
|
|
o.BackendGalleries = append(o.BackendGalleries, galleries...)
|
|
}
|
|
}
|
|
|
|
func WithGalleries(galleries []Gallery) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Galleries = append(o.Galleries, galleries...)
|
|
}
|
|
}
|
|
|
|
func WithContext(ctx context.Context) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Context = ctx
|
|
}
|
|
}
|
|
|
|
func WithYAMLConfigPreload(configFile string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.PreloadModelsFromPath = configFile
|
|
}
|
|
}
|
|
|
|
func WithJSONStringPreload(configFile string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.PreloadJSONModels = configFile
|
|
}
|
|
}
|
|
func WithConfigFile(configFile string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.ConfigFile = configFile
|
|
}
|
|
}
|
|
|
|
func WithUploadLimitMB(limit int) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.UploadLimitMB = limit
|
|
}
|
|
}
|
|
|
|
func WithThreads(threads int) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
if threads == 0 { // 0 is not allowed
|
|
threads = xsysinfo.CPUPhysicalCores()
|
|
}
|
|
o.Threads = threads
|
|
}
|
|
}
|
|
|
|
func WithContextSize(ctxSize int) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.ContextSize = ctxSize
|
|
}
|
|
}
|
|
|
|
func WithLlamaCPPTunnelCallback(callback func(tunnels []string)) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.LlamaCPPTunnelCallback = callback
|
|
}
|
|
}
|
|
|
|
func WithMLXTunnelCallback(callback func(tunnels []string)) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.MLXTunnelCallback = callback
|
|
}
|
|
}
|
|
|
|
func WithF16(f16 bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.F16 = f16
|
|
}
|
|
}
|
|
|
|
func WithDebug(debug bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Debug = debug
|
|
}
|
|
}
|
|
|
|
func WithTracingMaxItems(items int) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.TracingMaxItems = items
|
|
}
|
|
}
|
|
|
|
func WithTracingMaxBodyBytes(bytes int) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.TracingMaxBodyBytes = bytes
|
|
}
|
|
}
|
|
|
|
func WithGeneratedContentDir(generatedContentDir string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.GeneratedContentDir = generatedContentDir
|
|
}
|
|
}
|
|
|
|
func WithUploadDir(uploadDir string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.UploadDir = uploadDir
|
|
}
|
|
}
|
|
|
|
func WithDataPath(dataPath string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.DataPath = dataPath
|
|
}
|
|
}
|
|
|
|
// WithDisableStats turns off the billing recorder. CLI: --disable-stats.
|
|
func WithDisableStats(disable bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.DisableStats = disable
|
|
}
|
|
}
|
|
|
|
// WithPIIConfigPath points the routing PII filter at a YAML config
|
|
// file. CLI: --pii-config.
|
|
func WithPIIConfigPath(path string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.PIIConfigPath = path
|
|
}
|
|
}
|
|
|
|
// WithDisablePII turns the regex PII filter off. CLI: --disable-pii.
|
|
func WithDisablePII(disable bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.DisablePII = disable
|
|
}
|
|
}
|
|
|
|
// WithMITMListen sets the address the cloudproxy MITM listener
|
|
// binds on. Empty = disabled. CLI: --mitm-listen.
|
|
func WithMITMListen(addr string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.MITMListen = addr
|
|
}
|
|
}
|
|
|
|
// WithMITMCADir sets the directory used to persist the MITM proxy
|
|
// CA cert + key. CLI: --mitm-ca-dir.
|
|
func WithMITMCADir(dir string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.MITMCADir = dir
|
|
}
|
|
}
|
|
|
|
|
|
func WithDynamicConfigDir(dynamicConfigsDir string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.DynamicConfigsDir = dynamicConfigsDir
|
|
}
|
|
}
|
|
|
|
func WithDynamicConfigDirPollInterval(interval time.Duration) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.DynamicConfigsDirPollInterval = interval
|
|
}
|
|
}
|
|
|
|
func WithApiKeys(apiKeys []string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.ApiKeys = apiKeys
|
|
}
|
|
}
|
|
|
|
func WithAgentJobRetentionDays(days int) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentJobRetentionDays = days
|
|
}
|
|
}
|
|
|
|
func WithOpenResponsesStoreTTL(ttl time.Duration) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.OpenResponsesStoreTTL = ttl
|
|
}
|
|
}
|
|
|
|
func WithEnforcedPredownloadScans(enforced bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.EnforcePredownloadScans = enforced
|
|
}
|
|
}
|
|
|
|
func WithOpaqueErrors(opaque bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.OpaqueErrors = opaque
|
|
}
|
|
}
|
|
|
|
func WithLoadToMemory(models []string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.LoadToMemory = models
|
|
}
|
|
}
|
|
|
|
func WithSubtleKeyComparison(subtle bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.UseSubtleKeyComparison = subtle
|
|
}
|
|
}
|
|
|
|
func WithDisableApiKeyRequirementForHttpGet(required bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.DisableApiKeyRequirementForHttpGet = required
|
|
}
|
|
}
|
|
|
|
func WithAPIAddress(address string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.APIAddress = address
|
|
}
|
|
}
|
|
|
|
var DisableMetricsEndpoint AppOption = func(o *ApplicationConfig) {
|
|
o.DisableMetrics = true
|
|
}
|
|
|
|
func WithHttpGetExemptedEndpoints(endpoints []string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.HttpGetExemptedEndpoints = []*regexp.Regexp{}
|
|
for _, epr := range endpoints {
|
|
r, err := regexp.Compile(epr)
|
|
if err == nil && r != nil {
|
|
o.HttpGetExemptedEndpoints = append(o.HttpGetExemptedEndpoints, r)
|
|
} else {
|
|
xlog.Warn("Error while compiling HTTP Get Exemption regex, skipping this entry.", "error", err, "regex", epr)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// Agent Pool options
|
|
|
|
var DisableAgentPool = func(o *ApplicationConfig) {
|
|
o.AgentPool.Enabled = false
|
|
}
|
|
|
|
func WithAgentPoolAPIURL(url string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.APIURL = url
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolAPIKey(key string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.APIKey = key
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolDefaultModel(model string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.DefaultModel = model
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolMultimodalModel(model string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.MultimodalModel = model
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolTranscriptionModel(model string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.TranscriptionModel = model
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolTranscriptionLanguage(lang string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.TranscriptionLanguage = lang
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolTTSModel(model string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.TTSModel = model
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolStateDir(dir string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.StateDir = dir
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolTimeout(timeout string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.Timeout = timeout
|
|
}
|
|
}
|
|
|
|
var EnableAgentPoolSkills = func(o *ApplicationConfig) {
|
|
o.AgentPool.EnableSkills = true
|
|
}
|
|
|
|
func WithAgentPoolVectorEngine(engine string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.VectorEngine = engine
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolEmbeddingModel(model string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.EmbeddingModel = model
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolCustomActionsDir(dir string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.CustomActionsDir = dir
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolDatabaseURL(url string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.DatabaseURL = url
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolMaxChunkingSize(size int) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.MaxChunkingSize = size
|
|
}
|
|
}
|
|
|
|
func WithAgentPoolChunkOverlap(overlap int) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.ChunkOverlap = overlap
|
|
}
|
|
}
|
|
|
|
var EnableAgentPoolLogs = func(o *ApplicationConfig) {
|
|
o.AgentPool.EnableLogs = true
|
|
}
|
|
|
|
func WithAgentPoolCollectionDBPath(path string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.CollectionDBPath = path
|
|
}
|
|
}
|
|
|
|
func WithAgentHubURL(url string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.AgentPool.AgentHubURL = url
|
|
}
|
|
}
|
|
|
|
// Auth options
|
|
|
|
func WithAuthEnabled(enabled bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.Enabled = enabled
|
|
}
|
|
}
|
|
|
|
func WithAuthDatabaseURL(url string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.DatabaseURL = url
|
|
}
|
|
}
|
|
|
|
func WithAuthGitHubClientID(clientID string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.GitHubClientID = clientID
|
|
}
|
|
}
|
|
|
|
func WithAuthGitHubClientSecret(clientSecret string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.GitHubClientSecret = clientSecret
|
|
}
|
|
}
|
|
|
|
func WithAuthBaseURL(baseURL string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.BaseURL = baseURL
|
|
}
|
|
}
|
|
|
|
func WithAuthAdminEmail(email string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.AdminEmail = email
|
|
}
|
|
}
|
|
|
|
func WithAuthRegistrationMode(mode string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.RegistrationMode = mode
|
|
}
|
|
}
|
|
|
|
func WithAuthDisableLocalAuth(disable bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.DisableLocalAuth = disable
|
|
}
|
|
}
|
|
|
|
func WithAuthOIDCIssuer(issuer string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.OIDCIssuer = issuer
|
|
}
|
|
}
|
|
|
|
func WithAuthOIDCClientID(clientID string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.OIDCClientID = clientID
|
|
}
|
|
}
|
|
|
|
func WithAuthOIDCClientSecret(clientSecret string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.OIDCClientSecret = clientSecret
|
|
}
|
|
}
|
|
|
|
func WithAuthAPIKeyHMACSecret(secret string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.APIKeyHMACSecret = secret
|
|
}
|
|
}
|
|
|
|
func WithAuthDefaultAPIKeyExpiry(expiry string) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.Auth.DefaultAPIKeyExpiry = expiry
|
|
}
|
|
}
|
|
|
|
// WithDisableLocalAIAssistant hard-disables the in-process admin MCP server.
|
|
// When set, the chat-handler branch for metadata.localai_assistant=true
|
|
// returns a "feature unavailable" error.
|
|
func WithDisableLocalAIAssistant(disabled bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.DisableLocalAIAssistant = disabled
|
|
}
|
|
}
|
|
|
|
// WithExposeNodeHeader enables the X-LocalAI-Node response header on
|
|
// inference endpoints. Default off; the node ID reveals internal cluster
|
|
// topology and is opt-in for that reason.
|
|
func WithExposeNodeHeader(enabled bool) AppOption {
|
|
return func(o *ApplicationConfig) {
|
|
o.ExposeNodeHeader = enabled
|
|
}
|
|
}
|
|
|
|
// ToConfigLoaderOptions returns a slice of ConfigLoader Option.
|
|
// Some options defined at the application level are going to be passed as defaults for
|
|
// all the configuration for the models.
|
|
// This includes for instance the context size or the number of threads.
|
|
// If a model doesn't set configs directly to the config model file
|
|
// it will use the defaults defined here.
|
|
func (o *ApplicationConfig) ToConfigLoaderOptions() []ConfigLoaderOption {
|
|
return []ConfigLoaderOption{
|
|
LoadOptionContextSize(o.ContextSize),
|
|
LoadOptionDebug(o.Debug),
|
|
LoadOptionF16(o.F16),
|
|
LoadOptionThreads(o.Threads),
|
|
ModelPath(o.SystemState.Model.ModelsPath),
|
|
}
|
|
}
|
|
|
|
// ToRuntimeSettings converts ApplicationConfig to RuntimeSettings for API responses and JSON serialization.
|
|
// This provides a single source of truth - ApplicationConfig holds the live values,
|
|
// and this method creates a RuntimeSettings snapshot for external consumption.
|
|
func (o *ApplicationConfig) ToRuntimeSettings() RuntimeSettings {
|
|
// Create local copies for pointer fields
|
|
watchdogEnabled := o.WatchDog
|
|
watchdogIdle := o.WatchDogIdle
|
|
watchdogBusy := o.WatchDogBusy
|
|
singleBackend := o.SingleBackend
|
|
maxActiveBackends := o.MaxActiveBackends
|
|
memoryReclaimerEnabled := o.MemoryReclaimerEnabled
|
|
memoryReclaimerThreshold := o.MemoryReclaimerThreshold
|
|
forceEvictionWhenBusy := o.ForceEvictionWhenBusy
|
|
lruEvictionMaxRetries := o.LRUEvictionMaxRetries
|
|
threads := o.Threads
|
|
contextSize := o.ContextSize
|
|
f16 := o.F16
|
|
debug := o.Debug
|
|
tracingMaxItems := o.TracingMaxItems
|
|
tracingMaxBodyBytes := o.TracingMaxBodyBytes
|
|
enableTracing := o.EnableTracing
|
|
enableBackendLogging := o.EnableBackendLogging
|
|
cors := o.CORS
|
|
csrf := o.DisableCSRF
|
|
corsAllowOrigins := o.CORSAllowOrigins
|
|
p2pToken := o.P2PToken
|
|
p2pNetworkID := o.P2PNetworkID
|
|
federated := o.Federated
|
|
galleries := o.Galleries
|
|
backendGalleries := o.BackendGalleries
|
|
autoloadGalleries := o.AutoloadGalleries
|
|
autoloadBackendGalleries := o.AutoloadBackendGalleries
|
|
autoUpgradeBackends := o.AutoUpgradeBackends
|
|
preferDevelopmentBackends := o.PreferDevelopmentBackends
|
|
apiKeys := o.ApiKeys
|
|
agentJobRetentionDays := o.AgentJobRetentionDays
|
|
|
|
// Format timeouts as strings
|
|
var idleTimeout, busyTimeout, watchdogInterval string
|
|
if o.WatchDogIdleTimeout > 0 {
|
|
idleTimeout = o.WatchDogIdleTimeout.String()
|
|
} else {
|
|
idleTimeout = "15m" // default
|
|
}
|
|
if o.WatchDogBusyTimeout > 0 {
|
|
busyTimeout = o.WatchDogBusyTimeout.String()
|
|
} else {
|
|
busyTimeout = "5m" // default
|
|
}
|
|
if o.WatchDogInterval > 0 {
|
|
watchdogInterval = o.WatchDogInterval.String()
|
|
} else {
|
|
watchdogInterval = "2s" // default
|
|
}
|
|
var lruEvictionRetryInterval string
|
|
if o.LRUEvictionRetryInterval > 0 {
|
|
lruEvictionRetryInterval = o.LRUEvictionRetryInterval.String()
|
|
} else {
|
|
lruEvictionRetryInterval = "1s" // default
|
|
}
|
|
var openResponsesStoreTTL string
|
|
if o.OpenResponsesStoreTTL > 0 {
|
|
openResponsesStoreTTL = o.OpenResponsesStoreTTL.String()
|
|
} else {
|
|
openResponsesStoreTTL = "0" // default: no expiration
|
|
}
|
|
|
|
// Agent Pool settings
|
|
agentPoolEnabled := o.AgentPool.Enabled
|
|
agentPoolDefaultModel := o.AgentPool.DefaultModel
|
|
agentPoolEmbeddingModel := o.AgentPool.EmbeddingModel
|
|
agentPoolMaxChunkingSize := o.AgentPool.MaxChunkingSize
|
|
agentPoolChunkOverlap := o.AgentPool.ChunkOverlap
|
|
agentPoolEnableLogs := o.AgentPool.EnableLogs
|
|
agentPoolCollectionDBPath := o.AgentPool.CollectionDBPath
|
|
agentPoolVectorEngine := o.AgentPool.VectorEngine
|
|
agentPoolDatabaseURL := o.AgentPool.DatabaseURL
|
|
agentPoolAgentHubURL := o.AgentPool.AgentHubURL
|
|
|
|
// LocalAI Assistant settings
|
|
localAIAssistantEnabled := !o.DisableLocalAIAssistant
|
|
|
|
// Branding settings
|
|
instanceName := o.Branding.InstanceName
|
|
instanceTagline := o.Branding.InstanceTagline
|
|
logoFile := o.Branding.LogoFile
|
|
logoHorizontalFile := o.Branding.LogoHorizontalFile
|
|
faviconFile := o.Branding.FaviconFile
|
|
|
|
mitmListen := o.MITMListen
|
|
|
|
return RuntimeSettings{
|
|
WatchdogEnabled: &watchdogEnabled,
|
|
WatchdogIdleEnabled: &watchdogIdle,
|
|
WatchdogBusyEnabled: &watchdogBusy,
|
|
WatchdogIdleTimeout: &idleTimeout,
|
|
WatchdogBusyTimeout: &busyTimeout,
|
|
WatchdogInterval: &watchdogInterval,
|
|
SingleBackend: &singleBackend,
|
|
MaxActiveBackends: &maxActiveBackends,
|
|
MemoryReclaimerEnabled: &memoryReclaimerEnabled,
|
|
MemoryReclaimerThreshold: &memoryReclaimerThreshold,
|
|
ForceEvictionWhenBusy: &forceEvictionWhenBusy,
|
|
LRUEvictionMaxRetries: &lruEvictionMaxRetries,
|
|
LRUEvictionRetryInterval: &lruEvictionRetryInterval,
|
|
Threads: &threads,
|
|
ContextSize: &contextSize,
|
|
F16: &f16,
|
|
Debug: &debug,
|
|
TracingMaxItems: &tracingMaxItems,
|
|
TracingMaxBodyBytes: &tracingMaxBodyBytes,
|
|
EnableTracing: &enableTracing,
|
|
EnableBackendLogging: &enableBackendLogging,
|
|
CORS: &cors,
|
|
CSRF: &csrf,
|
|
CORSAllowOrigins: &corsAllowOrigins,
|
|
P2PToken: &p2pToken,
|
|
P2PNetworkID: &p2pNetworkID,
|
|
Federated: &federated,
|
|
Galleries: &galleries,
|
|
BackendGalleries: &backendGalleries,
|
|
AutoloadGalleries: &autoloadGalleries,
|
|
AutoloadBackendGalleries: &autoloadBackendGalleries,
|
|
AutoUpgradeBackends: &autoUpgradeBackends,
|
|
PreferDevelopmentBackends: &preferDevelopmentBackends,
|
|
ApiKeys: &apiKeys,
|
|
AgentJobRetentionDays: &agentJobRetentionDays,
|
|
OpenResponsesStoreTTL: &openResponsesStoreTTL,
|
|
AgentPoolEnabled: &agentPoolEnabled,
|
|
AgentPoolDefaultModel: &agentPoolDefaultModel,
|
|
AgentPoolEmbeddingModel: &agentPoolEmbeddingModel,
|
|
AgentPoolMaxChunkingSize: &agentPoolMaxChunkingSize,
|
|
AgentPoolChunkOverlap: &agentPoolChunkOverlap,
|
|
AgentPoolEnableLogs: &agentPoolEnableLogs,
|
|
AgentPoolCollectionDBPath: &agentPoolCollectionDBPath,
|
|
AgentPoolVectorEngine: &agentPoolVectorEngine,
|
|
AgentPoolDatabaseURL: &agentPoolDatabaseURL,
|
|
AgentPoolAgentHubURL: &agentPoolAgentHubURL,
|
|
LocalAIAssistantEnabled: &localAIAssistantEnabled,
|
|
InstanceName: &instanceName,
|
|
InstanceTagline: &instanceTagline,
|
|
LogoFile: &logoFile,
|
|
LogoHorizontalFile: &logoHorizontalFile,
|
|
FaviconFile: &faviconFile,
|
|
MITMListen: &mitmListen,
|
|
}
|
|
}
|
|
|
|
// ApplyRuntimeSettings applies RuntimeSettings to ApplicationConfig.
|
|
// Only non-nil fields in RuntimeSettings are applied.
|
|
// Returns true if watchdog-related settings changed (requiring restart).
|
|
func (o *ApplicationConfig) ApplyRuntimeSettings(settings *RuntimeSettings) (requireRestart bool) {
|
|
if settings == nil {
|
|
return false
|
|
}
|
|
|
|
if settings.WatchdogEnabled != nil {
|
|
o.WatchDog = *settings.WatchdogEnabled
|
|
requireRestart = true
|
|
}
|
|
if settings.WatchdogIdleEnabled != nil {
|
|
o.WatchDogIdle = *settings.WatchdogIdleEnabled
|
|
if o.WatchDogIdle {
|
|
o.WatchDog = true
|
|
}
|
|
requireRestart = true
|
|
}
|
|
if settings.WatchdogBusyEnabled != nil {
|
|
o.WatchDogBusy = *settings.WatchdogBusyEnabled
|
|
if o.WatchDogBusy {
|
|
o.WatchDog = true
|
|
}
|
|
requireRestart = true
|
|
}
|
|
if settings.WatchdogIdleTimeout != nil {
|
|
if dur, err := time.ParseDuration(*settings.WatchdogIdleTimeout); err == nil {
|
|
o.WatchDogIdleTimeout = dur
|
|
requireRestart = true
|
|
}
|
|
}
|
|
if settings.WatchdogBusyTimeout != nil {
|
|
if dur, err := time.ParseDuration(*settings.WatchdogBusyTimeout); err == nil {
|
|
o.WatchDogBusyTimeout = dur
|
|
requireRestart = true
|
|
}
|
|
}
|
|
if settings.WatchdogInterval != nil {
|
|
if dur, err := time.ParseDuration(*settings.WatchdogInterval); err == nil {
|
|
o.WatchDogInterval = dur
|
|
requireRestart = true
|
|
}
|
|
}
|
|
if settings.MaxActiveBackends != nil {
|
|
o.MaxActiveBackends = *settings.MaxActiveBackends
|
|
o.SingleBackend = (*settings.MaxActiveBackends == 1)
|
|
requireRestart = true
|
|
} else if settings.SingleBackend != nil {
|
|
o.SingleBackend = *settings.SingleBackend
|
|
if *settings.SingleBackend {
|
|
o.MaxActiveBackends = 1
|
|
} else {
|
|
o.MaxActiveBackends = 0
|
|
}
|
|
requireRestart = true
|
|
}
|
|
if settings.MemoryReclaimerEnabled != nil {
|
|
o.MemoryReclaimerEnabled = *settings.MemoryReclaimerEnabled
|
|
if *settings.MemoryReclaimerEnabled {
|
|
o.WatchDog = true
|
|
}
|
|
requireRestart = true
|
|
}
|
|
if settings.MemoryReclaimerThreshold != nil {
|
|
if *settings.MemoryReclaimerThreshold > 0 && *settings.MemoryReclaimerThreshold <= 1.0 {
|
|
o.MemoryReclaimerThreshold = *settings.MemoryReclaimerThreshold
|
|
requireRestart = true
|
|
}
|
|
}
|
|
if settings.ForceEvictionWhenBusy != nil {
|
|
o.ForceEvictionWhenBusy = *settings.ForceEvictionWhenBusy
|
|
// This setting doesn't require restart, can be updated dynamically
|
|
}
|
|
if settings.LRUEvictionMaxRetries != nil {
|
|
o.LRUEvictionMaxRetries = *settings.LRUEvictionMaxRetries
|
|
// This setting doesn't require restart, can be updated dynamically
|
|
}
|
|
if settings.LRUEvictionRetryInterval != nil {
|
|
if dur, err := time.ParseDuration(*settings.LRUEvictionRetryInterval); err == nil {
|
|
o.LRUEvictionRetryInterval = dur
|
|
// This setting doesn't require restart, can be updated dynamically
|
|
}
|
|
}
|
|
if settings.Threads != nil {
|
|
o.Threads = *settings.Threads
|
|
}
|
|
if settings.ContextSize != nil {
|
|
o.ContextSize = *settings.ContextSize
|
|
}
|
|
if settings.F16 != nil {
|
|
o.F16 = *settings.F16
|
|
}
|
|
if settings.Debug != nil {
|
|
o.Debug = *settings.Debug
|
|
}
|
|
if settings.EnableTracing != nil {
|
|
o.EnableTracing = *settings.EnableTracing
|
|
}
|
|
if settings.TracingMaxItems != nil {
|
|
o.TracingMaxItems = *settings.TracingMaxItems
|
|
}
|
|
if settings.TracingMaxBodyBytes != nil {
|
|
o.TracingMaxBodyBytes = *settings.TracingMaxBodyBytes
|
|
}
|
|
if settings.EnableBackendLogging != nil {
|
|
o.EnableBackendLogging = *settings.EnableBackendLogging
|
|
}
|
|
if settings.CORS != nil {
|
|
o.CORS = *settings.CORS
|
|
}
|
|
if settings.CSRF != nil {
|
|
o.DisableCSRF = *settings.CSRF
|
|
}
|
|
if settings.CORSAllowOrigins != nil {
|
|
o.CORSAllowOrigins = *settings.CORSAllowOrigins
|
|
}
|
|
if settings.P2PToken != nil {
|
|
o.P2PToken = *settings.P2PToken
|
|
}
|
|
if settings.P2PNetworkID != nil {
|
|
o.P2PNetworkID = *settings.P2PNetworkID
|
|
}
|
|
if settings.Federated != nil {
|
|
o.Federated = *settings.Federated
|
|
}
|
|
if settings.Galleries != nil {
|
|
o.Galleries = *settings.Galleries
|
|
}
|
|
if settings.BackendGalleries != nil {
|
|
o.BackendGalleries = *settings.BackendGalleries
|
|
}
|
|
if settings.AutoloadGalleries != nil {
|
|
o.AutoloadGalleries = *settings.AutoloadGalleries
|
|
}
|
|
if settings.AutoloadBackendGalleries != nil {
|
|
o.AutoloadBackendGalleries = *settings.AutoloadBackendGalleries
|
|
}
|
|
if settings.AutoUpgradeBackends != nil {
|
|
o.AutoUpgradeBackends = *settings.AutoUpgradeBackends
|
|
}
|
|
if settings.PreferDevelopmentBackends != nil {
|
|
o.PreferDevelopmentBackends = *settings.PreferDevelopmentBackends
|
|
}
|
|
if settings.AgentJobRetentionDays != nil {
|
|
o.AgentJobRetentionDays = *settings.AgentJobRetentionDays
|
|
}
|
|
if settings.OpenResponsesStoreTTL != nil {
|
|
if *settings.OpenResponsesStoreTTL == "0" || *settings.OpenResponsesStoreTTL == "" {
|
|
o.OpenResponsesStoreTTL = 0 // No expiration
|
|
} else if dur, err := time.ParseDuration(*settings.OpenResponsesStoreTTL); err == nil {
|
|
o.OpenResponsesStoreTTL = dur
|
|
}
|
|
// This setting doesn't require restart, can be updated dynamically
|
|
}
|
|
// Agent Pool settings
|
|
if settings.AgentPoolEnabled != nil {
|
|
o.AgentPool.Enabled = *settings.AgentPoolEnabled
|
|
requireRestart = true
|
|
}
|
|
if settings.AgentPoolDefaultModel != nil {
|
|
o.AgentPool.DefaultModel = *settings.AgentPoolDefaultModel
|
|
requireRestart = true
|
|
}
|
|
if settings.AgentPoolEmbeddingModel != nil {
|
|
o.AgentPool.EmbeddingModel = *settings.AgentPoolEmbeddingModel
|
|
requireRestart = true
|
|
}
|
|
if settings.AgentPoolMaxChunkingSize != nil {
|
|
o.AgentPool.MaxChunkingSize = *settings.AgentPoolMaxChunkingSize
|
|
requireRestart = true
|
|
}
|
|
if settings.AgentPoolChunkOverlap != nil {
|
|
o.AgentPool.ChunkOverlap = *settings.AgentPoolChunkOverlap
|
|
requireRestart = true
|
|
}
|
|
if settings.AgentPoolEnableLogs != nil {
|
|
o.AgentPool.EnableLogs = *settings.AgentPoolEnableLogs
|
|
requireRestart = true
|
|
}
|
|
if settings.AgentPoolCollectionDBPath != nil {
|
|
o.AgentPool.CollectionDBPath = *settings.AgentPoolCollectionDBPath
|
|
requireRestart = true
|
|
}
|
|
if settings.AgentPoolVectorEngine != nil {
|
|
o.AgentPool.VectorEngine = *settings.AgentPoolVectorEngine
|
|
requireRestart = true
|
|
}
|
|
if settings.AgentPoolDatabaseURL != nil {
|
|
o.AgentPool.DatabaseURL = *settings.AgentPoolDatabaseURL
|
|
requireRestart = true
|
|
}
|
|
if settings.AgentPoolAgentHubURL != nil {
|
|
o.AgentPool.AgentHubURL = *settings.AgentPoolAgentHubURL
|
|
requireRestart = true
|
|
}
|
|
|
|
// LocalAI Assistant: read live at request entry by the chat handler, so
|
|
// flipping the disable flag takes effect on the next request without a
|
|
// restart.
|
|
if settings.LocalAIAssistantEnabled != nil {
|
|
o.DisableLocalAIAssistant = !*settings.LocalAIAssistantEnabled
|
|
}
|
|
|
|
// Branding: read live by the public /api/branding endpoint and asset
|
|
// server, so changes apply on the next request without a restart.
|
|
if settings.InstanceName != nil {
|
|
o.Branding.InstanceName = *settings.InstanceName
|
|
}
|
|
if settings.InstanceTagline != nil {
|
|
o.Branding.InstanceTagline = *settings.InstanceTagline
|
|
}
|
|
if settings.LogoFile != nil {
|
|
o.Branding.LogoFile = *settings.LogoFile
|
|
}
|
|
if settings.LogoHorizontalFile != nil {
|
|
o.Branding.LogoHorizontalFile = *settings.LogoHorizontalFile
|
|
}
|
|
if settings.FaviconFile != nil {
|
|
o.Branding.FaviconFile = *settings.FaviconFile
|
|
}
|
|
|
|
if settings.MITMListen != nil {
|
|
o.MITMListen = *settings.MITMListen
|
|
}
|
|
|
|
// Note: ApiKeys requires special handling (merging with startup keys) - handled in caller
|
|
|
|
return requireRestart
|
|
}
|
|
|
|
// func WithMetrics(meter *metrics.Metrics) AppOption {
|
|
// return func(o *StartupOptions) {
|
|
// o.Metrics = meter
|
|
// }
|
|
// }
|