LocalAI/docs/content/features/runtime-settings.md

+++ disableToc = false title = "Runtime Settings" weight = 82 url = '/features/runtime-settings' +++

LocalAI provides a web-based interface for managing application settings at runtime. These settings can be configured through the web UI and are automatically persisted to a configuration file, allowing changes to take effect immediately without requiring a restart.

Accessing Runtime Settings

Navigate to the Settings page from the management interface at http://localhost:8080/manage. The settings page provides a comprehensive interface for configuring various aspects of LocalAI.

Available Settings

Watchdog Settings

The watchdog monitors backend activity and can automatically stop idle or overly busy models to free up resources.

• Watchdog Enabled: Master switch to enable/disable the watchdog
• Watchdog Idle Enabled: Enable stopping backends that are idle longer than the idle timeout
• Watchdog Busy Enabled: Enable stopping backends that are busy longer than the busy timeout
• Watchdog Idle Timeout: Duration threshold for idle backends (default: 15m)
• Watchdog Busy Timeout: Duration threshold for busy backends (default: 5m)

Changes to watchdog settings are applied immediately by restarting the watchdog service.

Backend Configuration

• Max Active Backends: Maximum number of active backends (loaded models). When exceeded, the least recently used model is automatically evicted. Set to 0 for unlimited, 1 for single-backend mode
• Force Eviction When Busy: Allow evicting models even when they have active API calls (default: disabled for safety). Warning: Enabling this can interrupt active requests
• LRU Eviction Max Retries: Maximum number of retries when waiting for busy models to become idle before eviction (default: 30)
• LRU Eviction Retry Interval: Interval between retries when waiting for busy models (default: 1s)

Note: The "Single Backend" setting is deprecated. Use "Max Active Backends" set to 1 for single-backend behavior.

LRU Eviction Behavior

By default, LocalAI will skip evicting models that have active API calls to prevent interrupting ongoing requests. When all models are busy and eviction is needed:

1. The system will wait for models to become idle
2. It will retry eviction up to the configured maximum number of retries
3. The retry interval determines how long to wait between attempts
4. If all retries are exhausted, the system will proceed (which may cause out-of-memory errors if resources are truly exhausted)

You can configure these settings via the web UI or through environment variables. See [VRAM Management]({{%relref "advanced/vram-management" %}}) for more details.

Performance Settings

• Threads: Number of threads used for parallel computation (recommended: number of physical cores)
• Context Size: Default context size for models (default: 512)
• F16: Enable GPU acceleration using 16-bit floating point
• VRAM Budget: Cap on VRAM used for model allocation (for example 80% or 12GB; empty means no cap). See [VRAM Management]({{%relref "advanced/vram-management" %}})

Debug and Logging

• Debug Mode: Enable debug logging (deprecated, use log-level instead)

API Security

• CORS: Enable Cross-Origin Resource Sharing
• CORS Allow Origins: Comma-separated list of allowed CORS origins
• CSRF: Enable CSRF protection middleware
• API Keys: Manage API keys for authentication (one per line or comma-separated)

For multi-user authentication with roles, OAuth, and usage tracking, see [Authentication & Authorization]({{%relref "features/authentication" %}}).

P2P Settings

Configure peer-to-peer networking for distributed inference:

• P2P Token: Authentication token for P2P network
• P2P Network ID: Network identifier for P2P connections
• Federated Mode: Enable federated mode for P2P network

Changes to P2P settings automatically restart the P2P stack with the new configuration.

Gallery Settings

Manage model and backend galleries:

• Model Galleries: JSON array of gallery objects with url and name fields, plus an optional mirrors list of fallback URLs (see [Gallery mirrors]({{%relref "features/model-gallery#gallery-mirrors" %}}))
• Backend Galleries: JSON array of backend gallery objects, which accept the same mirrors key
• Autoload Galleries: Automatically load model galleries on startup
• Autoload Backend Galleries: Automatically load backend galleries on startup

Agent Pool Settings

Configure the built-in agent platform (see [Agents]({{%relref "features/agents" %}}) for full documentation):

• Agent Pool Enabled: Enable or disable the agent pool feature
• Default Model: Default LLM model for new agents
• Embedding Model: Model used for knowledge base embeddings (default: granite-embedding-107m-multilingual)
• Max Chunking Size: Maximum chunk size for document ingestion (default: 400)
• Chunk Overlap: Overlap between document chunks (default: 0)
• Enable Logs: Enable detailed agent logging
• Collection DB Path: Custom path for the collections database

Note: Most agent pool settings require a restart to take effect.

Configuration Persistence

All settings are automatically saved to runtime_settings.json in the LOCALAI_CONFIG_DIR directory (default: BASEPATH/configuration). This file is watched for changes, so modifications made directly to the file will also be applied at runtime.

Settings Precedence

Every runtime setting follows a single precedence rule:

1. Environment variables and CLI flags (highest priority)
2. Configuration files (runtime_settings.json, api_keys.json)
3. Default values (lowest priority)

The same rule is applied identically in all three places where settings can change:

• At boot: values persisted in runtime_settings.json fill in every setting that was not explicitly set via an environment variable or CLI flag.
• On POST /api/settings (the web UI Settings page): if a setting is controlled by an environment variable, it cannot be modified through the web interface. The settings page will indicate when a setting is controlled by an environment variable.
• On manual file edits: the configuration watcher hot-applies edits to runtime_settings.json with the same env-over-file semantics as boot, so editing the file by hand behaves like restarting with that file. For example, hand-editing vram_budget installs the new VRAM allocation cap live, without a restart.

Known Limitations

• An environment variable explicitly set to its default value is indistinguishable from an unset one, so the value from runtime_settings.json wins in that case. In particular, an explicit LOCALAI_THREADS=0 behaves like an unset LOCALAI_THREADS.
• A field previously changed via the API (or the web UI) looks environment-set to the file watcher, so a manual file edit of that same field is not hot-applied: it takes effect on the next restart.

Example Configuration

The runtime_settings.json file follows this structure:

{
  "watchdog_enabled": true,
  "watchdog_idle_enabled": true,
  "watchdog_busy_enabled": false,
  "watchdog_idle_timeout": "15m",
  "watchdog_busy_timeout": "5m",
  "max_active_backends": 0,
  "force_eviction_when_busy": false,
  "lru_eviction_max_retries": 30,
  "lru_eviction_retry_interval": "1s",
  "threads": 8,
  "context_size": 2048,
  "f16": false,
  "debug": false,
  "cors": true,
  "csrf": false,
  "cors_allow_origins": "*",
  "p2p_token": "",
  "p2p_network_id": "",
  "federated": false,
  "galleries": [
    {
      "url": "github:mudler/LocalAI/gallery/index.yaml@master",
      "name": "localai"
    }
  ],
  "backend_galleries": [
    {
      "url": "github:mudler/LocalAI/backend/index.yaml@master",
      "name": "localai"
    }
  ],
  "autoload_galleries": true,
  "autoload_backend_galleries": true,
  "api_keys": []
}

API Keys Management

API keys can be managed through the runtime settings interface. Keys can be entered one per line or comma-separated.

Important Notes:

• API keys from environment variables are always included and cannot be removed via the UI
• Runtime API keys are stored in runtime_settings.json
• For backward compatibility, API keys can also be managed via api_keys.json
• Empty arrays will clear all runtime API keys (but preserve environment variable keys)

Dynamic Configuration

The runtime settings system supports dynamic configuration file watching. When LOCALAI_CONFIG_DIR is set, LocalAI monitors the following files for changes:

• runtime_settings.json - Unified runtime settings
• api_keys.json - API keys (for backward compatibility)
• external_backends.json - External backend configurations

Changes to these files are automatically detected and applied without requiring a restart, using the same precedence rule described in Settings Precedence.

Best Practices

1. Use Environment Variables for Production: For production deployments, use environment variables for critical settings to ensure they cannot be accidentally changed via the web UI.

2. Backup Configuration Files: Before making significant changes, consider backing up your runtime_settings.json file.

3. Monitor Resource Usage: When enabling watchdog features, monitor your system to ensure the timeout values are appropriate for your workload.

4. Secure API Keys: API keys are sensitive information. Ensure proper file permissions on configuration files (they should be readable only by the LocalAI process).

5. Test Changes: Some settings (like watchdog timeouts) may require testing to find optimal values for your specific use case.

Troubleshooting

Settings Not Applying

If settings are not being applied:

1. Check if the setting is controlled by an environment variable
2. Verify the LOCALAI_CONFIG_DIR is set correctly
3. Check file permissions on runtime_settings.json
4. Review application logs for configuration errors

Watchdog Not Working

If the watchdog is not functioning:

1. Ensure "Watchdog Enabled" is turned on
2. Verify at least one of the idle or busy watchdogs is enabled
3. Check that timeout values are reasonable for your workload
4. Review logs for watchdog-related messages

P2P Not Starting

If P2P is not starting:

1. Verify the P2P token is set (non-empty)
2. Check network connectivity
3. Ensure the P2P network ID matches across nodes (if using federated mode)
4. Review logs for P2P-related errors