mirror/LocalAI
1
0 Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-10-29 01:28:43 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
a9236ef74c349be3aff0258bd2b90ce85514f6ca
LocalAI/docs/content/reference/cli-reference.md
T
mudler-agentandEttore Di Giacinto 7c74d26555 feat(addons): run the add-on owner as its own workload (#12867) 
* feat(addons): generate and load a static carrier key set

A split deployment needs the owner and the frontends to share
certificates and a signing key that outlive a process. The loopback
carrier makes fresh keys at every start, so it cannot serve that.

GenerateCarrierKeys writes a CA certificate, an owner leaf and a
frontend leaf, and the frontend signing key pair. It drops the CA key,
so nobody can issue a third leaf, and it refuses to overwrite any file.
LoadCarrierConfig reads only the files of one role and refuses key
files that group or other users can access.

The owner never forwards, but NewCarrier still requires a signing key,
so the owner config gets a throwaway key in memory instead of the
frontends' private key.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): add util addons-carrier-keys

An operator who runs the add-on owner and its frontends as separate
workloads needs a way to make the shared key set. The command wraps
GenerateCarrierKeys: --out names the directory and --owner-host, which
can repeat, sets the names the owner certificate is valid for. It prints
which files belong on the owner and which on the frontends.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): add the owner and frontend roles to local-ai run

Every local-ai run process is its own add-on owner, so a distributed
deployment cannot scale its frontends: replicas that share a data
volume fight over the owner lock.

--addons-role selects combined (today's behaviour, the default), owner
or frontend. The owner and frontend roles also take --addons-owner-address
and --addons-carrier-dir, and the owner takes --addons-owner-listen. They
use the fixed carrier identities and the key set from the carrier key
directory. The frontend role has no add-on root, so it never creates the
add-ons folder. A bad role setup stops startup with a message that names
the flag.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): keep the frontend role away from owner state

A frontend holds no add-on state, but it still wrote the library
sources file of the data path, which is the owner's file. The frontend
role now skips the library option and reads only the galleries named by
the flags, and logs a warning when it has no account database, because
its pins are then kept per frontend.

The add-on manager rejects a relative root even for a forwarding
frontend, so the frontend config names the absolute path where an owner
would keep its state. Nothing creates or opens it, and a spec checks
that the data path stays empty.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(addons): run the owner and frontends as separate applications

The split roles are only useful if an owner and its frontends work as
separate processes that share nothing but the carrier keys and the owner
address. A spec now builds them as separate Application values with
their own data paths and keys from GenerateCarrierKeys.

It covers a frontend that lists apps and reads add-on status through the
owner, two frontends on one key set, a frontend that starts while the
owner is down and answers unavailable, a second owner refused on the
same root, and an owner that restarts on the same root and address and
still serves the installed add-on to the same frontend.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(addons): describe running the owner as its own workload

An operator who runs several frontend replicas had no documented way to
stop each one from being its own add-on owner. The add-ons page now has
a section for the owner and frontend roles: one owner replica with its
own volume and a recreate-style rollout, a stable owner address, key
generation and per-role mounts, a Compose example, what a restart and an
owner outage look like, the models-path limit, and the frontend limits.
The singleton section names the roles and the distributed-state mode,
and the distributed mode page links to the new section.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): gate the addons feature in /api/features

One function, Application.AddonsFeature, decides whether the add-on
side is on. A combined or owner process answers true when add-ons are
enabled and the owner manager exists. A frontend answers true when the
owner answered the carrier's audience call within 2 s, and keeps that
answer for 10 s so /api/features does not dial the owner per request.

/api/features reports the result as "addons", and fine_tuning and
quantization take the same value, because both are add-ons. The
frontend cache is per-instance state.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): answer 503 on every add-on route when add-ons are off

With --addons-enabled=false the real add-on, fine-tuning and
quantization routes and services are not registered. One catch-all per
prefix answers the typed unavailable response of an absent owner, for
any path under it, so a client cannot tell which routes exist. The
response is the one the gateway already writes (gateway.WriteUnavailable
over typedError), not a second format.

With add-ons on, the same routes sit behind RequireAddonsFeature, which
reads Application.AddonsFeature, so a frontend that cannot reach the
owner gives the same answer. The per-user feature middleware still runs
after the gate. No endpoint is added, so the swagger, /api/instructions
and RouteFeatureRegistry surfaces do not change.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): build no carrier for a frontend with add-ons off

addonConfigFor already returned the zero config before it read a role
flag when add-ons are disabled. Say so on the way out: when a role,
owner address, listen address or carrier directory flag is set, log one
info line that the flags are ignored. A frontend with
--addons-enabled=false needs no owner address and no key directory,
loads no keys and builds no carrier.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ui): hide add-on entries when the addons feature is off

The UI reads features.addons from /api/features. When it is false:
the sidebar shows no Add-ons entry and no pinned apps, the 404 page
lists no add-on destination, the shell does not call /api/addons/pins
or /api/addons/builtin, and /app/addons, /app/addons/*, /app/fine-tune
and /app/quantize lead back to /app. The Build hub tabs for add-ons,
fine-tuning and quantization were already keyed on features and now
follow the gate. An absent key (an older server) counts as on.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(addons): describe the addons feature gate

The --addons-enabled row now says the UI hides the add-on entries and
every add-on, fine-tuning and quantization route answers 503. A new
subsection describes the addons key of /api/features, the 10 s frontend
cache as per-instance state, and that a frontend with add-ons off needs
no owner address or key directory. The fine-tuning and quantization
pages say they are off with --addons-enabled=false.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(addons): give a frontend no add-on state root

OwnerConfig.validate required an absolute root for every role, so a
frontend named a path under the data directory that nothing was allowed
to open. Only the owner has a state root now: validate checks it when
the instance is the owner, and the frontend role leaves Root empty.
Nothing on a frontend reads or creates it.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(addons): make the carrier signing key optional for the owner

NewCarrier required an Ed25519 signing key of every instance, so the
owner role loaded a throwaway key in memory that it never used. The
owner never signs an outbound request, so it now needs none:
NewCarrier accepts an empty key when the instance is the owner and
still requires a whole key from a frontend, and a malformed key is
refused for both. LoadCarrierConfig no longer generates one. The owner
still verifies frontend signatures with frontend-sign.pub.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Keep the owner address out of the add-on identity file

The owner address is runtime configuration, so storing it and comparing
the whole file made a moved owner refuse its own root. The identity file
now holds the owner ID and storage ID only. An older file that still has
an address loads, and the address is ignored. A different owner ID or
storage ID is still refused.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(addons): state that an existing root loads under the owner role

The owner identity no longer stores its address, so a root written by a
single node loads under the owner role when the owner and storage
identity match. Say so, and keep the caveat about absolute paths in the
journal.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(addons): refuse library-source routes on a frontend

A frontend must keep no owner state, but the library-source routes, the
ApplyLibraries call and the startup official-libraries step still opened
and wrote addons/sources.json in its data path.

Mark the frontend role in the add-on config, also when add-ons are off.
On a frontend, LibrarySources returns the unavailable error,
ApplyLibraries does nothing and the startup step skips the store. The
six /api/library/sources routes answer 503 with the body that the add-on
feature gate writes, and an X-LocalAI-Reason header names the cause.
Reads are refused too: the carrier has no verb for them. Owner and
combined roles are unchanged.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(addons): put the reason in the unavailable body

WriteUnavailable wrote the plain text "addon request failed", and the
library-source refusal put its reason in an X-LocalAI-Reason header.

Write the LocalAI error envelope instead, with a message, a type and the
code "unavailable", and keep the 503 status. The message names the
cause: add-ons are off on this instance, the add-on owner is not
reachable, or library sources are managed on the owner. Remove the
header.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): serve library calls on the add-on owner

Add the carrier verbs a frontend needs to work on the library through
the owner: library-list, library-add, library-update, library-enable,
library-disable, library-remove, library-backend-resolve and
library-backend-releases. A call carries the operation in one
LibraryCall field of the wire request, so an old owner rejects it as an
invalid binding. Every verb but the three reads is a mutation, and the
reply is an envelope of data or a typed error, so a frontend gives the
status the owner's own routes give.

The owner serves them through gateway.LibraryAdmin, implemented in the
application on the same LocalAdmin as the owner's HTTP routes. It runs
the owner's administrator check before any store call.

Add two peer-authenticated reads that take no actor and leave the
audience reply unchanged: GET /addons/capabilities (protocol, calls and
the owner's OS and architecture) and GET /addons/library/snapshot (the
gallery-relevant source list and its version).

Split the backend install: Installer.Resolve is the owner's half
(resolve, pick the meta backend for the asker's capability, pin by
digest) and returns a Plan; the node that installs runs the collision
check and the gallery install itself. Move the source administration
behind library.Admin and add library.ReconcileInfos for a source list
read from the owner.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): forward library routes from a frontend to the owner

A frontend refused the library-source routes and did not serve the
backend library routes. Forward them instead, and keep the owner the
only writer of the sources.

The library-source routes work on library.Admin: the local store on a
combined or owner process, the carrier on a frontend. The frontend
checks the calling account (a legacy API key is refused with 401), and
the owner checks the administrator right again before any store call.
With add-ons off, a frontend answers these routes with a 503.

A frontend asks the owner for its source list every 60 seconds through
the peer-authenticated snapshot call and rebuilds its model and backend
gallery lists, and the official-libraries switch, only when the version
changed. If the owner does not answer, it keeps the last lists and marks
them stale. Changing official_libraries on a frontend is refused with
409.

The backend library install is split. The owner resolves the entry and
pins the image; the frontend runs the gallery install on its own node,
with its own size limit and strict mode. A frontend refuses library
calls to an owner that does not serve them, or that runs on another OS
or architecture.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(addons): document library operations on a frontend

Replace the frontend limits that said a frontend refuses the library
routes with what happens now: the library sources and the backend
library go through the owner, the gallery lists follow the owner every
60 seconds, official_libraries is set on the owner, and the owner and
the frontends must run on one OS and architecture.

Add specs for the split owner: a frontend lists sources through the
owner, a source added on a frontend reaches the owner and then the other
frontend after a refresh, a write with the owner down answers 503 with
the error envelope, official_libraries answers 409 on a frontend and a
change on the owner reaches the frontend's gallery lists, the lists are
kept and marked stale while the owner is down, a platform mismatch and a
legacy API key are refused, and a frontend installs a backend into its
own path from the owner's plan.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): check the shared models path with a probe

Exports write into the models path of the owner, and frontends load
models from their own path. A wrong mount fails silently: the export
succeeds and the model never appears.

The owner writes a probe file with a random nonce into its models path
at start. A frontend fetches the nonce over the carrier and checks at
start, then every 60 seconds, that the same file is visible at the same
absolute path. On a mismatch it logs an error, reports model_exports
false and blocks its export and import routes. The owner refuses an
export or import while its own probe is missing, and keeps running if
it cannot write the probe. An unreachable or older owner gives
"unknown" and leaves exports on.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(api): describe the library refusal and forwarding

The library-source and backend-library routes forward to the add-on
owner on a frontend and answer 503 when the owner cannot serve them.
Add the 503 to their swagger annotations, regenerate the swagger files,
and say in /api/instructions that a frontend forwards these routes.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): let frontends run the backend update check through the owner

In named-user mode a frontend has no user to act as, so the background
backend update check failed there. Add one read-only system call for
the release lookup. The owner accepts it only on the peer-authenticated
route from a pinned frontend, only for backend-releases, and without an
actor. Calls through /addons/forward cannot carry the mark, and every
user-facing call still goes through the administrator check.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(mcp): expose the backend update check as a tool

Add list_backend_library_updates, a read-only tool for
GET /backends/library/updates. The in-process client reads the
library installer, and the HTTP client calls the route. The six
library source routes stay out of the catalog: the assistant has no
per-call actor and no approval card for library changes.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(api): add export status routes for fine-tuning and quantization

The add-on UIs could not know that model exports are blocked until a user
hit the refusal of the export route. Add GET /api/fine-tuning/status and
GET /api/quantization/status. Each returns {"model_exports": bool}, from
the same check that drives the model_exports key of /api/features.

The routes sit behind the add-ons gate, so they answer 503 when add-ons
are off. They are listed in the swagger spec, the instructions registry,
the route-feature registry and the feature docs.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(addons): disable exports when the models probe fails

The fine-tuning and quantization pages now read the new status route
and turn off the export form, the checkpoint export button and the
import button when model_exports is false. A short reason shows next
to the control. A failed or slow answer leaves the controls on, as the
server does for an unknown check.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-10-24 14:29:39 +02:00

336 lines
27 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
+++
disableToc = false
title = "CLI Reference"
weight = 25
url = '/reference/cli-reference'
+++
Complete reference for all LocalAI command-line interface (CLI) parameters and environment variables.
For client-side text inference measurements, see [Benchmark text models]({{% relref "features/benchmark" %}}).
These options configure the LocalAI server process. To configure an individual
model, see [Model Configuration]({{% relref "advanced/model-configuration" %}}).
> **Note:** All CLI flags can also be set via environment variables. Environment variables take precedence over CLI flags. See [.env files]({{%relref "advanced/advanced-usage#env-files" %}}) for configuration file support.
## Global Flags
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `-h, --help` | | Show context-sensitive help | |
| `--log-level` | `info` | Set the level of logs to output [error,warn,info,debug,trace] | `$LOCALAI_LOG_LEVEL` |
| `--credentials-file` | | YAML file with credentials for private registries, galleries and download hosts. See [Private Registries and Galleries]({{% relref "advanced/private-sources" %}}) | `$LOCALAI_CREDENTIALS_FILE` |
| `--debug` | `false` | **DEPRECATED** - Use `--log-level=debug` instead. Enable debug logging | `$LOCALAI_DEBUG`, `$DEBUG` |
## Diagnostics flags
See [Profiling and request diagnostics]({{% relref "features/diagnostics" %}}) for sampling semantics, phase boundaries, and protected pod captures.
These options apply only to `local-ai run`, including distributed frontends.
Diagnostics state is per-instance, not shared between replicas. Debug logging enables neither feature.
The profiling listener is separate from the public API and accepts only numeric loopback addresses with explicit ports.
`--preload-backend-only` validates these options but does not start profiling or sampling.
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--pprof` | `false` | Enable the private loopback profiling listener | `$LOCALAI_PPROF` |
| `--pprof-address` | `127.0.0.1:6060` | Numeric loopback IP and port (1–65535), including bracketed IPv6; validated even when disabled | `$LOCALAI_PPROF_ADDRESS` |
| `--pprof-mutex-profile-fraction` | `0` | Nonnegative mutex sampling fraction; nonzero requires pprof | `$LOCALAI_PPROF_MUTEX_PROFILE_FRACTION` |
| `--pprof-block-profile-rate` | `0` | Nonnegative block sampling rate in nanoseconds; nonzero requires pprof | `$LOCALAI_PPROF_BLOCK_PROFILE_RATE` |
| `--request-phase-timing` | `false` | Emit sanitized phase timing at Info, independently of pprof; warn/error filtering suppresses events | `$LOCALAI_REQUEST_PHASE_TIMING` |
Sampling adds overhead. The process applies configured rates once and resets them to zero on shutdown.
There is no runtime toggle, continuous capture, or public API route for diagnostics.
The private listener has no authentication: other processes in the same network namespace can access profiles.
Profiles can contain secrets; protect captured files. The `/debug/pprof/cmdline` endpoint is unavailable.
Shutdown allows five seconds for cleanup, then force-closes profiling connections.
Existing application initialization operations may not promptly observe cancellation.
The signal callback still returns after its cleanup budget expires.
If initialization fails without returning an application handle, this lifecycle cannot shut down resources created inside that failed initialization.
Application shutdown has no context parameter; work that ignores cancellation can outlive the cleanup budget.
## Storage Flags
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--models-path` | `BASEPATH/models` | Path containing models used for inferencing | `$LOCALAI_MODELS_PATH`, `$MODELS_PATH` |
| `--download-staging-path` | `BASEPATH/downloading` | Path where in-flight downloads are staged before extraction | `$LOCALAI_DOWNLOAD_STAGING_PATH`, `$DOWNLOAD_STAGING_PATH` |
| `--data-path` | `BASEPATH/data` | Path for persistent data (collectiondb, agent state, tasks, jobs). Separates mutable data from configuration | `$LOCALAI_DATA_PATH` |
| `--generated-content-path` | `TMPDIR/localai-UID/generated/content` | Location for assets generated by backends (e.g. stablediffusion, images, audio, videos). Defaults under the OS temp dir (`$TMPDIR`, falling back to `/tmp`), scoped to the current user's UID so accounts sharing a host never collide. | `$LOCALAI_GENERATED_CONTENT_PATH`, `$GENERATED_CONTENT_PATH` |
| `--upload-path` | `TMPDIR/localai-UID/upload` | Path to store uploads from files API. Defaults under the OS temp dir (`$TMPDIR`, falling back to `/tmp`), scoped to the current user's UID. | `$LOCALAI_UPLOAD_PATH`, `$UPLOAD_PATH` |
| `--localai-config-dir` | `BASEPATH/configuration` | Directory for dynamic loading of certain configuration files (currently runtime_settings.json, api_keys.json, and external_backends.json). See [Runtime Settings]({{%relref "features/runtime-settings" %}}) for web-based configuration. | `$LOCALAI_CONFIG_DIR` |
| `--localai-config-dir-poll-interval` | | Time duration to poll the LocalAI Config Dir if your system has broken fsnotify events (example: `1m`) | `$LOCALAI_CONFIG_DIR_POLL_INTERVAL` |
| `--models-config-file` | | YAML file containing a list of model backend configs (alias: `--config-file`) | `$LOCALAI_MODELS_CONFIG_FILE`, `$CONFIG_FILE` |
| `--addon-host-ops` | `fine-tuning=finetune.,quantization=quantize.` | Add-ons that may call host operations, as a comma list of `addon-id=operation-prefix`. An add-on that is not listed gets no host operation channel; an empty value lists none. See [Host operations]({{% relref "features/addons" %}}) | `$LOCALAI_ADDON_HOST_OPS` |
| `--addons-role` | `combined` | How this process takes part in the add-on owner: `combined` (the node is its own owner), `owner` or `frontend`. See [Turning the owner on or off]({{% relref "features/addons#turning-the-owner-on-or-off" %}}) | `$LOCALAI_ADDONS_ROLE` |
| `--addons-owner-address` | | `https://host:port` of the add-on owner carrier, with no path. Required for the `owner` and `frontend` roles | `$LOCALAI_ADDONS_OWNER_ADDRESS` |
| `--addons-owner-listen` | | `owner` role only: the address the carrier listens on. Empty listens on all interfaces, on the port of `--addons-owner-address` | `$LOCALAI_ADDONS_OWNER_LISTEN` |
| `--addons-carrier-dir` | | Directory with the carrier key set from `local-ai util addons-carrier-keys`. Required for the `owner` and `frontend` roles | `$LOCALAI_ADDONS_CARRIER_DIR` |
| `--artifact-download-concurrency` | `1` | How many files of a model artifact to download at once. `1` downloads sequentially. Raising it helps artifacts split into many files on a fast link, at the cost of more concurrent load on the models volume. Whole files only — a single file is never split, so resume and per-file checksum verification are unaffected | `$LOCALAI_ARTIFACT_DOWNLOAD_CONCURRENCY` |
Backend processes receive a private scratch directory through `TMPDIR`, `TMP`,
and `TEMP`. LocalAI removes that directory when the backend exits and removes
abandoned directories left by a LocalAI crash before starting another backend.
Set `$LOCALAI_BACKEND_TEMP_DIR` to choose their base volume. LocalAI always
appends `localai-UID/backend-runtime`; the default base is `TMPDIR`.
## Backend Flags
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--backends-path` | `BASEPATH/backends` | Path containing backends used for inferencing | `$LOCALAI_BACKENDS_PATH`, `$BACKENDS_PATH` |
| `--backends-system-path` | `/var/lib/local-ai/backends` | Path containing system backends used for inferencing | `$LOCALAI_BACKENDS_SYSTEM_PATH`, `$BACKEND_SYSTEM_PATH` |
| `--external-backends` | | A list of external backends to load from gallery on boot | `$LOCALAI_EXTERNAL_BACKENDS`, `$EXTERNAL_BACKENDS` |
| `--external-grpc-backends` | | A list of external gRPC backends (format: `BACKEND_NAME:URI`) | `$LOCALAI_EXTERNAL_GRPC_BACKENDS`, `$EXTERNAL_GRPC_BACKENDS` |
| `--backend-galleries` | | JSON list of backend galleries | `$LOCALAI_BACKEND_GALLERIES`, `$BACKEND_GALLERIES` |
| `--autoload-backend-galleries` | `true` | Automatically load backend galleries on startup | `$LOCALAI_AUTOLOAD_BACKEND_GALLERIES`, `$AUTOLOAD_BACKEND_GALLERIES` |
| `--max-active-backends` | `0` | Maximum number of active backends (loaded models). When exceeded, the least recently used model is evicted. Set to `0` for unlimited, `1` for single-backend mode | `$LOCALAI_MAX_ACTIVE_BACKENDS`, `$MAX_ACTIVE_BACKENDS` |
| `--single-active-backend` | `false` | **DEPRECATED** - Use `--max-active-backends=1` instead. Allow only one backend to be run at a time | `$LOCALAI_SINGLE_ACTIVE_BACKEND`, `$SINGLE_ACTIVE_BACKEND` |
| `--preload-backend-only` | `false` | Do not launch the API services, only the preloaded models/backends are started (useful for multi-node setups) | `$LOCALAI_PRELOAD_BACKEND_ONLY`, `$PRELOAD_BACKEND_ONLY` |
| `--enable-watchdog-idle` | `false` | Enable watchdog for stopping backends that are idle longer than the watchdog-idle-timeout | `$LOCALAI_WATCHDOG_IDLE`, `$WATCHDOG_IDLE` |
| `--watchdog-idle-timeout` | `15m` | Threshold beyond which an idle backend should be stopped | `$LOCALAI_WATCHDOG_IDLE_TIMEOUT`, `$WATCHDOG_IDLE_TIMEOUT` |
| `--enable-watchdog-busy` | `false` | Enable watchdog for stopping backends that are busy longer than the watchdog-busy-timeout | `$LOCALAI_WATCHDOG_BUSY`, `$WATCHDOG_BUSY` |
| `--watchdog-busy-timeout` | `5m` | Threshold beyond which a busy backend should be stopped | `$LOCALAI_WATCHDOG_BUSY_TIMEOUT`, `$WATCHDOG_BUSY_TIMEOUT` |
| `--watchdog-interval` | `500ms` | Interval between watchdog checks (e.g., `500ms`, `5s`, `1m`) | `$LOCALAI_WATCHDOG_INTERVAL`, `$WATCHDOG_INTERVAL` |
| `--force-eviction-when-busy` | `false` | Force eviction even when models have active API calls (default: false for safety). **Warning:** Enabling this can interrupt active requests | `$LOCALAI_FORCE_EVICTION_WHEN_BUSY`, `$FORCE_EVICTION_WHEN_BUSY` |
| `--lru-eviction-max-retries` | `30` | Maximum number of retries when waiting for busy models to become idle before eviction | `$LOCALAI_LRU_EVICTION_MAX_RETRIES`, `$LRU_EVICTION_MAX_RETRIES` |
| `--lru-eviction-retry-interval` | `1s` | Interval between retries when waiting for busy models to become idle (e.g., `1s`, `2s`) | `$LOCALAI_LRU_EVICTION_RETRY_INTERVAL`, `$LRU_EVICTION_RETRY_INTERVAL` |
| `--model-load-failure-cooldown` | `10s` | After a model load fails, refuse new load attempts for that model for this long (HTTP 503 + `Retry-After`) so a client polling a broken model doesn't respawn a crashing backend every request. Doubles per consecutive failure up to 5m; reset on success. `0` disables | `$LOCALAI_MODEL_LOAD_FAILURE_COOLDOWN`, `$MODEL_LOAD_FAILURE_COOLDOWN` |
For more information on VRAM management, see [VRAM and Memory Management]({{%relref "advanced/vram-management" %}}).
## Models Flags
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--galleries` | | JSON list of galleries | `$LOCALAI_GALLERIES`, `$GALLERIES` |
| `--autoload-galleries` | `true` | Automatically load galleries on startup | `$LOCALAI_AUTOLOAD_GALLERIES`, `$AUTOLOAD_GALLERIES` |
| `--vram-persistent-cache` | `true` | Persist successful remote VRAM metadata probes across restarts | `$LOCALAI_VRAM_PERSISTENT_CACHE`, `$VRAM_PERSISTENT_CACHE` |
| `--preload-models` | | A list of models to apply in JSON at start | `$LOCALAI_PRELOAD_MODELS`, `$PRELOAD_MODELS` |
| `--models` | | A list of model configuration URLs to load | `$LOCALAI_MODELS`, `$MODELS` |
| `--preload-models-config` | | A list of models to apply at startup. Path to a YAML config file | `$LOCALAI_PRELOAD_MODELS_CONFIG`, `$PRELOAD_MODELS_CONFIG` |
| `--load-to-memory` | | A list of models to load into memory at startup | `$LOCALAI_LOAD_TO_MEMORY`, `$LOAD_TO_MEMORY` |
> **Note:** You can also pass model configuration URLs as positional arguments: `local-ai run MODEL_URL1 MODEL_URL2 ...`
## Performance Flags
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--f16` | `false` | Enable GPU acceleration | `$LOCALAI_F16`, `$F16` |
| `-t, --threads` | | Number of threads used for parallel computation. Usage of the number of physical cores in the system is suggested | `$LOCALAI_THREADS`, `$THREADS` |
| `--context-size` | | Default context size for models (`-1` = each model's full trained context from GGUF metadata) | `$LOCALAI_CONTEXT_SIZE`, `$CONTEXT_SIZE` |
## API Flags
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--address` | `:8080` | Bind address for the API server | `$LOCALAI_ADDRESS`, `$ADDRESS` |
| `--max-concurrent-backend-requests` | `1024` | Process-wide ceiling for concurrent backend inference operations. Excess inference receives HTTP 429 with `Retry-After`; UI and administrative endpoints remain available | `$LOCALAI_MAX_CONCURRENT_BACKEND_REQUESTS`, `$MAX_CONCURRENT_BACKEND_REQUESTS` |
| `--cors` | `false` | Enable CORS (Cross-Origin Resource Sharing) | `$LOCALAI_CORS`, `$CORS` |
| `--cors-allow-origins` | | Comma-separated list of allowed CORS origins | `$LOCALAI_CORS_ALLOW_ORIGINS`, `$CORS_ALLOW_ORIGINS` |
| `--disable-csrf` | `false` | Disable CSRF middleware (enabled by default) | `$LOCALAI_DISABLE_CSRF` |
| `--disable-http-compression` | `false` | Disable gzip compression of HTTP responses. Compression is enabled by default; streaming endpoints (streaming chat completions, SSE bridges, WebSocket upgrades) and already-compressed formats are never compressed | `$LOCALAI_DISABLE_HTTP_COMPRESSION` |
| `--http-compression-min-length` | `1024` | Minimum response size in bytes before gzip compression is applied. Smaller responses are sent as-is because the gzip envelope would outweigh the saving | `$LOCALAI_HTTP_COMPRESSION_MIN_LENGTH` |
| `--upload-limit` | `15` | Default upload-limit in MB | `$LOCALAI_UPLOAD_LIMIT`, `$UPLOAD_LIMIT` |
| `--api-keys` | | List of API Keys to enable API authentication. When this is set, all requests must be authenticated with one of these API keys | `$LOCALAI_API_KEY`, `$API_KEY` |
| `--disable-webui` | `false` | Disables the web user interface. When set to true, the server will only expose API endpoints without serving the web interface | `$LOCALAI_DISABLE_WEBUI`, `$DISABLE_WEBUI` |
| `--disable-runtime-settings` | `false` | Disables the runtime settings feature. When set to true, the server will not load runtime settings from the `runtime_settings.json` file and the settings web interface will be disabled | `$LOCALAI_DISABLE_RUNTIME_SETTINGS`, `$DISABLE_RUNTIME_SETTINGS` |
| `--disable-gallery-endpoint` | `false` | Disable the gallery endpoints | `$LOCALAI_DISABLE_GALLERY_ENDPOINT`, `$DISABLE_GALLERY_ENDPOINT` |
| `--disable-metrics-endpoint` | `false` | Disable the `/metrics` endpoint | `$LOCALAI_DISABLE_METRICS_ENDPOINT`, `$DISABLE_METRICS_ENDPOINT` |
| `--machine-tag` | | If not empty, add that string to Machine-Tag header in each response. Useful to track response from different machines using multiple P2P federated nodes | `$LOCALAI_MACHINE_TAG`, `$MACHINE_TAG` |
## Hardening Flags
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--disable-predownload-scan` | `false` | If true, disables the best-effort security scanner before downloading any files | `$LOCALAI_DISABLE_PREDOWNLOAD_SCAN` |
| `--disable-backend-install` | `false` | If true, refuse to install or upgrade backends from galleries, OCI images, URLs or paths, including automatic installs, so the running backend set stays exactly what was deployed and tested; only backends already present or provided via `--backends-system-path` are used | `$LOCALAI_DISABLE_BACKEND_INSTALL`, `$DISABLE_BACKEND_INSTALL` |
| `--opaque-errors` | `false` | If true, all error responses are replaced with blank 500 errors. This is intended only for hardening against information leaks and is normally not recommended | `$LOCALAI_OPAQUE_ERRORS` |
| `--use-subtle-key-comparison` | `false` | If true, API Key validation comparisons will be performed using constant-time comparisons rather than simple equality. This trades off performance on each request for resilience against timing attacks | `$LOCALAI_SUBTLE_KEY_COMPARISON` |
| `--disable-api-key-requirement-for-http-get` | `false` | If true, a valid API key is not required to issue GET requests to portions of the web UI. This should only be enabled in secure testing environments | `$LOCALAI_DISABLE_API_KEY_REQUIREMENT_FOR_HTTP_GET` |
| `--http-get-exempted-endpoints` | `^/$,^/app(/.*)?$,^/browse(/.*)?$,^/login/?$,^/explorer/?$,^/assets/.*$,^/static/.*$,^/swagger.*$` | If `--disable-api-key-requirement-for-http-get` is overridden to true, this is the list of endpoints to exempt. Only adjust this in case of a security incident or as a result of a personal security posture review | `$LOCALAI_HTTP_GET_EXEMPTED_ENDPOINTS` |
## Authentication Flags
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--auth-enabled` | `false` | Enable user authentication and authorization | `$LOCALAI_AUTH` |
| `--auth-database-url` | `{DataPath}/database.db` | Database URL for auth - `postgres://...` for PostgreSQL, or a file path for SQLite | `$LOCALAI_AUTH_DATABASE_URL`, `$DATABASE_URL` |
| `--github-client-id` | | GitHub OAuth App Client ID (auto-enables auth when set) | `$GITHUB_CLIENT_ID` |
| `--github-client-secret` | | GitHub OAuth App Client Secret | `$GITHUB_CLIENT_SECRET` |
| `--oidc-issuer` | | OIDC issuer URL for auto-discovery | `$LOCALAI_OIDC_ISSUER` |
| `--oidc-client-id` | | OIDC Client ID (auto-enables auth when set) | `$LOCALAI_OIDC_CLIENT_ID` |
| `--oidc-client-secret` | | OIDC Client Secret | `$LOCALAI_OIDC_CLIENT_SECRET` |
| `--auth-base-url` | | Base URL for OAuth callbacks (e.g. `http://localhost:8080`) | `$LOCALAI_BASE_URL` |
| `--auth-admin-email` | | Email address to auto-promote to admin role on login | `$LOCALAI_ADMIN_EMAIL` |
| `--auth-registration-mode` | `open` | Registration mode: `open`, `approval`, or `invite` | `$LOCALAI_REGISTRATION_MODE` |
| `--disable-local-auth` | `false` | Disable local email/password registration and login (for OAuth/OIDC-only setups) | `$LOCALAI_DISABLE_LOCAL_AUTH` |
See [Authentication & Authorization]({{%relref "features/authentication" %}}) for full documentation.
## Chat Flags
Use `local-ai chat` to run the built-in terminal agent against a LocalAI server.
The agent can run shell commands, delegate to sub-agents, and use MCP tools. Read-only
calls run on their own; everything else goes through an approval prompt you answer.
See [Terminal agent]({{% relref "features/terminal-agent" %}}) for the full feature page.
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--endpoint` | `http://127.0.0.1:8080` | LocalAI server endpoint. The `/v1` path is added automatically when omitted. | `$LOCALAI_CHAT_ENDPOINT` |
| `--model` | | Model to use. Defaults to the only model the server offers, or asks when there are several. | |
| `--api-key` | | API key to use when the LocalAI server requires authentication. | `$LOCALAI_API_KEY`, `$API_KEY` |
| `--config-dir` | `~/.config/localai/chat` | Directory holding the agent's config, plugins, and skills. | `$LOCALAI_CHAT_CONFIG_DIR` |
| `--trace-dir` | | Write a session LLM trace (NDJSON) to this directory. | `$LOCALAI_CHAT_TRACE_DIR` |
| `--cli` | `false` | Plain CLI mode instead of the full-screen interface. Pipe stdin for one-shot use. | |
| `--tui` | `false` | Force the full-screen interface. | |
| `--height` | | Run as an inline drop-down of this height, e.g. `40%`. | |
| `--tmux` / `--no-tmux` | | Control the tmux split. | |
| `--init` | | Print the shell integration script for `Ctrl+Space` (zsh, bash, or fish). | |
| `--yolo` | `false` | Auto-approve every tool call. Use with care. | `$LOCALAI_CHAT_YOLO` |
The agent's own subcommands are reached by passing them through. LocalAI's flags
must come first, because everything after the first positional argument is
forwarded verbatim:
```bash
local-ai chat plugin install https://github.com/user/plugin
local-ai chat skill list
local-ai chat mcp add my-server -- npx -y @modelcontextprotocol/server-filesystem /tmp
local-ai chat --config-dir /srv/agent plugin list # flags first
```
{{% notice warning %}}
In a script, pass `--yes` to those subcommands. Without it, a non-interactive
`plugin install` installs the plugin, leaves it disabled, and still exits `0`.
{{% /notice %}}
Summon the agent from any shell prompt with `Ctrl+Space`:
```bash
echo 'eval "$(local-ai chat --init zsh)"' >> ~/.zshrc
```
Inside a session:
- `/models` lists the models the server offers, marking the current one.
- `/model <name>` switches model, keeping the conversation.
- `/compact` summarizes the conversation so far to free up context.
- `/skill <name>`, `/agent <name> <task>`, `/attach <file>`, `/goal <text>`.
If no server is reachable, the agent offers to start one for the session
(interactive terminals only) and otherwise points you at `local-ai run`.
Piped use needs `--cli`, and exits `0` when it answers:
```bash
echo "what is 2+2" | local-ai chat --cli
```
Read-only tools still run in a piped session. A tool call that is not read-only has
nobody to approve it, so it is denied and the session exits `3`, a code chosen to be
distinct from the `1` a failure reports.
## P2P Flags
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--p2p` | `false` | Enable P2P mode | `$LOCALAI_P2P`, `$P2P` |
| `--p2p-dht-interval` | `360` | Interval for DHT refresh (used during token generation) | `$LOCALAI_P2P_DHT_INTERVAL`, `$P2P_DHT_INTERVAL` |
| `--p2p-otp-interval` | `9000` | Interval for OTP refresh (used during token generation) | `$LOCALAI_P2P_OTP_INTERVAL`, `$P2P_OTP_INTERVAL` |
| `--p2ptoken` | | Token for P2P mode (optional) | `$LOCALAI_P2P_TOKEN`, `$P2P_TOKEN`, `$TOKEN` |
| `--p2p-network-id` | | Network ID for P2P mode, can be set arbitrarily by the user for grouping a set of instances | `$LOCALAI_P2P_NETWORK_ID`, `$P2P_NETWORK_ID` |
| `--federated` | `false` | Enable federated instance | `$LOCALAI_FEDERATED`, `$FEDERATED` |
## Library sources
`local-ai library` manages the sources LocalAI reads add-ons, backends and models from. The sources are stored under the data path (`addons/sources.json`), the file the server reads, so the commands work with the server stopped.
| Command | What it does |
|---------|--------------|
| `local-ai library list` | List every source with its kind, id, URL, and whether it is enabled, a shipped default, or removed |
| `local-ai library add <url> --kind addon\|backend\|model [--id ID] [--name NAME] [--format library\|gallery]` | Add a source of your own |
| `local-ai library remove <id>` | Remove a source. A removed default leaves a tombstone, so it does not come back at the next start |
| `local-ai library enable <id>` | Turn a source on. This also brings back a removed default |
| `local-ai library disable <id>` | Turn a source off without removing it |
All `local-ai library` commands accept `--official-libraries` (`LOCALAI_OFFICIAL_LIBRARIES`, default `true`). With `--official-libraries=false` the list shows the shipped default sources as off. The same switch is the runtime setting `official_libraries` and the toggle "Use official libraries" in Settings. See [Library sources](/features/addons/#library-sources).
When two enabled sources list the same entry (same kind and id), your own source wins over a shipped default. If two of your own sources list it, the entry is refused and the message names both sources: disable or remove one of them. The `--galleries` and `--backend-galleries` flags keep working: each gallery they name is added as a source of your own (kind `model` or `backend`, format `gallery`) at startup, unless it is already a source.
## Other Commands
LocalAI supports several subcommands beyond `run`:
- `local-ai models` - Manage LocalAI models and definitions
- `local-ai backends` - Manage LocalAI backends and definitions
- `local-ai chat` - Run the built-in terminal agent against a LocalAI server
- `local-ai tts` - Convert text to speech
- `local-ai sound-generation` - Generate audio files from text or audio
- `local-ai transcript` - Convert audio to text
- `local-ai agent` - Run agents standalone without the full LocalAI server
- `local-ai mcp-server` - Run the LocalAI admin tool surface as a stdio MCP server (controls a remote LocalAI instance over HTTP)
- `local-ai worker` - Start a worker for distributed mode (generic, backend-agnostic)
- `local-ai p2p-worker` - Run workers to distribute workload via p2p (llama.cpp-only)
- `local-ai agent-worker` - Start an agent worker for distributed mode (executes agent chats via NATS)
- `local-ai util` - Utility commands
- `local-ai explorer` - Run P2P explorer
- `local-ai federated` - Run LocalAI in federated mode
- `local-ai completion` - Generate shell completion scripts for bash, zsh, or fish
Use `local-ai <command> --help` for more information on each command.
### Add-on carrier keys
`local-ai util addons-carrier-keys` makes the key set that a separate add-on owner and its frontends share. See [Running the owner as its own workload]({{% relref "features/addons#running-the-owner-as-its-own-workload" %}}) for how the set is used.
| Flag | Description |
|------|-------------|
| `--out` | Directory to write the key set to. Required. The command fails if the directory already holds any of the files, and it never overwrites a file. |
| `--owner-host` | DNS name or IP address of the add-on owner, as the frontends dial it. Required. Repeat the flag for more than one. |
```bash
local-ai util addons-carrier-keys --out ./carrier-keys --owner-host addons.owner.example
```
The command prints which files go to the owner and which go to the frontends. It writes the files with mode `0600` in a directory with mode `0700`, and it does not keep the CA key.
## Examples
### Basic Usage
```bash
./local-ai run
./local-ai run --models-path /path/to/models --address :9090
./local-ai run --f16
```
### Environment Variables
```bash
export LOCALAI_MODELS_PATH=/path/to/models
export LOCALAI_ADDRESS=:9090
export LOCALAI_F16=true
./local-ai run
```
### Advanced Configuration
```bash
./local-ai run \
--models model1.yaml model2.yaml \
--enable-watchdog-idle \
--watchdog-idle-timeout=10m \
--p2p \
--federated
```
## Related Documentation
- See [Advanced Usage]({{%relref "advanced/advanced-usage" %}}) for configuration examples
- See [VRAM and Memory Management]({{%relref "advanced/vram-management" %}}) for memory management options
Reference in new issue View Git Blame Copy Permalink