mirror of
https://github.com/mudler/LocalAI.git
synced 2026-08-03 12:00:43 -04:00
d0119bf62c
* chore(deps): bump cogito to v0.11 ahead of the nib harness nib is the agent harness that becomes 'local-ai chat'. It requires cogito v0.11, so pull that bump forward on its own: minimal version selection would apply it to LocalAI anyway, and both repos use cogito and cogito/clients. Landing it separately keeps the harness change reviewable. nib itself is not pinned yet. Nothing in LocalAI imports it, and 'go mod tidy' runs as a goreleaser before-hook in CI, so an unimported require line does not survive. It lands with its first importer. No LocalAI call site needed a change. Both cogito.WithMaxAttempts callers guard the argument above zero, so v0.11's new clamp is unreachable, and LocalAI's Multimedia values implement only URL(), so v0.11's new TypedMultimedia routing treats them as images exactly as v0.10 did. Binary size (cmd/local-ai): 200,301,381 -> 200,336,045 bytes (+34,664). A throwaway probe that links nib measured 201,042,243 bytes (+740,862 over the pre-change baseline). Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): resolve and seed the agent state directory Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): write the agent config atomically and tighten its modes Replacing config.yaml in place truncated it first, so an interrupted write would have destroyed the api_key nib keeps in the same file. Stage through a sibling temp file and rename over the target instead, and match nib's 0700 directory mode. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): probe the endpoint and classify failures Probe lists what a LocalAI endpoint advertises and separates the two failures that need different advice: nothing listening, and rejected credentials. go-openai reports a rejected key as one of two concrete types depending on the error body, and both occur against a real LocalAI. The normal error handler sends an OpenAI error envelope, which arrives as *openai.APIError; the opaque-errors handler replies with a bare status and no body, which arrives as *openai.RequestError. Classifying on only one of them misses half the cases, so the status is read from either. A cancelled probe is not reported as an unreachable server, because it learned nothing about the endpoint, and neither is a reply that could not be parsed, because something did answer. Both would otherwise send the user off to start a server that may already be running. The model list is returned verbatim and in server order. LocalAI lists whatever it finds in the models directory, including stray archives and dotfiles, and deciding which advertised ids are real belongs to whoever presents them. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): resolve the model from flag, config, or the server Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(chat): pin that model resolution sorts a copy of the caller's slice The sort spec asserted only on what the chooser was offered, so replacing the defensive copy with an in-place sort of req.Available still passed all 37 specs. Assert the input slice's order after the call, so the guarantee cannot be dropped silently by a later refactor. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): offer to start a server when none is reachable Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): bound the server wait and pin readiness and stop semantics Set cmd.WaitDelay so a backend subprocess holding the child's stderr pipe cannot block cmd.Wait forever, which would leave exited unclosed, burn the whole shutdown grace on a clean exit, and leak the waiter goroutine. Two test gaps closed alongside it: the readiness spec now counts polls, so treating 503 as ready is observable, and Stop's single-interrupt contract is pinned by giving StartedServer interrupt/kill hooks that a spec can count. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(chat): drive Stop through one process interface, hide exec plumbing Two independent interrupt/kill func fields plus a nil check admitted wirings no test could distinguish: the pair swapped, so a SIGKILL would strand the backends SIGINT exists to let local-ai run clean up, or kill left nil, so a wedged server never escalates. One two-method interface that *os.Process already satisfies leaves nothing to swap and nothing to nil. Also translate exec.ErrWaitDelay, whose text names an os/exec struct field, into what the user can act on. os/exec only substitutes that sentinel when the process exited without an error of its own, so no exit status is swallowed. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): replace the REPL with the built-in agent local-ai chat is now the nib agent harness compiled into the binary: tool use behind an approval gate, sub-agents, MCP, plugins, and skills, all auto-configured against the local server. The REPL goes with it. Its model listing and its 401 classifier were duplicates of the ones Probe now owns, and the classifier was the version that misreads a bare 401 with no OpenAI error envelope, so keeping either would leave the package with two divergent answers to the same question. github.com/mudler/nib lands in go.mod in this commit rather than earlier: go mod tidy runs as a goreleaser before-hook on every PR, so a require line with no importer is stripped before it reaches CI. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(chat): split the pre-agent phase out of Run and pin it Everything before the handoff is testable and nothing after it is: once app.Run owns the terminal there is no seam left. prepare draws that line, takes interactivity as a parameter so the prompts can be driven over a pipe, and hands Run the state dir, the model, and any server it started. The questions move onto one prompter that owns its buffered reader. A fresh bufio.Reader per question reads ahead and discards what it buffered, so the model choice typed behind an answer to "start a server?" was lost and the next question saw EOF. choose answers with a list index and refuses an empty offer, so a value that was never on the list cannot reach ResolveModel, which persists it and starts every later run against it. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): bound each server check with a deadline Nothing bounded the model listing, so pointing chat at an address that accepts the connection and then never replies left the user with no output and no offer to start a server. The budget is context.WithTimeout rather than a cancel plus a timer. Probe deliberately refuses to call an endpoint unreachable on a context.Canceled, since a caller who gave up learned nothing about the server, and only honours a deadline. A cancel-based budget therefore expires as the one error that suppresses ErrUnreachable, exactly for the hung servers the offer exists to rescue. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): tell the user when their model choice cannot be saved The choice is meant to be asked for once. When saving it fails the user is silently asked again on the next run, and the only trace was an xlog.Warn: the agent runs at log level error, and a --log-level=error run swallows it entirely. ModelRequest gains Notify for exactly this class of problem, one that is worth telling the user about but not worth failing over, and the chat wiring points it at the same writer the question was asked on. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): stop a session's server when the process is signalled A server started for the session is stopped by a deferred call, and a signal skips deferred calls: a SIGTERM between the spawn and the exit left 'local-ai run' reparented to init with nothing left that knew to shut it down. Ctrl+C was already safe, but only incidentally, because the child shares this process' foreground process group. A signal handler rather than Pdeathsig on the child. Pdeathsig is Linux-only and, in Go, is delivered when the OS thread that forked exits rather than when the process does, so it can fire on a healthy parent. Setpgid would break the Ctrl+C that works today by taking the child out of the foreground group. SIGHUP joins SIGINT and SIGTERM: a terminal program whose terminal is gone has nobody left to talk to. The same context is what cancels the agent, which nib leaves to its embedder. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): only skip the server checks for work that stays local Two argument shapes were classified wrongly. Every 'mcp ...' invocation counted as management, so 'local-ai chat mcp --stdio', which serves the agent over MCP and needs a model like any other session, was handed an empty one. And --init, whose shell snippet a user pastes into an rc file long before any server exists, went the other way: it demanded a running server to print a static string. The mcp split is asked of nib's own IsMCPManageSubcommand rather than restated here, so a verb added upstream cannot drift out of this list. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): exit with the agent's status instead of reporting it twice nib writes what went wrong to stderr and returns nothing but an exit code, so returning that error unchanged had main log "Error running the application error=exit status 1" underneath the message the user had just read. The refusal to render the full-screen interface into a pipe is the one they meet in practice: it names --cli, and burying that hides the fix. ExitCodeError says "already reported, exit with this status". main honours it and prints nothing more, so a piped or redirected chat still fails a script the way it should. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(chat): route interactive chatter through one writer helper The prompts and notices all write to a terminal, where a failed write is not worth failing the session over and the read that follows the question reports the real problem. say says that once instead of five discarded error returns. The command's one-line help comes along: chat is no longer "an interactive chat session". Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(chat): record why the agent gets this process' streams Injecting them is what makes nib refuse to draw its full-screen interface into a pipe and name --cli, instead of rendering onto a terminal the caller may not own. The tradeoff is worth stating where the wiring is. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): stop the session's server on cancellation, not on the way out The deferred Stop is reached only if the agent returns, and cancelling the context does not make it: nib hands the TUI to bubbletea without the context, so what actually unwinds a running session today is bubbletea's own SIGINT and SIGTERM handler. SIGHUP has no such backstop, and registering for it removed the default disposition that used to end the process outright, so kill -HUP left a live TUI with a cancelled context and the started server still running. runSession watches the context alongside the agent and stops the server the moment it is cancelled, so the guarantee no longer depends on what the agent does with cancellation. Stop is idempotent, so the deferred call stays correct and free. The doc comment on shutdownContext described the mechanism it was supposed to work by rather than the one that does. Corrected, bubbletea's handler included. ResolveModel now checks the chooser's answer against what it offered. The shipped chooser answers by list index and cannot be wrong, but ModelChooser is exported, the answer is persisted, and every later run starts against it, so the invariant belongs at the consumer. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(chat): bump nib to v0.5.1 v0.5.1 carries four fixes that matter to 'local-ai chat': - --init now names the embedder's command, so the emitted widget invokes 'local-ai chat' rather than a bare 'nib' the user does not have. - A piped CLI session that succeeds exits 0 instead of failing with EOF. - EOF at a tool-approval prompt denies the call rather than approving it, and the session exits 3 (app.ExitCodeApprovalNoInput) so a script can tell "answered" from "refused to act" without reading stdout. Read-only tools are unaffected and still run. ExitStatus already unwraps app.ExitError, so the code propagates with no change here. - RunTUI passes the context to bubbletea and gives up bubbletea's own signal handler, which makes shutdownContext the single owner of the signal and stops a SIGHUP leaving a wedged TUI behind. Verified against a live server on 127.0.0.1:8080: the three --init shells, a piped prompt exiting 0, a denied 'touch' that left no file and exited 3, a read-only 'ls' that still ran and exited 0, and a SIGHUP that unwound a TUI running under a pty. Two comment blocks in run.go described the old TUI behavior and are now wrong, so they are corrected in the same change. No behavior change: both shutdownContext and runSession are untouched, and stopping the server on cancellation is still worth keeping independent of how promptly nib unwinds. One known gap, not addressed here. The widget --init now emits runs 'output=$(local-ai chat --height 50%)', and runAgent injects Stdout unconditionally, so under $(...) nib refuses the TUI for a non-terminal stream. This is the cost the runAgent comment already anticipated, now that the snippets no longer hardcode standalone nib. Ctrl+Space should not be documented until that is decided. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): let nib own stdout, so the Ctrl+Space widget works The widget 'local-ai chat --init' emits runs 'output=$(local-ai chat --height 50%)', which puts a pipe on stdout by construction. runAgent injected os.Stdout unconditionally, and nib refuses every mode but --cli when a stream it was handed is not a terminal, so Ctrl+Space printed "Re-run with --cli to use the injected streams" and inserted nothing. Verified against a pty before and after. nib reads a nil stream as "not injected" and falls back to the process stream, which is how an embedder asks for nib's own behavior. That is what stdout needs: the interface renders on /dev/tty but writes the chosen command to stdout even when stdout is a pipe, and that write is the whole of the shell-capture idiom. Stdin is deliberately left injected. A piped or redirected stdin really is ignored by the interface, so the refusal is the honest answer there, and it is the one users meet: 'echo q | local-ai chat' still says to re-run with --cli, once, exit 1. Nilling stdin the way stdout is nilled would delete that silently. Stderr is not gated by nib at all and is unchanged. One case does change and cannot be kept: 'local-ai chat > out.txt' from a terminal no longer refuses, because it is indistinguishable from the widget. It renders on /dev/tty and writes the capture line to the file, which is what standalone nib does. The app.Options literal moves into agentOptions so the decision is reachable from a spec rather than being a detail of a function that takes the terminal. Both sides of the asymmetry are pinned: reinstating 'Stdout: opts.Out' fails "hands nib nothing for the process stdout", and nilling stdin fails "hands the process stdin over". Also rewrites the last comments describing the pre-v0.5.1 behavior. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(chat): say what the stream refusal actually keys on Two comments still called it the refusal to render the interface "into a pipe". That was true when both stdin and stdout were injected, but a pipe on stdout no longer refuses, so the wording now points at precisely the case that was un-refused to make Ctrl+Space work. Only a stdin that cannot be read triggers it, and both comments now say so and name the command a user meets it with, 'echo q | local-ai chat'. The agentOptions doc also said a "file a caller chose" stays injected and refused, which reads as though 'local-ai chat > out.txt' still refuses. It does not: a shell redirect arrives as os.Stdout and is nil-ed like the widget's pipe, because the two differ only in being a regular file rather than a FIFO and nib's gate does not look at that. What stays injected is a writer an in-process caller chose for itself. Says that now, in the doc and in the spec comment that had the same ambiguity. Comments only. No behavior change. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: local-ai chat is now the built-in terminal agent `local-ai chat` was a plain chat prompt and is now an agent that runs shell commands behind an approval gate, so the pages that described a REPL were wrong rather than merely thin. Adds a Terminal agent feature page at /features/terminal-agent covering the approval gate, piped runs and their exit codes, Ctrl+Space, model resolution, state directory, and the pass-through management commands (including the `--yes` caveat that leaves a plugin installed but disabled in a script). The three-way "looking for something else" notice becomes four-way and moves into an agentic-routing shortcode. Four hand-kept copies of the same paragraph is what produced the drift the new page would otherwise have added to; the shortcode takes `current=` so each page still marks itself, and errors the build on a name that is not one of the four. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * website: the agent is in the binary, not a second install The nib section sold a separate tool you also install, with a GitHub link as the only way in, which is now the wrong order: the agent ships compiled into local-ai, and the standalone binary is the second reason to care rather than the first. Leads with `local-ai chat`, keeps nib as the SSH-anywhere story, and adds a docs CTA pointing at the new Terminal agent page. id="nib" is left alone because localai.io/#nib is linked from outside. The two credits on the demo clip named nib as the thing that drove the machine; they now credit the agent in LocalAI, which is the same agent. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * website: fix the exit keys, the plugin warning, and the redirect gap Three claims on the chat-agent pages that the code does not back. try-it-out told readers to press Ctrl+D. nib has no Ctrl+D handler: the full-screen interface quits on Esc or Ctrl+C, and Ctrl+D is only an exit in --cli, where it arrives as ordinary tty EOF. That sentence had replaced the removed /exit and /quit text, so the page was left with no working way to leave a session. Document both modes, since they differ. The plugin warning said nothing tells you the install stopped short. It does: the command prints that the plugin was left disabled. What it does not do is say so in its exit code, which is 0 either way. That is the part a script cannot work around, and it is the reason to pass --yes. Overstating it in the paragraph that gives the advice only makes the advice easier to dismiss. Redirecting stdout no longer refuses; the interface goes to /dev/tty and only the yanked command reaches the file. It is what lets the Ctrl+Space widget capture a command at all, since a redirect and out=$(...) are the same thing to the stream gate. It was documented nowhere. A non-terminal stdin is still refused, and the new text says which of the two it is. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): make the CLI flags outrank the agent config file local-ai chat routed --endpoint, --model, --api-key, --trace-dir and --yolo through nib's app.Options.Defaults. Defaults are seeds: they sit beneath the config file, so the file silently undoes them. That made the flags accepted and inert, and not in an edge case, since EnsureStateDir writes base_url on the first run and the interactive picker writes model, so from the second run on the file carried a value for both. Observed against a live server: with base_url: http://127.0.0.1:9999/v1 in the config and --endpoint http://127.0.0.1:8080 on the command line, the probe hit 8080 and every agent turn posted to 9999. With model: gemma-4-e2b-it-qat-q4_0 in the config, --model lfm2.5-8b-a1b was ignored on the wire. nib v0.6.0 adds app.Options.Overrides, applied above the config file and above the bare environment block. Move the whole block there: all five values are decisions this invocation already made on the user's behalf, and a flag the config file can undo is not a flag. Nothing is left in Defaults, because LocalAI's one genuine seed, the initial base_url, is written into the config file by EnsureStateDir rather than handed to nib. Two limits come with the channel and are documented on agentOptions rather than worked around. An override can only raise a field, since nib cannot tell "set to the zero value" from "not set", so --yolo can turn approval off but nothing on the command line turns it back on over an approval_mode: auto in the file. And nib's own NIB_TRACE_DIR and NIB_YOLO are resolved after the config load and still outrank these, deliberately, upstream. The existing spec pinned that the right values reach app.Options, which they always did, which is exactly why it could not see nib discarding them. The new specs resolve the config the way app.Run resolves it, against a real config file that disagrees with every flag, and one asserts Defaults stays empty. docs/content/features/terminal-agent.md already documented --model as winning over the saved model; that was false before this change and is true now, so no docs edit was needed. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): document intentional config file read Assisted-by: Codex:gpt-5 [gosec] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
18 KiB
18 KiB
+++ disableToc = false title = "CLI Reference" weight = 25 url = '/reference/cli-reference' +++
Complete reference for all LocalAI command-line interface (CLI) parameters and environment variables.
Note: All CLI flags can also be set via environment variables. Environment variables take precedence over CLI flags. See [.env files]({{%relref "advanced/advanced-usage#env-files" %}}) for configuration file support.
Global Flags
| Parameter | Default | Description | Environment Variable |
|---|---|---|---|
-h, --help |
Show context-sensitive help | ||
--log-level |
info |
Set the level of logs to output [error,warn,info,debug,trace] | $LOCALAI_LOG_LEVEL |
--debug |
false |
DEPRECATED - Use --log-level=debug instead. Enable debug logging |
$LOCALAI_DEBUG, $DEBUG |
Storage Flags
| Parameter | Default | Description | Environment Variable |
|---|---|---|---|
--models-path |
BASEPATH/models |
Path containing models used for inferencing | $LOCALAI_MODELS_PATH, $MODELS_PATH |
--data-path |
BASEPATH/data |
Path for persistent data (collectiondb, agent state, tasks, jobs). Separates mutable data from configuration | $LOCALAI_DATA_PATH |
--generated-content-path |
TMPDIR/localai-UID/generated/content |
Location for assets generated by backends (e.g. stablediffusion, images, audio, videos). Defaults under the OS temp dir ($TMPDIR, falling back to /tmp), scoped to the current user's UID so accounts sharing a host never collide. |
$LOCALAI_GENERATED_CONTENT_PATH, $GENERATED_CONTENT_PATH |
--upload-path |
TMPDIR/localai-UID/upload |
Path to store uploads from files API. Defaults under the OS temp dir ($TMPDIR, falling back to /tmp), scoped to the current user's UID. |
$LOCALAI_UPLOAD_PATH, $UPLOAD_PATH |
--localai-config-dir |
BASEPATH/configuration |
Directory for dynamic loading of certain configuration files (currently runtime_settings.json, api_keys.json, and external_backends.json). See [Runtime Settings]({{%relref "features/runtime-settings" %}}) for web-based configuration. | $LOCALAI_CONFIG_DIR |
--localai-config-dir-poll-interval |
Time duration to poll the LocalAI Config Dir if your system has broken fsnotify events (example: 1m) |
$LOCALAI_CONFIG_DIR_POLL_INTERVAL |
|
--models-config-file |
YAML file containing a list of model backend configs (alias: --config-file) |
$LOCALAI_MODELS_CONFIG_FILE, $CONFIG_FILE |
Backend Flags
| Parameter | Default | Description | Environment Variable |
|---|---|---|---|
--backends-path |
BASEPATH/backends |
Path containing backends used for inferencing | $LOCALAI_BACKENDS_PATH, $BACKENDS_PATH |
--backends-system-path |
/var/lib/local-ai/backends |
Path containing system backends used for inferencing | $LOCALAI_BACKENDS_SYSTEM_PATH, $BACKEND_SYSTEM_PATH |
--external-backends |
A list of external backends to load from gallery on boot | $LOCALAI_EXTERNAL_BACKENDS, $EXTERNAL_BACKENDS |
|
--external-grpc-backends |
A list of external gRPC backends (format: BACKEND_NAME:URI) |
$LOCALAI_EXTERNAL_GRPC_BACKENDS, $EXTERNAL_GRPC_BACKENDS |
|
--backend-galleries |
JSON list of backend galleries | $LOCALAI_BACKEND_GALLERIES, $BACKEND_GALLERIES |
|
--autoload-backend-galleries |
true |
Automatically load backend galleries on startup | $LOCALAI_AUTOLOAD_BACKEND_GALLERIES, $AUTOLOAD_BACKEND_GALLERIES |
--max-active-backends |
0 |
Maximum number of active backends (loaded models). When exceeded, the least recently used model is evicted. Set to 0 for unlimited, 1 for single-backend mode |
$LOCALAI_MAX_ACTIVE_BACKENDS, $MAX_ACTIVE_BACKENDS |
--single-active-backend |
false |
DEPRECATED - Use --max-active-backends=1 instead. Allow only one backend to be run at a time |
$LOCALAI_SINGLE_ACTIVE_BACKEND, $SINGLE_ACTIVE_BACKEND |
--preload-backend-only |
false |
Do not launch the API services, only the preloaded models/backends are started (useful for multi-node setups) | $LOCALAI_PRELOAD_BACKEND_ONLY, $PRELOAD_BACKEND_ONLY |
--enable-watchdog-idle |
false |
Enable watchdog for stopping backends that are idle longer than the watchdog-idle-timeout | $LOCALAI_WATCHDOG_IDLE, $WATCHDOG_IDLE |
--watchdog-idle-timeout |
15m |
Threshold beyond which an idle backend should be stopped | $LOCALAI_WATCHDOG_IDLE_TIMEOUT, $WATCHDOG_IDLE_TIMEOUT |
--enable-watchdog-busy |
false |
Enable watchdog for stopping backends that are busy longer than the watchdog-busy-timeout | $LOCALAI_WATCHDOG_BUSY, $WATCHDOG_BUSY |
--watchdog-busy-timeout |
5m |
Threshold beyond which a busy backend should be stopped | $LOCALAI_WATCHDOG_BUSY_TIMEOUT, $WATCHDOG_BUSY_TIMEOUT |
--watchdog-interval |
500ms |
Interval between watchdog checks (e.g., 500ms, 5s, 1m) |
$LOCALAI_WATCHDOG_INTERVAL, $WATCHDOG_INTERVAL |
--force-eviction-when-busy |
false |
Force eviction even when models have active API calls (default: false for safety). Warning: Enabling this can interrupt active requests | $LOCALAI_FORCE_EVICTION_WHEN_BUSY, $FORCE_EVICTION_WHEN_BUSY |
--lru-eviction-max-retries |
30 |
Maximum number of retries when waiting for busy models to become idle before eviction | $LOCALAI_LRU_EVICTION_MAX_RETRIES, $LRU_EVICTION_MAX_RETRIES |
--lru-eviction-retry-interval |
1s |
Interval between retries when waiting for busy models to become idle (e.g., 1s, 2s) |
$LOCALAI_LRU_EVICTION_RETRY_INTERVAL, $LRU_EVICTION_RETRY_INTERVAL |
--model-load-failure-cooldown |
10s |
After a model load fails, refuse new load attempts for that model for this long (HTTP 503 + Retry-After) so a client polling a broken model doesn't respawn a crashing backend every request. Doubles per consecutive failure up to 5m; reset on success. 0 disables |
$LOCALAI_MODEL_LOAD_FAILURE_COOLDOWN, $MODEL_LOAD_FAILURE_COOLDOWN |
For more information on VRAM management, see [VRAM and Memory Management]({{%relref "advanced/vram-management" %}}).
Models Flags
| Parameter | Default | Description | Environment Variable |
|---|---|---|---|
--galleries |
JSON list of galleries | $LOCALAI_GALLERIES, $GALLERIES |
|
--autoload-galleries |
true |
Automatically load galleries on startup | $LOCALAI_AUTOLOAD_GALLERIES, $AUTOLOAD_GALLERIES |
--preload-models |
A list of models to apply in JSON at start | $LOCALAI_PRELOAD_MODELS, $PRELOAD_MODELS |
|
--models |
A list of model configuration URLs to load | $LOCALAI_MODELS, $MODELS |
|
--preload-models-config |
A list of models to apply at startup. Path to a YAML config file | $LOCALAI_PRELOAD_MODELS_CONFIG, $PRELOAD_MODELS_CONFIG |
|
--load-to-memory |
A list of models to load into memory at startup | $LOCALAI_LOAD_TO_MEMORY, $LOAD_TO_MEMORY |
Note: You can also pass model configuration URLs as positional arguments:
local-ai run MODEL_URL1 MODEL_URL2 ...
Performance Flags
| Parameter | Default | Description | Environment Variable |
|---|---|---|---|
--f16 |
false |
Enable GPU acceleration | $LOCALAI_F16, $F16 |
-t, --threads |
Number of threads used for parallel computation. Usage of the number of physical cores in the system is suggested | $LOCALAI_THREADS, $THREADS |
|
--context-size |
Default context size for models (-1 = each model's full trained context from GGUF metadata) |
$LOCALAI_CONTEXT_SIZE, $CONTEXT_SIZE |
API Flags
| Parameter | Default | Description | Environment Variable |
|---|---|---|---|
--address |
:8080 |
Bind address for the API server | $LOCALAI_ADDRESS, $ADDRESS |
--cors |
false |
Enable CORS (Cross-Origin Resource Sharing) | $LOCALAI_CORS, $CORS |
--cors-allow-origins |
Comma-separated list of allowed CORS origins | $LOCALAI_CORS_ALLOW_ORIGINS, $CORS_ALLOW_ORIGINS |
|
--csrf |
false |
Enable Fiber CSRF middleware | $LOCALAI_CSRF |
--disable-http-compression |
false |
Disable gzip compression of HTTP responses. Compression is enabled by default; streaming endpoints (streaming chat completions, SSE bridges, WebSocket upgrades) and already-compressed formats are never compressed | $LOCALAI_DISABLE_HTTP_COMPRESSION |
--http-compression-min-length |
1024 |
Minimum response size in bytes before gzip compression is applied. Smaller responses are sent as-is because the gzip envelope would outweigh the saving | $LOCALAI_HTTP_COMPRESSION_MIN_LENGTH |
--upload-limit |
15 |
Default upload-limit in MB | $LOCALAI_UPLOAD_LIMIT, $UPLOAD_LIMIT |
--api-keys |
List of API Keys to enable API authentication. When this is set, all requests must be authenticated with one of these API keys | $LOCALAI_API_KEY, $API_KEY |
|
--disable-webui |
false |
Disables the web user interface. When set to true, the server will only expose API endpoints without serving the web interface | $LOCALAI_DISABLE_WEBUI, $DISABLE_WEBUI |
--disable-runtime-settings |
false |
Disables the runtime settings feature. When set to true, the server will not load runtime settings from the runtime_settings.json file and the settings web interface will be disabled |
$LOCALAI_DISABLE_RUNTIME_SETTINGS, $DISABLE_RUNTIME_SETTINGS |
--disable-gallery-endpoint |
false |
Disable the gallery endpoints | $LOCALAI_DISABLE_GALLERY_ENDPOINT, $DISABLE_GALLERY_ENDPOINT |
--disable-metrics-endpoint |
false |
Disable the /metrics endpoint |
$LOCALAI_DISABLE_METRICS_ENDPOINT, $DISABLE_METRICS_ENDPOINT |
--machine-tag |
If not empty, add that string to Machine-Tag header in each response. Useful to track response from different machines using multiple P2P federated nodes | $LOCALAI_MACHINE_TAG, $MACHINE_TAG |
Hardening Flags
| Parameter | Default | Description | Environment Variable |
|---|---|---|---|
--disable-predownload-scan |
false |
If true, disables the best-effort security scanner before downloading any files | $LOCALAI_DISABLE_PREDOWNLOAD_SCAN |
--opaque-errors |
false |
If true, all error responses are replaced with blank 500 errors. This is intended only for hardening against information leaks and is normally not recommended | $LOCALAI_OPAQUE_ERRORS |
--use-subtle-key-comparison |
false |
If true, API Key validation comparisons will be performed using constant-time comparisons rather than simple equality. This trades off performance on each request for resilience against timing attacks | $LOCALAI_SUBTLE_KEY_COMPARISON |
--disable-api-key-requirement-for-http-get |
false |
If true, a valid API key is not required to issue GET requests to portions of the web UI. This should only be enabled in secure testing environments | $LOCALAI_DISABLE_API_KEY_REQUIREMENT_FOR_HTTP_GET |
--http-get-exempted-endpoints |
^/$,^/app(/.*)?$,^/browse(/.*)?$,^/login/?$,^/explorer/?$,^/assets/.*$,^/static/.*$,^/swagger.*$ |
If --disable-api-key-requirement-for-http-get is overridden to true, this is the list of endpoints to exempt. Only adjust this in case of a security incident or as a result of a personal security posture review |
$LOCALAI_HTTP_GET_EXEMPTED_ENDPOINTS |
Authentication Flags
| Parameter | Default | Description | Environment Variable |
|---|---|---|---|
--auth-enabled |
false |
Enable user authentication and authorization | $LOCALAI_AUTH |
--auth-database-url |
{DataPath}/database.db |
Database URL for auth - postgres://... for PostgreSQL, or a file path for SQLite |
$LOCALAI_AUTH_DATABASE_URL, $DATABASE_URL |
--github-client-id |
GitHub OAuth App Client ID (auto-enables auth when set) | $GITHUB_CLIENT_ID |
|
--github-client-secret |
GitHub OAuth App Client Secret | $GITHUB_CLIENT_SECRET |
|
--oidc-issuer |
OIDC issuer URL for auto-discovery | $LOCALAI_OIDC_ISSUER |
|
--oidc-client-id |
OIDC Client ID (auto-enables auth when set) | $LOCALAI_OIDC_CLIENT_ID |
|
--oidc-client-secret |
OIDC Client Secret | $LOCALAI_OIDC_CLIENT_SECRET |
|
--auth-base-url |
Base URL for OAuth callbacks (e.g. http://localhost:8080) |
$LOCALAI_BASE_URL |
|
--auth-admin-email |
Email address to auto-promote to admin role on login | $LOCALAI_ADMIN_EMAIL |
|
--auth-registration-mode |
open |
Registration mode: open, approval, or invite |
$LOCALAI_REGISTRATION_MODE |
--disable-local-auth |
false |
Disable local email/password registration and login (for OAuth/OIDC-only setups) | $LOCALAI_DISABLE_LOCAL_AUTH |
See [Authentication & Authorization]({{%relref "features/authentication" %}}) for full documentation.
Chat Flags
Use local-ai chat to run the built-in terminal agent against a LocalAI server.
The agent can run shell commands, delegate to sub-agents, and use MCP tools. Read-only
calls run on their own; everything else goes through an approval prompt you answer.
See [Terminal agent]({{% relref "features/terminal-agent" %}}) for the full feature page.
| Parameter | Default | Description | Environment Variable |
|---|---|---|---|
--endpoint |
http://127.0.0.1:8080 |
LocalAI server endpoint. The /v1 path is added automatically when omitted. |
$LOCALAI_CHAT_ENDPOINT |
--model |
Model to use. Defaults to the only model the server offers, or asks when there are several. | ||
--api-key |
API key to use when the LocalAI server requires authentication. | $LOCALAI_API_KEY, $API_KEY |
|
--config-dir |
~/.config/localai/chat |
Directory holding the agent's config, plugins, and skills. | $LOCALAI_CHAT_CONFIG_DIR |
--trace-dir |
Write a session LLM trace (NDJSON) to this directory. | $LOCALAI_CHAT_TRACE_DIR |
|
--cli |
false |
Plain CLI mode instead of the full-screen interface. Pipe stdin for one-shot use. | |
--tui |
false |
Force the full-screen interface. | |
--height |
Run as an inline drop-down of this height, e.g. 40%. |
||
--tmux / --no-tmux |
Control the tmux split. | ||
--init |
Print the shell integration script for Ctrl+Space (zsh, bash, or fish). |
||
--yolo |
false |
Auto-approve every tool call. Use with care. | $LOCALAI_CHAT_YOLO |
The agent's own subcommands are reached by passing them through. LocalAI's flags must come first, because everything after the first positional argument is forwarded verbatim:
local-ai chat plugin install https://github.com/user/plugin
local-ai chat skill list
local-ai chat mcp add my-server -- npx -y @modelcontextprotocol/server-filesystem /tmp
local-ai chat --config-dir /srv/agent plugin list # flags first
{{% notice warning %}}
In a script, pass --yes to those subcommands. Without it, a non-interactive
plugin install installs the plugin, leaves it disabled, and still exits 0.
{{% /notice %}}
Summon the agent from any shell prompt with Ctrl+Space:
echo 'eval "$(local-ai chat --init zsh)"' >> ~/.zshrc
Inside a session:
/modelslists the models the server offers, marking the current one./model <name>switches model, keeping the conversation./compactsummarizes the conversation so far to free up context./skill <name>,/agent <name> <task>,/attach <file>,/goal <text>.
If no server is reachable, the agent offers to start one for the session
(interactive terminals only) and otherwise points you at local-ai run.
Piped use needs --cli, and exits 0 when it answers:
echo "what is 2+2" | local-ai chat --cli
Read-only tools still run in a piped session. A tool call that is not read-only has
nobody to approve it, so it is denied and the session exits 3, a code chosen to be
distinct from the 1 a failure reports.
P2P Flags
| Parameter | Default | Description | Environment Variable |
|---|---|---|---|
--p2p |
false |
Enable P2P mode | $LOCALAI_P2P, $P2P |
--p2p-dht-interval |
360 |
Interval for DHT refresh (used during token generation) | $LOCALAI_P2P_DHT_INTERVAL, $P2P_DHT_INTERVAL |
--p2p-otp-interval |
9000 |
Interval for OTP refresh (used during token generation) | $LOCALAI_P2P_OTP_INTERVAL, $P2P_OTP_INTERVAL |
--p2ptoken |
Token for P2P mode (optional) | $LOCALAI_P2P_TOKEN, $P2P_TOKEN, $TOKEN |
|
--p2p-network-id |
Network ID for P2P mode, can be set arbitrarily by the user for grouping a set of instances | $LOCALAI_P2P_NETWORK_ID, $P2P_NETWORK_ID |
|
--federated |
false |
Enable federated instance | $LOCALAI_FEDERATED, $FEDERATED |
Other Commands
LocalAI supports several subcommands beyond run:
local-ai models- Manage LocalAI models and definitionslocal-ai backends- Manage LocalAI backends and definitionslocal-ai chat- Run the built-in terminal agent against a LocalAI serverlocal-ai tts- Convert text to speechlocal-ai sound-generation- Generate audio files from text or audiolocal-ai transcript- Convert audio to textlocal-ai agent- Run agents standalone without the full LocalAI serverlocal-ai mcp-server- Run the LocalAI admin tool surface as a stdio MCP server (controls a remote LocalAI instance over HTTP)local-ai worker- Start a worker for distributed mode (generic, backend-agnostic)local-ai p2p-worker- Run workers to distribute workload via p2p (llama.cpp-only)local-ai agent-worker- Start an agent worker for distributed mode (executes agent chats via NATS)local-ai util- Utility commandslocal-ai explorer- Run P2P explorerlocal-ai federated- Run LocalAI in federated modelocal-ai completion- Generate shell completion scripts for bash, zsh, or fish
Use local-ai <command> --help for more information on each command.
Examples
Basic Usage
./local-ai run
./local-ai run --models-path /path/to/models --address :9090
./local-ai run --f16
Environment Variables
export LOCALAI_MODELS_PATH=/path/to/models
export LOCALAI_ADDRESS=:9090
export LOCALAI_F16=true
./local-ai run
Advanced Configuration
./local-ai run \
--models model1.yaml model2.yaml \
--enable-watchdog-idle \
--watchdog-idle-timeout=10m \
--p2p \
--federated
Related Documentation
- See [Advanced Usage]({{%relref "advanced/advanced-usage" %}}) for configuration examples
- See [VRAM and Memory Management]({{%relref "advanced/vram-management" %}}) for memory management options