LocalAI/docs/content/features/model-failover.md

+++ disableToc = false title = "Model Failover" weight = 15 url = "/features/model-failover/" +++

A failover chain is a model name that is served by an ordered list of other models. LocalAI sends each request to the first healthy target. When a target fails, the request moves to the next target, and later requests stay there until the first target has recovered.

Use it to serve a model from a remote LocalAI or another OpenAI-compatible provider, and to fall back to a local model when the remote one is down.

Declaring a chain

name: assistant-llm
failover:
  targets:
    - model: argus-llm          # for example a localai-proxy or cloud-proxy model
    - model: gemma-local
      warm: true                # keep it loaded

Clients call assistant-llm. Each target is a normal model config. A chain has no backend and no parameters.model.

Optional settings, with their defaults:

failover:
  probe:
    interval: 15s     # how often an idle target is checked
    timeout: 5s
  trip:
    errors: 1         # failures within the window that mark a target down
    window: 30s
  recovery:
    probes: 3         # test requests a target must pass before it is used again
    min_dwell: 60s    # minimum time on a lower target before moving back

Rules:

• A chain needs at least 2 targets. A target can be an alias, but not another chain.
• A chain cannot also set alias or backend.
• Responses name the chain as the model. The X-LocalAI-Served-Model header names the target that served the request.
• A remote (localai-proxy or cloud-proxy) target receives its own model name, never the chain name: proxy.upstream_model, or the target name when upstream_model is empty. The health check looks for the same name.

How the target is chosen

• The active target is the first healthy target in the list.
• When a target fails, LocalAI marks it down and moves to the next target at once.
• LocalAI moves back to a higher target only when that target has passed recovery.probes test requests and the current target has been active for at least recovery.min_dwell. This stops an unstable upstream from moving traffic back and forth.
• When all targets are down, the chain is degraded. Each request still tries every target in order.

Retry inside a request

When a target fails before the response starts, LocalAI sends the same request to the next target. The client does not see the failure.

• LocalAI does not retry after the first byte of a response is sent (for example after the first streamed token). The request fails, the target is marked down, and the next request uses the next target.
• A target that is at its concurrency limit (an admission rejection) or that is disabled is skipped for that request without being marked down.
• LocalAI does not retry client errors (4xx), such as a prompt that is too long, because the next target would reject it too. A 4xx counts neither as a success nor as a failure for the target.
• Request bodies larger than 32 MiB are not retried.

When the primary did not serve the request, the response has the header X-LocalAI-Failover: fallback, or X-LocalAI-Failover: degraded when all targets were down.

Health checks

Target	Regular check	Check before moving back
Remote (localai-proxy, cloud-proxy)	GET /v1/models on the upstream lists the model	one small real request, for example a 1-token completion
Local, warm: true	the backend answers a health check. A check never loads the model: while it is not loaded, the check passes and real requests judge it	one small real request. While the model is not loaded, the target is used again after min_dwell
Local, not warm	none: judged only by real requests; it is never loaded only to check it	none: the target is used again after min_dwell

A request that succeeds counts as a check, so a busy target is almost never probed.

When a target is in more than one chain, its check settings come from the first of those chains in name order.

Warm targets

warm: true loads a local target at startup and protects it from idle and LRU eviction, so a switch does not wait for the model to load. Warm targets count toward the active backend limit (--max-active-backends) like any pinned model: LocalAI never evicts them to make room, and if they fill the limit, a new model still loads rather than being blocked.

warm applies only to local targets. On a remote (localai-proxy or cloud-proxy) target it has no effect, and LocalAI logs a warning when it loads the chain.

Realtime pipelines

A pipeline stage can name a chain:

name: assistant
pipeline:
  vad: silero-vad
  transcription: whisper-chain
  llm: assistant-llm
  tts: voice-chain

LocalAI resolves the chain for every call of the stage, in full pipelines and in transcription-only and sound-detection-only sessions. When a chain switches, the session stays open and keeps its conversation. The next turn uses the new target.

The session receives a localai.model.failover event for each chain stage when it starts (reason: initial) and each time a chain switches:

{"type":"localai.model.failover","chain":"assistant-llm","stage":"llm",
 "from":"argus-llm","to":"gemma-local","state":"fallback","reason":"trip"}

Example: stages on a remote LocalAI

This pipeline runs its transcription, LLM and TTS stages on a remote LocalAI (argus) through [localai-proxy]({{% relref "operations/cloud-proxy" %}}) models, and uses local models when the remote instance is down. Each stage has its own chain, so one stage can fail over while the others stay remote.

# Remote targets: each one names the model on the upstream LocalAI.
name: argus-stt
backend: localai-proxy
known_usecases: [transcript]
options:
  - realtime_pipeline:asr-pipeline   # upstream pipeline for live transcription
proxy:
  upstream_url: http://argus.lan:8080
  upstream_model: parakeet
---
name: argus-llm
backend: localai-proxy
known_usecases: [chat]
proxy:
  upstream_url: http://argus.lan:8080
  upstream_model: gemma-3-12b
---
name: argus-tts
backend: localai-proxy
known_usecases: [tts]
proxy:
  upstream_url: http://argus.lan:8080
  upstream_model: kokoro
---
# One chain per stage, remote first, local second.
name: stt-chain
failover:
  targets: [{model: argus-stt}, {model: whisper-local, warm: true}]
---
name: llm-chain
failover:
  targets: [{model: argus-llm}, {model: gemma-local, warm: true}]
---
name: tts-chain
failover:
  targets: [{model: argus-tts}, {model: piper-local}]
---
name: assistant
pipeline:
  vad: silero-vad
  transcription: stt-chain
  llm: llm-chain
  tts: tts-chain

The example shows the configs as one YAML stream; put each config in its own file in the models directory. When argus stops answering, the next call of each stage fails over to the local model and the session receives a localai.model.failover event for that stage. A remote target that does not support a call (it returns Unimplemented) is skipped for that call and is not marked down.

Limits:

• After a session.update that changes the pipeline, localai.model.failover events keep describing the chains from session start.
• A chain used as a router candidate, or as the classifier-mode scoring model, is not resolved per call.

Watching failover

• GET /api/failover lists every chain, its active target and the state of each target.
• GET /api/failover/{chain} returns one chain.
• GET /api/failover/events is a server-sent event stream. The first event is snapshot with the full state. Then chain.switched and target.state events follow.
• Metrics: localai_failover_switches_total{chain,from,to,reason} and localai_failover_target_up{target}.
• With tracing on, each skipped target appears in the Traces view with the error that made LocalAI skip it.

Pinning a target

An admin can force a chain to one target, for example during maintenance:

curl -X POST http://localhost:8080/api/failover/assistant-llm/pin \
  -H 'Content-Type: application/json' -d '{"target":"gemma-local"}'
curl -X DELETE http://localhost:8080/api/failover/assistant-llm/pin

While a chain is pinned, only the pinned target serves it. Health checks continue. On a single LocalAI instance, a restart removes the pin. In distributed mode, pins persist.

Assistant and MCP

The LocalAI Assistant and local-ai mcp-server offer list_failover_chains, pin_failover_target and unpin_failover_target. Create and edit chains with the model config tools, like any other model.

Distributed mode

In [distributed mode]({{%relref "features/distributed-mode" %}}), all frontends share one failover state:

• Pins apply to the whole cluster. LocalAI stores them in the database, so they persist across restarts. A pin set on one frontend applies on all.
• Target health and the active target of each chain are shared over NATS, so all frontends converge on the same target for a chain.
• One frontend, the probe leader, runs the health checks, decides fail-over and fail-back, and loads warm targets. The leader holds a PostgreSQL advisory lock and keeps it until it stops or its database connection fails. Then another frontend takes the lock and becomes the leader: immediately when the leader shuts down or its process exits, and within about 30 seconds when the leader's host or network fails.
• Warm targets stay loaded on the workers. The router and the replica reconciler treat them like pinned models and do not evict them.
• A frontend that starts late gets the current state within 10 seconds, because the leader sends its full state again every 10 seconds.
• If PostgreSQL is not available, no frontend holds the lock. Health checks and fail-back stop until the database is back. Requests still fail over to the next target when a target fails during the request.
• If a frontend cannot start the shared state, it logs an error and manages failover alone, as a single LocalAI instance does.

Limits

• Chains do not nest.
• See also [model aliases]({{%relref "features/model-aliases" %}}) and the [realtime API]({{%relref "features/openai-realtime" %}}).