LocalAI/docs/content/features/distributed-mode.md

+++ disableToc = false title = "Distributed Mode" weight = 71 url = "/features/distributed-mode/" +++

Distributed mode enables horizontal scaling of LocalAI across multiple machines using PostgreSQL for state, node registry and cross-replica fan-out. No message bus is needed: a deployment runs PostgreSQL and the frontends' own HTTP listener, and every worker is reached over the tunnel it dials outward. Unlike the [P2P/federation approach]({{% relref "features/distributed_inferencing" %}}), distributed mode is designed for production deployments and Kubernetes environments where you need centralized management, health monitoring, and deterministic routing.

{{% notice note %}} Distributed mode requires authentication enabled with a PostgreSQL database - SQLite is not supported. This is because the node registry, job store, and other distributed state are stored in PostgreSQL tables. {{% /notice %}}

Architecture Overview

Frontends are stateless LocalAI instances that receive API requests and route them to worker nodes via the SmartRouter. All frontends share state through PostgreSQL, which also carries every cross-replica event they broadcast.

Workers are generic processes that self-register with a frontend. They don't have a fixed backend type - the SmartRouter dynamically installs the required backend by calling the worker's backend.install control route through its tunnel when a model request arrives.

Scheduling Algorithm

The SmartRouter uses idle-first scheduling with preemptive eviction:

1. If the model is already loaded on a node → use it (per-model gRPC address)
2. Drop any node without room to store the model on its models filesystem (see Disk headroom)
3. If no node has the model → prefer nodes with enough free VRAM
4. Fall back to idle nodes (zero models), then least-loaded nodes
5. If no node has capacity → evict the least-recently-used model with zero in-flight requests to free a node
6. If all models are busy → wait (with timeout) for a model to become idle, then evict
7. POST /v1/control/backend/install through the worker's tunnel with backend name + model ID → worker starts a new gRPC process on a dynamic port
8. SmartRouter calls gRPC LoadModel on the model-specific port, records in DB

Each model gets its own gRPC backend process, so a single worker can serve multiple models simultaneously (e.g., a chat model and an embedding model).

Prerequisites

• PostgreSQL (with pgvector extension recommended for RAG) - used for node registry, job store, auth, and shared state
  • Each frontend replica holds one extra PostgreSQL session beyond its connection pool, pinned for the life of the process, and creates a bus_messages table. Both belong to the broadcast carrier that carries every cross-replica fan-out, including the four state.*.delta families (see Cross-replica in-memory state). Size max_connections for one additional session per frontend replica.
  • That session reports an application_name of localai_pgbus_<id>, so SELECT count(*) FROM pg_stat_activity WHERE application_name LIKE 'localai_pgbus_%' counts the replicas currently listening. If the carrier loses its session it redials and re-registers on its own; a broadcast published while it was down is not replayed, which is why nothing that must survive a gap is carried by a broadcast alone.
  • bus_messages holds only broadcasts too large for a PostgreSQL notification, and every replica retires rows older than ten minutes. The table is a spill buffer, not a log: it is not a place to read past events from.
• No message broker. Do not deploy one. Everything a frontend broadcasts travels on the PostgreSQL the deployment already runs; every verb a frontend addresses to a worker, including an agent cancel, is an HTTP route on that worker's own tunnel. If you are upgrading from a release that ran one, see Migrating off the message broker: your existing command lines keep working and the broker can be shut down.
• All services must be on the same network (or reachable via configured URLs)

Quick Start with Docker Compose

The easiest way to try distributed mode locally is with the provided Docker Compose file:

docker compose -f docker-compose.distributed.yaml up

This starts PostgreSQL, a LocalAI frontend, one worker node and one agent worker. Those four services are the whole deployment: there is no broker in the file and none to add. When you send an inference request, the SmartRouter automatically installs the needed backend on the worker and loads the model. See the file for details on adding GPU support, shared volumes, and additional workers.

{{% notice tip %}} Use docker-compose.distributed.yaml for quick local testing. For production, deploy PostgreSQL as a managed service and run frontends/workers on separate hosts. There is no message bus to deploy. {{% /notice %}}

Frontend Configuration

The frontend is a standard LocalAI instance with distributed mode enabled. These flags are added to the local-ai run command:

Flag	Env Var	Default	Description
--distributed	LOCALAI_DISTRIBUTED	false	Enable distributed mode
--instance-id	LOCALAI_INSTANCE_ID	auto UUID	Unique instance ID for this frontend
--distributed-advertise-addr	LOCALAI_DISTRIBUTED_ADVERTISE_ADDR	(derived)	host:port the other frontend replicas dial to reach this one. See Replica peer links.
--registration-token	LOCALAI_REGISTRATION_TOKEN	(empty)	Token that workers must provide to register
--registration-require-auth	LOCALAI_REGISTRATION_REQUIRE_AUTH	false	Fail startup when distributed mode is enabled but the registration token is empty (node endpoints and worker file-transfer would otherwise be unauthenticated)
--distributed-require-auth	LOCALAI_DISTRIBUTED_REQUIRE_AUTH	false	Umbrella switch. Implies --registration-require-auth, which is what guards registration, the worker control planes and file transfer. Set this in production instead of the granular flags.
--auto-approve-nodes	LOCALAI_AUTO_APPROVE_NODES	false	Auto-approve new worker nodes (skip admin approval)
--distributed-shared-models	LOCALAI_DISTRIBUTED_SHARED_MODELS	false	Assert that every node mounts the same models directory at the same path (a shared volume). When true, the router skips file staging entirely and workers load models directly from the shared path instead of re-downloading them. See Shared models directory.
--distributed-disk-headroom-check	LOCALAI_DISTRIBUTED_DISK_HEADROOM_CHECK	true	Reject worker nodes that lack free space to store the model, at scheduling time rather than partway through staging. When false, node selection ignores free disk; the check still runs and warns when it would have rejected every node. Also toggleable at runtime via the distributed_disk_headroom_check setting. See Disk headroom.
--auth	LOCALAI_AUTH	false	Must be true for distributed mode
--auth-database-url	LOCALAI_AUTH_DATABASE_URL	(required)	PostgreSQL connection URL
--backend-install-timeout	LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT	15m	How long the frontend waits for a worker to acknowledge a backend install before considering the request stalled. Raise it when workers pull large backend images over slow links. If a worker takes longer than this, the operation shows as "still installing in background" in the admin UI and clears once the worker finishes.
--backend-upgrade-timeout	LOCALAI_NATS_BACKEND_UPGRADE_TIMEOUT	15m	Same as the install timeout, applied to backend upgrades (force-reinstall).
--model-load-timeout	LOCALAI_NATS_MODEL_LOAD_TIMEOUT	(derived from checkpoint size)	Pins the deadline for the LoadModel gRPC call the frontend issues to a worker. Leave it unset: by default the deadline is derived from the checkpoint's on-disk size (see below), which is what the worker actually spends its load time reading. Set it only to pin a specific budget — the value is then used verbatim, including when it is shorter than the derived one, so an operator who wants fast failure gets it.
(env only)	LOCALAI_MODEL_LOAD_WAIT	60s	How long an inference request waits for a model that is still cold-loading onto a worker before it is answered with 503, a Retry-After header and live staging progress. The request is served the moment the model becomes ready, so a model already most of the way staged needs no client retry. Set to 0 to wait as long as the load takes — only safe when no ingress or load balancer with an idle timeout sits in front. See Requests for a model that is still loading.
--worker-reconnect-grace	LOCALAI_WORKER_RECONNECT_GRACE	90s	How long a worker whose tunnel was lost is treated as reconnecting rather than gone. Only after this window may the scheduler stop placing work on that worker, clean up its rows and release its models. Set it below the worker's own reconnect worst case and you will condemn workers that are re-homing exactly as designed. Measured on the database clock, so every replica agrees. See A lost tunnel is a departure, not an absence.
--expose-node-header	LOCALAI_EXPOSE_NODE_HEADER	false	When enabled, inference responses carry an X-LocalAI-Node header with the ID of the worker node that served the request. Coverage spans the OpenAI-compatible endpoints (chat completions, completions, embeddings, audio transcriptions, audio speech / TTS, image generations, image inpainting), the Jina rerank endpoint (/v1/rerank), the VAD endpoints (/v1/vad, /vad), and the Anthropic Messages (/v1/messages) and Ollama (/api/chat, /api/generate, /api/embed) shims. Useful for debugging, observability and load-balancer attribution. Off by default: the node ID reveals internal cluster topology and should not be exposed on a public endpoint. Best-effort: under heavy concurrency for the same model across multiple replicas, the header may reflect a recent routing decision rather than this exact request's. Acceptable for observability and debugging.

The three LOCALAI_NATS_*_TIMEOUT names above are control-RPC budgets, not broker settings, and are still read and enforced. They carry that prefix only because they were introduced alongside the message bus that distributed mode used to require; renaming them would break every existing deployment for cosmetics. Do not delete them when you shut the broker down.

Replica peer links

Frontend replicas record themselves in an instances table and open direct links to each other, so that a request arriving at one replica can be served by state another replica holds. Each replica publishes one address for this, and every other replica dials it: it is the address peers use, which is not necessarily the address the process binds. A replica behind a Kubernetes Service, a load balancer or a NAT binds one and is reached at another.

When LOCALAI_DISTRIBUTED_ADVERTISE_ADDR is unset, the address is derived: LocalAI asks the kernel which local address routes to PostgreSQL, and pairs it with the port it serves on. Every replica reaches the same database, so that address is on a network they demonstrably share.

That only holds while the database is on another host. If PostgreSQL runs on the same host or pod (compose, single-node, a sidecar), the route to it is loopback, and advertising a loopback address would send every peer to itself. LocalAI refuses to guess in that case. It starts anyway - refusing would break every single-host deployment, which has no peers to be unreachable by - and logs an error at startup:

ERROR This replica is not registered in the cluster: no advertised address. Peers cannot reach it,
      any worker whose tunnel lands here will be unroutable from every other replica, and this
      replica cannot relay OUT either, because a peer link is authenticated by the credential an
      instance row publishes and this replica has no row

The replica keeps serving every request that reaches it directly. What it cannot do is be reached by another replica, and on a multi-replica deployment that is worse than it sounds: a worker whose tunnel lands on this replica is unroutable from every other replica, because the ownership lookup only accepts an owner that is registered and live. This replica serves that worker fine; the others answer requests for it with no route from this replica to that worker. Behind a round-robin load balancer with N replicas, that is (N-1)/N of the traffic for that worker.

Because that symptom looks like a broken worker and not a misconfigured frontend, the replica repeats itself every five minutes for as long as it runs, and names the workers it is currently costing:

ERROR This replica is not registered in the cluster and holds worker tunnels: those workers are
      unroutable from every OTHER replica, and requests for their models fail there with no route.
      The workers are healthy; this replica is invisible   workers=[node-a node-b] worker_count=2

If you are chasing a worker that answers on one replica and 5xxs on the others, grep the frontend logs for that line before looking at the worker. Until a worker's tunnel lands here the same line appears at WARN with no workers named, which is the same misconfiguration not yet costing anything.

A single-replica deployment is unaffected: it has no peers, and it holds every tunnel itself. Set the address explicitly to fix a multi-replica one:

environment:
  LOCALAI_DISTRIBUTED_ADVERTISE_ADDR: "10.0.1.7:8080"   # or the pod IP, service DNS name, etc.

The peer link is served at /api/cluster/peer. A replica that stops heartbeating for 30 seconds is dropped from the table by the others, along with the worker-connection rows it owned.

A peer proves which replica it is

The route checks two credentials, and a dial needs both.

Credential	Sent as	Says
LOCALAI_REGISTRATION_TOKEN	Authorization: Bearer <token>	the dialler belongs to this deployment
The replica's own peer credential	X-LocalAI-Peer-Token: <secret>	the dialler is the replica named in ?id=

Each replica mints its own peer credential at startup, publishes only its SHA-256 in the instances table beside its address, and never sends the plaintext anywhere but the peer dial itself. The receiving replica resolves ?id= to that row and compares. This is the same shape as the worker tunnel credential, in the stronger direction: a worker's credential is minted by the frontend and handed to it once, while a replica's never leaves the process that made it.

Nothing needs configuring and nothing needs rotating. A restart mints a new secret, and the same registration that republishes a replica's address republishes the hash beside it.

What that closes: holding LOCALAI_REGISTRATION_TOKEN - which every worker does - no longer lets its holder open a peer link as some other replica. It can therefore no longer relay through that link to every worker tunnel a replica owns, no longer displace a real replica's inbound link by declaring its id, and no longer point the per-session receive window (roughly 31 GiB of unread data per session) at a replica of its choosing. Only replicas registered in the instances table, each proving its own row, can open a peer link at all.

What it does not close: replica-to-replica traffic is not encrypted, so anything that can read the wire between two replicas can read a credential off it, exactly as it could read the registration token. Keep /api/cluster/peer on a network only your replicas reach, and keep LOCALAI_REGISTRATION_TOKEN per-deployment.

{{% notice note %}} A replica with no advertised address cannot peer in either direction. It has no row in the instances table, so it has no credential published, so peers refuse its dials as an unproven identity - on top of already being unreachable itself. The startup error above names this. Set LOCALAI_DISTRIBUTED_ADVERTISE_ADDR. {{% /notice %}}

{{% notice warning %}} A replica that presents no peer credential is refused, not waved through. There is no fallback to the shared token alone: an old replica and an attacker send exactly the same thing, so accepting one accepts the other. During a rolling frontend upgrade this means an old replica cannot open a peer link to an upgraded one, and each refusal is logged by the upgraded replica:

WARN Refusing a peer link: the dialling replica presented no peer credential. It is running a
     release from before per-replica peer identity, or it never registered a credential of its
     own. Upgrade it; this replica will not accept an unproven peer id   peer=<replica id>

The dialling side logs the matching A peer refused this replica's credentials. The window closes as each frontend restarts, and it is bounded by the same frontend-first rollout the tunnel already requires. A refused peer is an authorization failure, never node absence: nothing is rescheduled, nothing is reaped, and requests that cannot be relayed during the window fail with no route from this replica to that worker. {{% /notice %}}

Cross-replica in-memory state

Several features keep state in a frontend's process memory and surface it over the API: fine-tune jobs, quantization jobs, agent tasks and Open Responses metadata. A round-robin load balancer sends a follow-up request to any replica, so each of those maps is kept current on every replica by a broadcast.

Those four families travel on PostgreSQL. Each mutation is a NOTIFY on the database the deployment already runs, and each replica holds one LISTEN session for it. There is nothing to configure: the carrier uses the same database URL as --auth-database-url / LOCALAI_AUTH_DATABASE_URL.

Map	Subject
Fine-tune jobs	state.finetune-jobs.delta
Quantization jobs	state.quant-jobs.delta
Agent tasks	state.agent-tasks.delta and state.agent-tasks.<user_id>.delta
Open Responses metadata	state.responses-metadata.delta

Cross-replica caches

A frontend also keeps caches that live for the life of the process rather than for the life of a request: which gallery operations are in flight and how far along they are, which admin operations have been admitted, which model files are being staged onto a worker, and which replica already holds the KV/prefix cache for a prompt. Each of those is kept current on every replica by a broadcast, and every one of them is on PostgreSQL.

Family	Subject	What a peer does with it
Gallery progress	gallery.<op_id>.progress	Merges the status so /api/operations answers the same on any replica
Gallery cancel	gallery.<op_id>.cancel	Stops the install, on whichever replica is running it
Operation cache admit	gallery.opcache.start	Learns that an operation was admitted, and whether it is a backend install
Operation cache dismiss	gallery.opcache.end	Drops the operation from its own map
Model cache invalidation	cache.invalidate.models	Reloads the model config from disk, or prunes a deleted one
Backend cache invalidation	cache.invalidate.backends	Refreshes its upgrade-available cache
Staging progress	staging.<model_id>.progress	Mirrors a transfer it did not perform, so the progress bar does not flicker
Prefix-cache observation	prefixcache.observe	Learns which replica already holds the prefix for a prompt
Prefix-cache invalidation	prefixcache.invalidate	Stops routing to a replica that is gone

An invalidation that does not arrive must never read as a cache that is valid. The two cache.invalidate.* families and prefixcache.invalidate are therefore published like any other broadcast: one too large for a notification is written to bus_messages and read back by the peer, never dropped to save the write. A missed staging event ages the peer's mirrored row out after a minute rather than inventing a transfer, for the same reason.

Gallery progress spills, routinely. A progress event carries one entry per worker, so on a fleet of a few tens of nodes it is past the 8000-byte cap on every tick and travels as a row. That is the ordinary path, not an error.

A prefix-cache observation never spills. It carries one hash per prefix block, and the extractor caps a chain at 64 blocks, so the largest observation a frontend can publish is a few kilobytes and fits in the notification itself. That bound is checked at startup: a build that raised the cap past what a notification carries would put a table write and a read-back on the inference path for every request whose prefix changed, so the frontend refuses to start and says so rather than running slowly and quietly. Nothing here drops an observation to stay under the cap.

Skills and collections are NOT replicated

Agent skills and RAG collections are the two features whose state is not on this list, and they are absent from it deliberately rather than by omission. Neither has a cross-replica invalidation, and neither should be given one, because there is nothing coherent for an invalidation to say.

Both are backed by the frontend's own state directory:

State	Where it lives	What is shared
Skill content, skill resources, git-repo clones, the skill search index	<state dir>/skills, or <state dir>/users/<user id>/skills	nothing
Collection contents and its file list	<state dir>/collections/collection_<name>.json plus the assets directory	nothing
Skill name, description and source	skills_metadata in PostgreSQL	the row
Collection vectors, with the postgres vector engine	the vector table in PostgreSQL	the vectors

A frontend replica writes a skill or a collection to its OWN disk. No replica copies it, and no broadcast could: a peer told to drop a cache entry would re-read a directory that does not contain the change, so the invalidation would be a guaranteed no-op for skills and, for a postgres collection, worse than one, since re-deriving the collection on a replica that has no local index file would produce a collection that answers with an empty file list against a populated vector store. The cache is not what is missing here; the shared storage is.

What that means when you run more than one frontend replica:

• A skill created on one replica is listed on every replica, because the list comes from skills_metadata. Reading it, searching it, exporting it or fetching its resources works only on the replica that wrote it.
• A collection created on one replica is not listed, searched or uploaded to on any other replica.

Two deployments avoid it. Mount ONE ReadWriteMany volume as the state directory (LOCALAI_AGENT_POOL_STATE_DIR, else LOCALAI_DATA_PATH) on every frontend replica, so all replicas read and write the same files; or route /api/agents/skills* and /api/agents/collections* to a single replica. A frontend running distributed logs this limitation once at startup, so it is visible in a deployment that did neither.

Job and agent streams across replicas

The same carrier moves the traffic whose subscriber is an open HTTP response rather than a cache: a job's progress stream, its result, its cancel, an agent's SSE events, an agent cancel and an Open Responses cancel.

Family	Subject	Read by
Job progress	jobs.<job_id>.progress	GET /api/agent/jobs/{id}/progress on any replica, and the trace persister on every replica
Job result	jobs.<job_id>.result	The result persister on every replica
Job cancel	jobs.<job_id>.cancel	Every replica, so the one holding the run can stop it
Agent events	agent.<agent>.events.<user_id>	GET /api/agents/{name}/sse/distributed on any replica, and the observable persister
Open Responses cancel	responses.<response_id>.cancel	The replica holding the generation

An agent cancel is deliberately NOT on that list. It has to reach the agent WORKER running the execution, and an agent worker has no database, so it can never listen on PostgreSQL; it is a control verb on that worker's own tunnel instead. See Cancelling an agent run.

This is what lets a user watch a job or an agent on one frontend while the work runs against another. No broadcast on this list is the only path to anything durable. A job's terminal state is written to its row by the replica that claimed the work, before that claim is released, so a dropped result costs an open stream its promptness and never costs the job its answer: a stream that is still open re-reads the row and closes on it. A cancel is a request and not a verdict: if it reaches nobody it has not been refused, and nothing in the API reports it as such.

Two carrier details are visible to an operator.

The 8000-byte notification cap. PostgreSQL refuses a pg_notify payload of 8000 bytes or more, and that limit is measured against the whole encoded notification, not just the value being replicated. A broadcast that does not fit is written to the bus_messages table and the notification carries the row id instead; the receiving replica reads the row and delivers the original bytes. This is an ordinary path and not an error: a fine-tune job carrying a long training message spills every time. Rows are retired ten minutes after they are written, by every replica, so bus_messages is a spill buffer and never a log of past events.

A broadcast is at most once, and is never replayed. A NOTIFY reaches the sessions that are listening when it is issued and nobody else. A replica whose session was down in that window never receives the change, and no error is reported anywhere. That is why every one of these maps is backed by a durable table: the broadcast says only that something changed, and the table says what it changed to. A replica re-reads its table when its listener reconnects, so a missed delta is a delay and never a value that reads as though it had never been set.

A slow subscriber loses broadcasts rather than stalling the carrier. Each subscription buffers 256 messages; past that, its broadcasts are dropped and logged at error level on the replica that took them. One blocked SSE writer must not be able to stop delivery for the whole deployment, which is what the alternative would mean. The same rule follows from it: what must survive a gap lives in a table.

Open Responses across replicas

A response created by POST /v1/responses is held by the replica that served the request. A round-robin load balancer sends the follow-up poll, the previous_response_id chain and the cancel to any replica, so that metadata is replicated to every frontend and is also written to a response_metadata table in PostgreSQL.

The table is what a replica re-hydrates from. Replication is a broadcast, and a broadcast reaches only the replicas that are subscribed at that moment: a replica whose subscription was down while a response was created never receives that notification. Without the table it would answer 404 for that response forever while its peers answered 200. With it, the replica re-reads the table when its subscription comes back and converges.

What crosses replicas and what does not:

State	Replicated	Why
Request body, response resource, output items, status, owner, expiry	Yes, in memory and in response_metadata	A poll, a previous_response_id chain or an item lookup on any replica has to resolve
Cancellation	The request is, the CancelFunc is not	The cancel is forwarded to the owning replica, which holds the function that stops generation
Stream resume buffer (starting_after)	No	It is the full token log; replicating it would put every generated token on the bus. A resume request that lands on the wrong replica is refused with an explicit error, never with a silently truncated event list

Retention. Each replica sweeps dead rows out of response_metadata every five minutes, and a row is dead when either of two things is true.

• It carries the expiry of the response it describes, and that expiry has passed. The expiry comes from the Open Responses store TTL, which is 0 (no expiration) by default:

  environment:
    LOCALAI_OPEN_RESPONSES_STORE_TTL: "1h"
  

• It carries no expiry, because the TTL is 0, and it is more than 24 hours old.

The 24-hour floor is the table's own bound and it is independent of the TTL. A TTL of 0 is a reasonable answer for the in-memory map it governs, which dies with the process; a table has no such bound, so without a floor response_metadata would grow for the life of the deployment and every restarting replica would re-hydrate every response the cluster had ever created.

The floor never overrides a TTL you set. A row that names an expiry is judged on that expiry alone, longer or shorter than 24 hours. What the floor bounds is only how long a response stays resolvable on a replica that did not create it: the owning replica keeps it in memory for exactly as long as the TTL says. Set a TTL that matches how long clients are allowed to poll for a response.

These rows carry the request body and the generated output, not just identifiers. They live in the same database as the rest of the cluster state.

Agent tasks are scoped to their tenant

Every frontend replica keeps agent task definitions in memory so that GET /api/agent/tasks answers from any replica. That in-memory copy is kept current by a broadcast on the PostgreSQL carrier described above, and the broadcast carries the owning user in the subject:

Map	Subject it publishes on	Subjects it applies
One user's tasks	state.agent-tasks.<user_id>.delta	that subject alone
The administrative, cluster-wide view	state.agent-tasks.delta	state.agent-tasks.delta and state.agent-tasks.*.delta

The user id is its own subject token, so one user's subject can never match another user's. A user's agent tasks are therefore visible only to that user and to the administrative view, which is the same scope the agent_tasks table already applies to reads.

DELETE /api/agent/tasks/{id} is scoped the same way. The delete carries the calling user down to the database, so a request naming a task id that belongs to another user removes nothing and answers 404. The administrative view keeps the unscoped delete, matching how an empty user id already means "every user" for the task and job listings.

Deployments that ran a release before this scoping existed may have in-memory copies of other users' tasks on their replicas. Nothing is written to the database by that, and a restart of the frontend clears it.

Per-user scoping needs the agent pool running, because that is what creates the per-user services. With LOCALAI_DISABLE_AGENTS=true, the agent task routes are still served, and they are served by one cluster-wide service that every authenticated caller shares.

Worker tunnels

A worker can open one long-lived, multiplexed tunnel to the frontend instead of listening on a port of its own. It dials GET /api/cluster/connect?id=<node id>, the connection is upgraded to a WebSocket, and every subsequent request the frontend makes to that worker travels as a stream inside it. Nothing dials into the worker, so a worker behind NAT, in another Kubernetes cluster or on a laptop needs no inbound port and no reachable address.

Each worker has its own tunnel credential

The dial is authenticated against that node's own tunnel credential, which is not the registration token. Registration mints a fresh random secret per node, returns the plaintext once in the registration response as tunnel_token, and stores only its SHA-256. So a leaked registration token no longer opens a tunnel: an attacker who has it, and who knows a node ID, still cannot authenticate as that worker. It cannot reach a worker through the peer link either, which is authenticated per replica in the same way.

A worker that presents a credential belonging to no node, or names a node ID the frontend has never seen, is refused with 401 before the WebSocket upgrade happens. A node still awaiting admin approval is refused with 403. A frontend that cannot read its node table answers 500 rather than 401, so a worker retries instead of re-registering under a new identity.

The credential is rotated on every registration. That follows from storing only the hash: a re-registering worker cannot be told the secret it already holds, so it is given a new one. The worker's live tunnel is unaffected, because the credential is checked when a tunnel is dialled and never again; what changes is which secret the next reconnect presents, and the worker learns it in the same response that rotated it.

A node that has not registered since upgrading cannot tunnel. Its row has no tunnel credential and the column cannot be back-filled, because the plaintext only ever existed in the response that minted it. Such a node is refused with 401 until it registers again, which a worker restart does. The frontend does not fall back to the registration token for these nodes.

Unlike the agent worker's API key, a tunnel credential is issued to a node still awaiting approval. It is inert until then: the tunnel route re-reads the node's status on every dial and refuses a pending one. Withholding it would instead strand workers that register exactly once, since approval on its own prompts no re-registration.

A tunnel credential does not replace LOCALAI_REGISTRATION_TOKEN. Without one, node registration itself is unauthenticated, so anyone who can reach the frontend can register a worker and be issued a tunnel credential for it. How far that gets them depends on auto-approve: with auto-approve on the node is healthy at once and the credential works immediately; with it off the node is pending and the credential is inert until an admin approves, so approval is the real gate. LocalAI warns about the missing token at startup.

Both backend and agent nodes are issued one. Earlier releases minted a credential only for backend nodes, because nothing dialled into an agent worker; the frontend now reaches an agent worker's MCP control verbs over a tunnel of its own, so an agent worker dials one too. A node whose type is neither has its tunnel credential cleared on every registration rather than merely not renewed, so what refuses it is an empty credential and not a second check that could drift from this one.

An agent worker's tunnel carries only the http tag: it runs no backend processes, so it does not offer the grpc tag at all. Its control server binds 127.0.0.1 on a port chosen by the kernel and advertises it nowhere, so an agent worker still opens no inbound port.

An agent worker no longer needs --nats-url, and connects to no message bus at all. Every verb the frontend addresses to a specific agent worker is a control RPC on the tunnel that worker holds: MCP tool execution, MCP discovery, the backend stop that flushes cached MCP sessions, agent execution, MCP CI runs, and now the cancel. The progress and result lines the worker asks the frontend to re-publish on its behalf travel back on that same response body, and the frontend re-publishes them on the PostgreSQL carrier.

Cancelling an agent run

A cancel names one execution by its message id, and nothing in the deployment records which worker holds it: the claim that dispatched the run names the claiming replica, and it is deleted when the run ends. So the frontend offers the cancel to every agent worker a live replica can reach, over each worker's own tunnel (POST /v1/control/agent/cancel), and each worker answers only for itself.

A caller gets one of three answers, and they are deliberately different facts:

Outcome	What it means
success	A worker answered that it cancelled the run, or the run was held by the replica the request landed on.
could not be delivered	At least one agent worker that might have been running it was not reached: its tunnel was lost inside the reconnect grace, its control plane refused the stream, or a peer holding it was unreachable. Nothing was learned. It is not a refusal and not a missing run.
no agent worker is running that execution	Every agent worker was reached and every one of them answered that it does not hold the run.

A worker that is reconnecting always produces the second answer. The cancel is not retried inside the request and not queued: retrying would hold the caller for the length of the reconnect grace, and the retry belongs with whoever owns the budget. Re-issue the cancel once the worker is connected again.

There is no nodes.<id>.* subject left, and an agent worker's minted JWT no longer grants mcp.tools.execute, mcp.discovery, nodes.<id>.backend.stop, agent.execute or jobs.mcp-ci.new. The --agent-subject and --agent-queue flags (LOCALAI_AGENT_SUBJECT, LOCALAI_AGENT_QUEUE) are gone: there is no subject for an agent worker to subscribe to and no queue group to be one of.

Dispatch is a claim, not a queue group

Queued work (agent runs, MCP CI jobs, plain task jobs) is written to a work_claims table rather than published onto a NATS queue group. A frontend replica takes one row at a time with SELECT ... FOR UPDATE SKIP LOCKED, picks a connected agent worker, and drives the work as a streaming control RPC over that worker's tunnel. The worker's progress, its agent events and its terminal result all arrive on the response body the claiming replica is already reading, and that replica persists the terminal line before it releases the claim.

This changes three behaviours an operator can see:

• A job with nowhere to run is now visible. A publish onto a queue group nobody had joined succeeded, and the job sat pending for ever with no trace. A claim row that nothing takes is still in the table.
• A plain task job (a task whose model configures no MCP servers) is now failed with a reason. No agent worker has ever served that kind of job, and it used to be published into silence. It is now marked failed with no worker in this deployment serves plain task jobs. This surfaces a pre-existing gap rather than introducing one.
• Dispatch is at-least-once instead of at-most-once. A transport failure (a lost tunnel, an unreachable peer, a replica that died mid-dispatch) returns the work to the pool and increments the row's attempts; only the worker's own answer, success or failure, removes it.

A claim held by a replica that has gone becomes claimable again; a claim held by a replica that is merely slow is never taken away from it. The reap asks whether the claim's owner is still a live replica in the instances table, on the database clock, and never how long the claim has been held: a job that legitimately runs for an hour on a heartbeating replica is left alone, and a claim whose owner stopped heartbeating is released on the next tick (within 30s, the replica-liveness window).

A frontend replica with no advertised peer address claims no work. Such a replica has no row in the instances table, so no peer can tell its claims from ones a dead replica left, and another replica would take the work away from it mid-run. It logs an error naming LOCALAI_DISTRIBUTED_ADVERTISE_ADDR and starts claiming as soon as it registers. This is the same configuration that already makes a replica's workers unroutable from its peers.

The tunnel lands on exactly one frontend replica, and that replica records itself as the owner of the worker's connection in the node_connections table. When the socket dies the claim is dropped, but the row stays behind with no owner and a disconnected_at stamp, so a worker that is re-dialling the load balancer can be told from one that has never connected. The row is deleted once that departure is older than ten liveness windows (five minutes). If the replica stalls long enough for its peers to reap it, it re-claims the tunnels it still holds on a live session as soon as it re-registers, skipping any whose socket has already gone. That re-claim needs the replica to have an advertised address: without one it never had an instance row to begin with, and its tunnels are usable only by the replica holding them.

Method	Path	Description
GET	/api/cluster/connect?id=<node id>	Worker opens its multiplexed tunnel (Authorization: Bearer <the node's own tunnel credential>)

The route is exempt from the normal session/API-key authentication (it authenticates itself, like /api/cluster/peer) and is registered in every deployment. Outside distributed mode there is no node table to check a token against, so it answers 503.

What the worker does with the tunnel

The worker holds the tunnel with one goroutine: it dials, serves the frontend's streams until the session dies, and dials again. Every stream opens with a small frame naming which local service it is for, and the worker answers before either side speaks the tunnelled protocol:

Tag	Goes to	Target
grpc	a backend process on this worker	the port; the host is discarded and only 127.0.0.1 is dialled, within the worker's own backend port range
http	the worker's own file-transfer and backend-log server (an agent worker's control server)	ignored; there is one such server and only the worker knows where it bound

An agent worker offers only the http row. It runs no backend processes, so the grpc tag has nothing to route to and a stream that names it is refused as an unknown tag.

The grpc row is the security boundary of the tunnel, and it is worth being explicit about it. A tunnel terminates inside the worker process, so a stream arriving on it can reach anything the worker can reach; if the frontend could name the host, whoever holds the frontend end could make every worker in the fleet dial arbitrary addresses on its private network. The worker therefore builds the dial address from a constant 127.0.0.1 and a port it has validated, and the string from the wire never reaches the dialler at all. The port range is the one the worker's own allocator hands to backend processes, which by default runs to 65535; setting LOCALAI_GRPC_MAX_PORT narrows the allocator and this range together, and a worker with a known backend count should set it.

A stream naming a tag the worker does not serve, a target outside that port range, or a local service it could not reach, is refused with a reason and the stream is ended rather than left open. Those refusals are distinct on the wire on purpose: an unknown tag and an out-of-range target are requests this worker will never serve, while an unreachable local service is a backend that has not started yet. A frontend gives up on the first two and may retry the third. One bad stream never affects the others or the session.

Reconnects use exponential backoff with jitter: the interval doubles from 500ms up to a ceiling of 30 seconds, and each wait is drawn between half of that interval and all of it, so no worker ever spins and a fleet that lost the same replica does not come back in lockstep. The interval returns to its floor only after a session that lasted at least 30 seconds. That last part is what stops a rolling frontend restart, where every dial succeeds and then dies moments later, from turning a fleet of workers into a retry storm against the first replica back up. A worker that is refused (401, 403) keeps retrying on the same schedule rather than exiting: a re-registration or an admin approval fixes both without restarting it.

What the frontend sends through it

Every connection the frontend makes to a worker now goes through that worker's tunnel. There are four, and all four are the same path underneath:

What	Protocol	Stream tag
Inference, model load, health checks	gRPC to a backend process	grpc
Model file staging, backend-log listing	HTTP to the worker's own server	http
Live backend-log streaming	WebSocket to the same server	http
The control plane: backend install/upgrade/list/stop/delete, model stop/unload/delete, models running, node stop, and the four object-store staging verbs	HTTP to the worker's own server	http

The worker control plane

A serve-backend worker serves the commands the frontend gives it as ordinary HTTP routes under /v1/control/, on the same loopback server that already carries file staging and backend logs, behind the same LOCALAI_REGISTRATION_TOKEN bearer check. They replace the fourteen nodes.<id>.* NATS subjects a worker used to subscribe to - the ten backend and model lifecycle verbs, plus the four object-store staging verbs (POST /v1/control/files/{ensure,stage,temp,listdir}, mounted only when the deployment configured an object store). The request bodies are unchanged and the reply fields keep their names and types, so nothing an operator inspects on the wire has a new shape. The one difference is that a worker now OMITS an empty reply field where the NATS handlers always emitted it, which a client reading a missing field as the zero value cannot tell apart.

POST /v1/control/backend/stop is served by BOTH kinds of worker, and the frontend sends it the same way to either. A serve-backend worker kills the backend process and recycles its port; an agent worker runs no backend processes and closes the MCP sessions it had cached for that backend. No nodes.<id>.* subject remains, so a worker never takes a control verb off the bus whatever its type.

files/listdir is the verb the change is most visible on. Its reply used to be sized against what the bus would carry, which put a wide model directory close to the limit; it is now a response body the frontend is already reading, so the listing is returned whole and nothing truncates it at either end.

Two of the routes stream. POST /v1/control/backend/install and /v1/control/backend/upgrade answer with application/x-ndjson: zero or more {"progress":{...}} lines carrying the same download-progress payload the per-op NATS subject carried, followed by exactly one {"reply":{...}} line, which is always the last line on the body. When the request carries an op_id, the first progress line has phase resolving and is written before any gallery work begins, so a cold install that spends minutes on a manifest is distinguishable from a stream that is broken. Nothing publishes install progress over NATS any more. An install that FAILS is still a 200 with a reply whose success is false. That is deliberate, and it is the same distinction the refusal table above draws: a non-2xx means the frontend could not get the request to the worker, which nothing may act on, while the worker's own verdict, including "there is no such backend", is evidence a reap guard may act on. A worker that answered 500 for a failed install would put its own verdict in the bucket reserved for a broken link.

A control request carries the caller's deadline and nothing else: the worker does not impose a timeout of its own on an install, and a caller that gives up cancels the download rather than leaving the worker pulling gigabytes for a response nobody will read.

On the frontend's side the ten verbs are ordinary HTTP calls on the http stream tag, so a control RPC to a worker another replica holds is relayed exactly like an inference request - same lookup, same one hop, same budget arithmetic as Reaching a worker another replica holds. There is nothing to subscribe to before an install: its progress lines share the install's own response, so no event can arrive before the caller is listening and there is no per-op subject to grant a permission for.

How a control RPC can FAIL is where absence is decided for the whole control plane, so the frontend maps every outcome onto exactly one row of the table below and never onto another. Only the two rows in which the WORKER spoke may be acted on; everything else is this frontend failing to reach it, which says nothing about the worker at all:

What happened	How the frontend reports it	May anything reap on it?
The worker refused the stream with one of its three evidence codes	the refusal itself, unwrapped	Yes. The worker spoke.
No route: no live owner, an unreachable peer, no relay path, a refusal code this frontend does not recognise, or the worker saying it learned nothing	"this frontend has no route to that worker"	No
The call ran out of budget	a deadline, which install and upgrade report as still installing on the worker	No
404 under /v1/control/	"the worker does not serve that control verb" - it is older than this frontend, and only the upgrade path acts on it, by re-issuing the legacy force-install	No
200 with a reply whose error is set	the worker's own answer, handed to the caller as-is	Yes, by the caller

A 5xx, or a body the frontend cannot decode, is in the second row and not the last: a worker's verdict arrives as a 200, so a 5xx is the server failing rather than answering.

The address the frontend holds for a backend (the per-replica port a worker reports after an install) is still what identifies it, and it is still what appears in logs and errors. What it no longer is, is somewhere the frontend connects to: it travels inside the tunnel as the stream's target, and the worker decides what to do with it.

A frontend with no way to reach a worker says so and fails. It does not fall back to connecting to the worker's advertised address. That fallback is what the tunnel exists to remove, and it is the kind of defect that works on a one-replica developer box and fails in production, so it is an error everywhere. The consequences are deliberately narrow: a model whose worker cannot be reached is not reaped, and its row is left alone, because a frontend that cannot reach a worker has learned nothing about whether that worker is still running the model.

Reaching a worker another replica holds

A worker's tunnel lands on exactly one replica, so with N replicas behind a load balancer roughly (N-1)/N of requests arrive somewhere else. Those requests are relayed: the replica that received the request looks up the owner in node_connections, joined against the live instances rows, opens a stream on its peer link to that owner, and the owner splices it onto the worker's tunnel. One hop, never two; a stale ownership row is answered with a routing refusal and the dialling replica resolves the owner again rather than being sent round a loop.

The dialling replica states how much time its own client has left in the frame that opens the relayed stream, and the owner bounds its work by the smaller of that and its own 15s ceiling. Neither number can lengthen the other: a patient client cannot park the owning replica, and an impatient one cannot be kept waiting on a budget it did not ask for.

These outcomes are kept apart on purpose, because they call for different actions:

Outcome	What it means	What acts on it
No live owner	No replica holds this worker's tunnel	No route right now; the worker's models are left alone
Not the owner	The routing was stale	Resolve the owner again
Peer unreachable	A replica exists and will not answer	Retry
No relay path	This replica cannot reach the owner at all	Report; requests here fail until it can
The worker refused	The worker answered and said no	Depends on WHICH refusal; see below

None of the first four is absence. A worker's presence is its heartbeat, and a route to it is a separate fact that can be false while the worker is registered, heartbeating and serving every request another replica sends it. So the frontend answers "no route", never "this worker is gone", and none of the first four causes a model to be rescheduled or a node_models row to be deleted.

The fifth is different, and deliberately so. A worker that refuses a stream has answered, which proves it is connected; what it is refusing is the stream to one backend process on it. That is the ordinary shape of a crashed backend now that workers listen on nothing: the worker's own dial to the process fails and it says so.

There are four refusals, and only three of them are evidence about a backend. The distinction decides whether a model's row is deleted, so an operator reading one of these in a log can tell what will happen next:

Refusal a worker sends	When	Row reaped?
the worker could not reach the local service for that stream	The worker's own dial to the backend process was refused. A crashed backend	Yes. Reloaded elsewhere, as a dead local backend would be
the worker does not serve that stream tag	The worker does not serve that kind of stream at all	Yes. Nothing clears this until the worker is upgraded, and the model re-registers somewhere that works
the worker rejected the stream request as malformed	The stored backend address is not a port in this worker's range	Yes. The row can never be reached, so reaping lets the model re-register a usable address
the worker could not serve that stream, for a reason that is not about the backend	The request frame did not arrive in the worker's 15s window, the worker's tunnel was being torn down, or it ran out of a local resource	No. These clear on their own; the request fails with "no route" and is retried

The fourth exists because the other three are acted on. A relayed request crosses a peer link before its frame reaches the worker, so on a congested link a frame can arrive late through nobody's fault; reported as one of the first three, that would evict a model that is loaded and serving. If you see the fourth in your logs, look at peer-link congestion or a worker that is reconnecting, not at the backend it names.

A refusal code the frontend does not recognise - a newer worker's vocabulary - is treated as "no route" as well, so a version skew costs a retry rather than a reaped replica.

That distinction is the whole point rather than a nicety. A scheduler told that a connected worker has gone away stops its backend and reclaims every model it is running, and the events that produce "no route" are ordinary ones: a frontend replica restarting, an ownership row a moment stale, a worker that has not dialled its tunnel yet. Absence has its own two mechanisms and neither of them is a failed request: a stale heartbeat (see --stale-node-threshold), and a tunnel departure older than the reconnect grace (see below).

There is no frontend-side fallback

LOCALAI_WORKER_TUNNEL=false is a fatal startup error on this release. It is not a degraded mode and not a rollback switch: the worker refuses to boot and prints why. Nothing else would be honest, because the setting stops the worker dialling its tunnel while no frontend path dials a worker's advertised address, and a worker on this release advertises none and listens on no routable interface, so a worker that started with it off would register, heartbeat, be scheduled onto, and fail every request. The rollback is to run the previous release on both sides.

Upgrade the frontends first

Upgrade every frontend replica, then restart the workers one at a time.

• Frontends first (correct). Old workers keep running, keep heartbeating and keep their node_models rows: the new frontend reports them as unroutable rather than as gone, so nothing is rescheduled and nothing is reaped. What fails is requests for models on a worker that has not been restarted yet. That is a real degraded window, but it is bounded by how fast you roll the workers, it heals itself as each one comes back, and no state is lost.
  • What you will see while it lasts: requests for models on a not-yet-restarted worker fail with "no route to the worker", while GET /api/nodes still shows that node healthy and heartbeating and its models still listed. Restart the worker and it clears. Nothing needs fixing; you are watching the window close.
  • While the frontends themselves are rolling, a frontend you have not restarted yet cannot open a peer link to one you have: it holds no peer credential and the upgraded replica refuses unproven ids. An upgraded replica dialling an older one still works, so the loss is one-directional. What it costs is relayed requests that land on a not-yet-restarted replica for a worker an upgraded replica owns; they fail with no route from this replica to that worker, which is a routing fact and not absence, so nothing is rescheduled or reaped. Both sides log it by name. Restart the remaining frontends and it clears.
• Workers first (this fails, do not do it). An old frontend has no /api/cluster/connect route for the worker to dial and rejects the new worker's registration outright, because the worker no longer sends an address and the old frontend requires one. A 4xx is a verdict rather than an outage, so the worker reports the reason on the first attempt and exits instead of retrying. Every worker you restart is a worker you take out of the fleet until the frontends are upgraded.

  • What you will see if you do it anyway: each restarted worker exits within a second or two of starting, with

    registration failed with status 400: {"error":{"code":400,"message":"address is required for backend workers","type":"node_error"}}: the frontend refused this registration
    

    The fleet drains one node per restart, and the nodes that are left are the ones you have not touched yet. Grep for address is required for backend workers if your log collector reflows the line.

A worker that cannot reach its frontend at the network level retries with exponential backoff and never gives up, so restarting a worker is all that is needed to close the frontend-first window. A worker whose registration is rejected does not retry, which is what makes the wrong order destructive rather than slow.

Rolling a frontend back requires restarting every worker

Registering against an upgraded frontend clears a node's address and http_address columns in the shared database, and re-registration is the only thing that ever writes them back. So a partial rollback does not restore the previous behaviour on its own: the old frontend code reads an empty address for every node that has registered since the upgrade and dials nothing. Roll the frontends back and then restart every worker so each one re-registers and repopulates its address. Rolling back is not a frontend-only operation.

Workers bind nothing routable

A worker on this release opens no inbound listener on a routable interface. Its backend gRPC processes and its HTTP file-transfer server all bind loopback, and the frontend reaches both through the tunnel. Concretely:

• No inbound firewall rule, published port, Service or Ingress is needed for a worker. A serve-backend worker needs outbound access to the frontend URL (LOCALAI_REGISTER_TO), and nothing else. An agent worker binds only loopback too, and needs outbound access to LOCALAI_REGISTER_TO and nothing else either: registration, heartbeats, its tunnel and every verb the frontend addresses to it all go there.
• LOCALAI_ADVERTISE_ADDR and LOCALAI_ADVERTISE_HTTP_ADDR are gone. There is nothing to advertise. Both are ignored if still set; remove them.
• LOCALAI_ADDR and LOCALAI_SERVE_ADDR are read for their port only. The port is the base of the backend port range, and port-1 is the HTTP file-transfer port. The host half names an interface nothing binds.
• The node's address and http_address fields in GET /api/nodes are empty, and are cleared for nodes that reported them before the upgrade.

A lost tunnel is a departure, not an absence

When a worker's tunnel goes, the frontend does not forget the worker. It records when the tunnel went, and for a grace period after that the worker is reported as reconnecting, not as gone. Only once the departure is older than the grace may anything act on the worker's absence: stop scheduling work onto it, clean up its rows, release its models.

That distinction exists because a worker loses its tunnel for entirely ordinary reasons. A frontend replica restarting during a rolling upgrade drops every tunnel it held, and each of those workers immediately re-dials the load balancer and lands on another replica. Treating that as "the worker is gone" would evict models mid-upgrade for a fleet that never actually went anywhere.

This applies to both worker kinds, under the same grace. An agent worker holds a tunnel of its own and is reached through it and through nothing else, so its node_connections row ages exactly like a backend worker's and means exactly the same thing: past the grace it is reported unhealthy in GET /api/nodes. Earlier releases exempted agent nodes by type, because an agent worker took its work over a message bus and a departed tunnel said nothing about it; there is no bus any more, so that exemption would hide the only symptom an unreachable agent worker has. What still differs is placement, not liveness: the scheduler never sees agent nodes (every placement query selects node_type = 'backend'), and backend listing and backend install/upgrade/delete still skip them, because an agent worker runs no backend processes.

A departure evicts the per-node state the frontend was holding. On the transition to gone, and once per departure rather than once per health cycle, the frontend drops that node's prefix-cache affinity entries in every model, its cached backend-probe results, its in-flight file-staging operations, and its rows in the per-node breakdown of every operation still open in GET /api/operations. A worker that comes back is re-probed rather than trusted, and an operation that was waiting on a node that left stops reporting it as still copying. Nothing is evicted for a worker that is merely reconnecting: the eviction is an act on absence, so it fires only where the demotion does.

Three things read this. The scheduler reads it before it places a cold load: a worker whose departure has outlived the grace is skipped and marked unhealthy, so every other frontend replica stops choosing it too. LRU eviction reads it before it hands back the node it freed capacity on, because a node full enough to be an eviction target is exactly the node the placement selectors never offer, so the scheduler's own check never sees it. The health monitor reads it on every cycle, which is what covers the case the heartbeat cannot see. A worker's heartbeat says its supervisor is alive; it says nothing about whether anything here can reach that worker's backends, because those are reached over the tunnel. A worker that heartbeats with a permanently dead tunnel (a proxy that stopped upgrading WebSockets, a rotated registration credential, a reconnect loop longer than the grace) is therefore marked unhealthy too, rather than staying listed healthy while every request for a model loaded on it fails "no route to that worker".

The log lines are Scheduled node has no tunnel and its departure outlived the reconnect grace, marking unhealthy and re-scheduling, Eviction target has no tunnel and its departure outlived the reconnect grace, marking unhealthy and evicting again, and Node is heartbeating but its tunnel has been gone longer than the reconnect grace; marking unhealthy. Each names the grace it used, and the last one names the node type as well, so an agent worker's demotion is not read as a backend worker's.

The demotion is a status change, not a deletion. The worker's node_models rows survive it. Status is enough to unwedge the node, because request routing and LRU eviction both select only healthy nodes, so the model stops being served from there and the next request places it somewhere reachable. Deleting rows on a presence read would give any future defect in that read the widest possible blast radius, for nothing the demotion does not already deliver.

Status is a trailing signal, though: it is only as fresh as the last health cycle. That is why the two paths that are about to commit work to a node, cold-load placement and eviction, read presence directly instead of trusting the status column. Everything else reads the column.

A returning heartbeat does not promote a node back on its own. Recovery needs the tunnel back, not just the supervisor: the health monitor re-promotes a demoted node only once presence reports it connected or reconnecting again. Before that check the two were conflated, and a heartbeating worker with a dead tunnel was promoted back to healthy on the next 15s cycle, every cycle.

The other three answers place work as normal. Reconnecting (the tunnel went inside the grace) and unknown (no connection row at all: the worker has never dialled, or its departure has already aged out of retention) are both non-verdicts; so is a failure to read the answer, because a database hiccup that excluded workers would cost the fleet its capacity for a reason that has nothing to do with any worker. In each of those cases scheduling proceeds, the node keeps its status, and the install that follows reports its own outcome.

A frontend refuses to start if either reader was built without a source of absence, because that failure has no other symptom: it looks exactly like a fleet that is fine.

Flag	Env var	Default	Description
--worker-reconnect-grace	LOCALAI_WORKER_RECONNECT_GRACE	90s	How long a worker whose tunnel was lost is treated as reconnecting rather than gone.

The default clears the worker's own worst-case reconnect. A worker retries with exponential backoff capped at 30s, and each attempt has a 10s dial budget, so a worker that waits the ceiling, hangs a dial, and waits the ceiling again is back at 70s. The backoff also only resets after a session that lasted 30s, which a replica accepting a dial and then dying denies, so a worker crossing a rolling restart really does climb to the ceiling rather than sitting near the 500ms floor. 90s clears that worst case with margin; 60s would sit under it.

What you trade by changing it:

• Lower: a worker that has genuinely gone away is declared absent sooner, so its rows are cleaned and its models released sooner. Set it below the worker's backoff ceiling and you will condemn workers that are re-homing exactly as designed.
• Higher: a rolling frontend restart is safer, because workers crossing it stay "reconnecting" for longer. The cost is that a worker that really has died keeps its rows for longer.

The window is measured on the database clock, not on any frontend's own clock, so every replica agrees to the second on when a worker's grace ran out. A departure row is kept well past the grace before it is purged; once purged, the worker reads as unknown rather than gone, and nothing acts on unknown.

The model load deadline scales with the checkpoint

The LoadModel deadline starts after the backend is installed and the model files are staged, so it covers only the worker backend's own checkpoint read and pipeline init. That work is proportional to the bytes on disk, which makes any fixed deadline a model-size cliff rather than a timeout: a 70 GB video checkpoint on a Jetson Thor worker failed reproducibly against the old fixed 5m default (rpc error: code = DeadlineExceeded after 953.5s of wall clock, roughly 11m of which was backend install and staging), and simply raising the constant would only move the cliff to the next larger model while making a genuinely wedged small model hang for the whole inflated duration.

So the deadline is derived per model:

budget = 5m + 20s per GiB of checkpoint,  capped at 6h

Checkpoint	Derived budget
2 GB	5m40s
70 GB	28m20s
600 GB	3h25m

The per-GiB rate is deliberately pessimistic — it corresponds to reading weights at about 54 MB/s, below what any supported storage sustains — because the two errors are not symmetric: a budget that is too long costs only failure latency on a load that was going to fail anyway, while a budget that is too short causes a guaranteed false failure on a load that was perfectly healthy.

The size is measured from the model files on the frontend's disk, over the same set of paths that get staged to the worker. If those files are not present locally — a backend handed a bare HuggingFace repo id fetches its own weights on the worker — there is nothing to measure and the budget stays at the plain 5m default. Pin LOCALAI_NATS_MODEL_LOAD_TIMEOUT for those models if their load is slow.

When the budget is exceeded, the error names the budget, the checkpoint size it was derived from, and the knob that overrides it, instead of surfacing a bare context deadline exceeded.

The cold-load lock ceiling

The router also bounds how long a single cold load may hold the per-model advisory lock, so a worker that dies mid-install cannot pin every other replica's request for that model. That bound is derived, not configured, and it is based on progress rather than on wall-clock time.

The load starts with a base budget of max(backend-install-timeout + model-load-timeout + 5m, 25m) — with the defaults, 15m + 5m + 5m = 25m. That budget covers the steps that report no progress: node selection, backend install, and the remote LoadModel call. Raising either timeout widens it in step, so a longer load deadline is never clipped.

That base is the hold's starting budget, not its maximum. Because the derived load budget above can exceed it — a 70 GB checkpoint's 28m20s against a 25m base — the hold is widened again as the router enters the load phase, by the derived budget plus the same 5m of slack. Without that step the ceiling would cancel a load that was still comfortably inside its own deadline.

While model files are staging, however, the deadline extends every time staging does real work, and expires only once staging has been silent for a 5-minute stall window. Real work means uploaded bytes, and also the resumable-upload verify phase: when a shard is already present on the worker from an earlier attempt, the frontend HEADs it and hashes the local copy to confirm it matches, then skips the transfer. That phase uploads nothing at all — on a 70 GB model resuming with 56 GB already staged it ran for six-plus consecutive minutes at ~45s per shard — so hashing counts as progress too. Otherwise a resumed transfer would be mistaken for a wedged one. Staging time is a function of checkpoint size and available bandwidth, not a constant: a 70 GB model at 26 MB/s needs about 45 minutes, and a 600 GB checkpoint needs hours. A fixed ceiling would therefore be a model-size cliff — every increase just moves the cliff to the next larger model. Extending on progress means a large model transfers for as long as it legitimately needs, while a worker that dies mid-transfer still releases the lock within the stall window.

An absolute cap of 24h ends the hold even if progress keeps arriving, so a degenerate peer trickling a few bytes at a time cannot pin the lock forever. No configuration is needed for either value; both are sized well above any legitimate transfer.

Requests for a model that is still loading

A cold load in distributed mode is a long-running background job: install the backend, stage multi-GB model files to the worker, then load the checkpoint. Staging a 35.7 GB GGUF onto a fresh worker takes roughly twenty minutes on a fast LAN — far longer than any HTTP request can be held open.

So the load does not run on the request. The first request for an unloaded model claims a durable model load job — that claim takes milliseconds and is the only part that holds the per-model advisory lock — and the job then runs in the background on the frontend replica that claimed it. Every other request for the same model, on any replica, attaches to that job as a waiter:

• It is served the moment the model is ready, with no client-side retry. A model already 90% staged usually needs no second request.
• It never starts a duplicate load and never blocks on the database lock. (Before this split, concurrent requests blocked on pg_advisory_lock for the whole load and were killed by the PostgreSQL role's statement_timeout — SQLSTATE 57014 — so from the operator's seat the model simply never loaded.)
• If the load fails, the waiter gets the real cause (worker out of disk), not an anonymous timeout.
• If the client disconnects, the load keeps going. It belongs to the job record, not to the request.

When the wait budget (LOCALAI_MODEL_LOAD_WAIT, default 60s) runs out, the request is answered with 503, a Retry-After header, and a body that says exactly where the load is:

{
  "error": {
    "message": "model Qwen3.6-27B-MTP-GGUF is staging on node nvidia-thor (41%, ETA ~11m)",
    "type": "model_loading",
    "code": "model_loading"
  },
  "loading": {
    "model": "Qwen3.6-27B-MTP-GGUF",
    "state": "staging",
    "node": "nvidia-thor",
    "progress": 41.2,
    "bytes_sent": 14730000000,
    "total_bytes": 35776484480,
    "file_index": 1,
    "total_files": 2,
    "eta_seconds": 660
  }
}

The error envelope keeps OpenAI clients working unchanged; loading is additive, so a client that understands it renders progress instead of an error. eta_seconds is derived from the job's own observed transfer rate and is omitted rather than guessed until enough bytes have moved for that rate to mean anything — a confidently wrong ETA on a twenty-minute wait is worse than none. state is one of pending (choosing a node), installing, staging (transferring files) or loading (the worker is reading the checkpoint).

The chat UI renders this state inline and retries automatically once the model reports ready. Poll GET /api/models/{id}/load-status for the same loading object at any time.

{{% notice note %}} A frontend replica that dies mid-load does not wedge the model: the job row carries a heartbeat and another replica reclaims a job whose heartbeat has stopped. The heartbeat is time-based, not byte-based, because a checkpoint load legitimately transfers zero bytes for many minutes. {{% /notice %}}

Migrating off the message broker

Earlier releases of distributed mode required a NATS cluster alongside PostgreSQL. They no longer do. Shut the broker down. Nothing in LocalAI opens a connection to one: the frontend's cross-replica fan-out is on PostgreSQL, queued work is a claim on a PostgreSQL table, a serve-backend worker takes every verb on its own tunnel, and an agent worker does too, including the cancel that was the last family on a bus.

A distributed deployment needs PostgreSQL and the frontends' own HTTP listener, and nothing else. Workers dial out to that listener and hold the tunnel open, so no worker needs an inbound port either. There is no broker client left in LocalAI at all: as of this release the nats-io modules are not in the build, so the binary cannot open a broker connection even if something asked it to.

There is no migration step and no cutover window. Stop the broker, delete its service from your compose file, chart or unit files, and delete the credentials you generated for it. A deployment that keeps running one is paying for infrastructure that carries nothing.

Your existing command lines still start. Every LOCALAI_NATS_* setting below is parsed and then ignored, so an unedited command line, unit file or Helm values file needs no change on the day you upgrade. They are hidden from --help, because there is nothing left to configure with them. They are scheduled for removal in the release after next; remove them from your own files at your convenience before then.

Flag	Env Var	Status
--nats-url	LOCALAI_NATS_URL	Accepted and ignored on the frontend, local-ai worker and local-ai agent-worker. The value is never dialled, so it may point at a broker that is already gone.
--nats-account-seed	LOCALAI_NATS_ACCOUNT_SEED	Accepted and ignored. The frontend mints no per-node broker credential: a register or approve response carries no nats_jwt and no nats_user_seed, and a worker that reads those keys finds nothing. Nodes are authenticated by their registration token and their tunnel token.
--nats-service-jwt / --nats-service-seed	LOCALAI_NATS_SERVICE_JWT / LOCALAI_NATS_SERVICE_SEED	Accepted and ignored: the frontend opens no bus connection to present them on.
--nats-worker-jwtttl	LOCALAI_NATS_WORKER_JWT_TTL	Accepted and ignored. No per-node broker credential is minted, so none has a lifetime.
--nats-require-auth	LOCALAI_NATS_REQUIRE_AUTH	Accepted and ignored. It used to make an agent worker wait through admin approval; use --distributed-require-auth for that (see below).
--nats-tlsca / --nats-tls-cert / --nats-tls-key	LOCALAI_NATS_TLS_*	Accepted and ignored. The paths are no longer checked for existence either, so a certificate deleted with the broker does not fail startup.

{{% notice warning %}} One behaviour changed, on the agent worker. --nats-require-auth used to make local-ai agent-worker wait through admin approval at registration instead of starting against a pending node. That wait is now asked for with --distributed-require-auth / LOCALAI_DISTRIBUTED_REQUIRE_AUTH, which already implied it. An agent worker started with only --nats-require-auth no longer waits: it registers, starts, and its tunnel dials are refused with 403 until an admin approves it, which is the historical default behaviour. If you relied on the wait, set --distributed-require-auth.

On the frontend, --distributed-require-auth now implies only --registration-require-auth. It used to also require broker credentials, and there are none to require. {{% /notice %}}

{{% notice warning %}} LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT, LOCALAI_NATS_BACKEND_UPGRADE_TIMEOUT and LOCALAI_NATS_MODEL_LOAD_TIMEOUT are not in the table above and must not be removed. Despite their names they were never broker settings: each one is a control-RPC budget the frontend applies to a worker, and each is still read and still enforced. They are documented with the other frontend flags in Frontend Configuration. The names are kept because renaming them would break every existing deployment for cosmetics. {{% /notice %}}

{{% notice note %}} LOCALAI_AUTH (HTTP users/sessions) is unrelated. HTTP registration still uses LOCALAI_REGISTRATION_TOKEN, and every worker control plane sits behind that same bearer check. {{% /notice %}}

Optional: S3 Object Storage

For multi-host deployments where workers don't share a filesystem, S3-compatible storage enables distributed file transfer (model files, configs). The frontend uploads the file to the bucket and then tells the worker to fetch it, over that worker's tunnel (POST /v1/control/files/ensure); the reverse direction (.../files/stage) has the worker upload one of its own files for the frontend to pull down. The bytes travel through the bucket, never through the tunnel:

Flag	Env Var	Default	Description
--storage-url	LOCALAI_STORAGE_URL	(empty)	S3 endpoint URL (e.g., http://minio:9000)
--storage-bucket	LOCALAI_STORAGE_BUCKET	localai	S3 bucket name
--storage-region	LOCALAI_STORAGE_REGION	us-east-1	S3 region
--storage-access-key	LOCALAI_STORAGE_ACCESS_KEY	(empty)	S3 access key
--storage-secret-key	LOCALAI_STORAGE_SECRET_KEY	(empty)	S3 secret key

A worker started without LOCALAI_STORAGE_URL does not serve the four staging verbs at all, and answers 404 for them, which is the same answer a frontend gets from a worker too old to know them.

When S3 is not configured, model files are transferred directly from the frontend to workers via HTTP - no shared filesystem needed. Each worker runs a small HTTP file transfer server alongside the gRPC backend process. This is the default and works out of the box.

For high-throughput or very large model files, S3 can be more efficient since it avoids streaming through the frontend.

Shared models directory

If every node (frontend and workers) mounts the same models directory at the same path - for example a shared volume or network filesystem, as shown in the "Shared Volume Mode" section of docker-compose.distributed.yaml - the model files are already present on each worker at their canonical path. In that case staging is wasted work: it copies files that already exist into a per-model subdirectory the worker then loads from, which shows up as a re-download of a model you already have.

Set LOCALAI_DISTRIBUTED_SHARED_MODELS=true (or --distributed-shared-models) on the frontend to skip staging entirely. The router then leaves the model's absolute paths untouched and the worker loads them directly from the shared volume.

This flag is a contract you assert: all nodes must mount identical paths. Leave it off (the default) when workers have independent models directories - the frontend stages files to them over HTTP (or S3) as described above.

Model artifact staging

For managed Hugging Face artifacts, the controller resolves the repository and downloads every selected file. Workers receive the committed snapshot through the existing directory stager. They never receive HF_TOKEN and do not contact Hugging Face for managed artifacts.

With LOCALAI_DISTRIBUTED_SHARED_MODELS enabled, workers use the shared absolute snapshot path and skip transfer. Otherwise, the controller stages the complete snapshot tree to each worker before loading the backend. With an object store configured the controller uploads to the bucket and commands the worker to fetch over its tunnel; without one it pushes the files to the worker's HTTP file transfer server directly.

{{% notice warning %}} Every controller and worker must have enough disk space for its own snapshot copy unless shared-models mode is enabled. Account for temporary partial files during installation as well as the committed snapshot. {{% /notice %}}

{{% notice warning %}} The worker HTTP file transfer server is authenticated by LOCALAI_REGISTRATION_TOKEN. If the token is empty, the server fails open - anyone who can reach the port gets read/write access to the worker's models/staging/data directories (a remote model-poisoning / exfiltration vector), and to the /v1/control/ routes that install, upgrade and delete backends and stop the node. The worker logs a loud warning at startup in this case. Always set LOCALAI_REGISTRATION_TOKEN in distributed mode, and set LOCALAI_DISTRIBUTED_REQUIRE_AUTH=true (frontend and workers) to make a missing token a hard startup error rather than a silent fail-open. On an agent worker it additionally makes registration wait through admin approval instead of starting against a pending node.

By default the server binds loopback, so "anyone who can reach the port" means a process on the worker host, and no firewall rule is required. Setting LOCALAI_HTTP_ADDR to a routable address opts back out of that and puts the fail-open case back on the network - if you do it, firewall the port. {{% /notice %}}

Watching Backend Installs

While a worker downloads a backend, the admin operations strip at the top of the UI shows real-time progress: a percentage, and, when the install targets several workers, a roll-up of how far the fan-out has got, 2 of 5 nodes done. Per-file byte counts are not on the strip; they are in the per-node detail below.

The per-node detail is on the Operate → Activity page ([Activity]({{% relref "operations/activity" %}})). When an install targets more than one worker, an N nodes tag appears on the operation card, with one row per worker showing:

• A status pill: Queued (gray), Downloading (blue), Worker busy (yellow), Done (green), or Failed (red).
• The file currently being downloaded with current/total bytes and percentage.
• A thin per-node progress bar.
• Any error returned by the worker.

The yellow Worker busy pill means the worker took longer than --backend-install-timeout to acknowledge but is most likely still working in the background. The admin UI clears it as soon as the worker finishes; no action is required from the operator.

If a worker is running an older LocalAI release that does not report progress, its row in the breakdown will still show terminal status (queued / done / failed / worker busy) but no per-file progress.

The Record on that page - what model and backend installs and removals have finished - is read from PostgreSQL rather than from the replica's memory. Every replica reports the same record, it survives restarts, a replica added by a scale-out or a rolling deploy reports it in full, and Clear history clears it for every replica.

Worker Configuration

Workers are started with the worker subcommand. Each worker is generic - it doesn't need a backend type at startup:

local-ai worker \
  --register-to http://frontend:8080 \
  --registration-token changeme

There is no broker flag here. A serve-backend worker dials one outbound tunnel to --register-to and serves every request the frontend makes of it over that. A --nats-url left over from an older command line is still accepted and ignored; see Migrating off the message broker.

Flag	Env Var	Default	Description
--addr	LOCALAI_ADDR	(unset)	Base port for backend gRPC processes. Only the port is used; nothing binds the host
--serve-addr	LOCALAI_SERVE_ADDR	0.0.0.0:50051	Same, used when --addr is unset
--grpc-max-port	LOCALAI_GRPC_MAX_PORT	65535	Highest port the worker may assign to a backend gRPC process. Each backend gets its own port, allocated upward from the base port, so the width of [base port, this] caps how many backends this worker can run at once (see Backend gRPC port range)
--http-addr	LOCALAI_HTTP_ADDR	127.0.0.1:{gRPC port - 1}	HTTP file transfer server bind address
--register-to	LOCALAI_REGISTER_TO	(required)	Frontend URL for self-registration
--node-name	LOCALAI_NODE_NAME	hostname	Human-readable node name
--registration-token	LOCALAI_REGISTRATION_TOKEN	(empty)	Token to authenticate with the frontend
--registration-require-auth	LOCALAI_REGISTRATION_REQUIRE_AUTH	false	Refuse to start the HTTP file-transfer server when no registration token is set (it would otherwise fail open)
--distributed-require-auth	LOCALAI_DISTRIBUTED_REQUIRE_AUTH	false	Umbrella switch implying --registration-require-auth
--heartbeat-interval	LOCALAI_HEARTBEAT_INTERVAL	10s	Interval between heartbeat pings
--worker-tunnel	LOCALAI_WORKER_TUNNEL	true	Hold one outbound multiplexed tunnel to the frontend and serve its requests over it, so this worker needs no inbound port (see Worker tunnels). Setting it to false is a fatal startup error, not a degraded mode: the frontend has no path that dials a worker's advertised address, so a worker without its tunnel is a worker nothing can reach. To run without tunnels, run the pre-tunnel release on both the worker and the frontend.
--backends-path	LOCALAI_BACKENDS_PATH	./backends	Path to backend binaries
--models-path	LOCALAI_MODELS_PATH	./models	Path to model files
--vram-budget	LOCALAI_VRAM_BUDGET	(empty)	Cap the VRAM this node advertises for model placement, as a percentage (e.g. 80%) or an absolute amount (e.g. 12GB). Empty uses all detected VRAM. See Per-node VRAM budget.

{{% notice tip %}} There is no advertise address. A worker states no endpoint at registration and binds nothing routable; the frontend reaches it through the tunnel it dials. --advertise-addr and --advertise-http-addr no longer exist. --addr and --http-addr remain, and set where the worker listens locally: only the port of --addr is used, and --http-addr binds loopback by default.

HTTP file transfer: Each worker also runs a small HTTP server for file transfer (model files, configs). It listens on loopback at the gRPC base port - 1 (e.g., if gRPC base is 50051, HTTP is on 50050). gRPC ports grow upward from the base port as additional models are loaded. {{% /notice %}}

Worker Health Probes

The worker's HTTP server (loopback, base port - 1, default 50050) exposes two unauthenticated probes. They are reachable from the worker host - which is where a container healthcheck runs - and not from the network:

Endpoint	Meaning
/healthz	Liveness. 200 whenever the process is up and serving. Deliberately independent of readiness, so a frontend restart that drops every tunnel does not trigger a restart storm across every worker.
/readyz	Readiness. 200 only when the worker is registered and it currently holds a tunnel session; 503 otherwise.

/readyz tracks the tunnel, because that is the only way anything reaches this worker: it binds loopback, advertises no address, and every request the frontend makes of it arrives as a stream inside that tunnel. It reports something the local supervisor cannot see on its own. The node registry's status and last_heartbeat are driven by an HTTP heartbeat to the frontend, a different network path - a worker can keep heartbeating while its tunnel is dead, and so appear healthy in the registry while being unreachable. The local probe closes that gap.

A 503 here is this container's own report that it cannot serve right now, and nothing else. It is not a claim that the worker is gone; the frontend decides that from the tunnel session it holds, aged against LOCALAI_WORKER_RECONNECT_GRACE. The worker keeps heartbeating throughout a tunnel outage for exactly that reason: withholding the heartbeat would report an unreachable worker as an absent one, on the one path that has no grace.

The container image's HEALTHCHECK detects worker mode and probes this endpoint automatically, deriving the port from LOCALAI_HTTP_ADDR, else LOCALAI_ADDR, else LOCALAI_SERVE_ADDR, minus one - the same order the worker itself uses. No HEALTHCHECK_ENDPOINT override is needed. Set HEALTHCHECK_ENDPOINT only when the bind address is passed as a CLI flag rather than an environment variable, or to pin an explicit URL.

Worker Port Configuration

A worker needs no address configuration at all. It binds only loopback and reaches the frontend outbound, so the defaults work behind NAT, in another cluster, or on a laptop:

environment:
  LOCALAI_REGISTER_TO: "http://frontend:8080"
  LOCALAI_REGISTRATION_TOKEN: "my-secret"

Set the variables below only to move the worker's local port range - for example when two workers share a host, or when the default range collides with something else. Only the port of each is used; the host half names an interface nothing binds.

Variable	Description	Default
LOCALAI_ADDR	Base port for backend gRPC processes, as host:port. port-1 is the HTTP file-transfer port	(unset; falls back to LOCALAI_SERVE_ADDR)
LOCALAI_SERVE_ADDR	Base port, as above, when LOCALAI_ADDR is unset	0.0.0.0:50051
LOCALAI_GRPC_MAX_PORT	Highest port assignable to a backend gRPC process	65535
LOCALAI_HTTP_ADDR	HTTP file transfer bind address. Bound exactly as given, so this is also the way to expose that server deliberately	127.0.0.1:{base port - 1}

LOCALAI_ADVERTISE_ADDR and LOCALAI_ADVERTISE_HTTP_ADDR no longer exist. They named the endpoint the frontend dialled; nothing dials a worker any more. Remove them.

Backend gRPC port range

Every backend process a worker starts listens on its own gRPC port, allocated upward from the worker's base port (LOCALAI_SERVE_ADDR, default 50051). LOCALAI_GRPC_MAX_PORT sets the top of that range. The width of [base port, LOCALAI_GRPC_MAX_PORT] is therefore a hard cap on how many backend processes one worker can run concurrently.

Set it when the worker shares a host with other services and you need to keep the rest of the ephemeral range clear, or when you want a worker's backend count bounded explicitly rather than by whatever the host happens to allow:

# Confine this worker's backends to 50051-50150 (100 concurrent backends).
LOCALAI_SERVE_ADDR=0.0.0.0:50051
LOCALAI_GRPC_MAX_PORT=50150

Leave it unset (the default) and the worker may use anything up to 65535.

Budget headroom above your real concurrency. When a backend stops, its port is held briefly before it can be reused, so a worker with heavy start/stop churn has more ports tied up than it has running backends at any instant. If the range does fill, backend starts fail with:

no free gRPC port in range: 50051-50150 is fully consumed by 100 running
backend(s), 12 port(s) still in quarantine and 0 port(s) already bound by
something outside this worker; raise LOCALAI_GRPC_MAX_PORT to widen the range

Raise LOCALAI_GRPC_MAX_PORT (or reduce how many models you schedule onto that worker). A value above 65535 is clamped, and a value below the base port is ignored in favour of the full range, so a typo degrades the setting rather than wedging every backend start on the node.

The worker checks that a port is actually free before it hands it out. Its own bookkeeping records only what this worker did, and the collision it cannot see is with something this worker never did: the default base port sits inside Linux's default ephemeral range (32768-60999, see net.ipv4.ip_local_port_range), so the kernel can give a port in the range to an outbound connection, or to any process that binds port 0, while the allocator still believes it free. A backend handed one of those dies on bind. Each candidate is therefore probed, and a port something else holds is skipped and retried later rather than dropped, since whatever holds it is usually an ephemeral connection that gives it back. The last count in the message above is how many candidates were skipped that way, and the worker logs one line per allocation when it skips any:

Skipped gRPC ports in this worker's range that something outside the worker
already holds ... skipped=3 allocated=50054

Seeing that regularly means the worker's range overlaps what the kernel is handing out. Move the range with LOCALAI_ADDR to a base port outside net.ipv4.ip_local_port_range (for example 61000) and the overlap goes away entirely. Nothing dials these ports from outside the worker, so the base port is free to be anything bindable.

NVIDIA GPU support

When running workers in a container, two runtime settings affect how VRAM usage is reported back to the frontend:

• NVIDIA_DRIVER_CAPABILITIES must include utility. Without it, the NVML library (and therefore nvidia-smi) is not available inside the container. CUDA compute still works, but the worker cannot query free VRAM and the Nodes page will show the node as fully used. Set NVIDIA_DRIVER_CAPABILITIES=compute,utility (or, with the NVIDIA CDI runtime, list capabilities: [gpu, utility] on the device reservation).

• Run the container with init: true (or docker run --init). The worker process becomes PID 1 in the container and cannot reap zombies on its own. Without an init, nvidia-smi calls can fail intermittently with waitid: no child processes, which briefly clears free-VRAM metrics.

Unified memory devices (Jetson, DGX Spark / GB10, Thor): these SoCs share one physical RAM between CPU and GPU. LocalAI detects them via /sys/devices/soc0/family and /sys/devices/soc0/soc_id (no nvidia-smi required) and reports system-RAM figures as VRAM. Free VRAM therefore tracks MemAvailable in /proc/meminfo. Workers report RAM metrics independently from VRAM on every registration and heartbeat. On unified-memory nodes, the available RAM and available VRAM values should therefore track each other closely; on discrete-GPU nodes they can change independently.

Node Labels

Workers can declare labels at startup for scheduling constraints:

Variable	Description	Example
LOCALAI_NODE_LABELS	Comma-separated key=value labels	tier=premium,gpu=a100,zone=us-east

Labels can also be managed via the admin API (see Label Management API below).

The system automatically applies hardware-detected labels on registration:

• gpu.vendor -- GPU vendor (nvidia, amd, intel, vulkan)
• gpu.vram -- GPU VRAM bucket (8GB, 16GB, 24GB, 48GB, 80GB+)
• node.name -- The node's registered name

How Workers Operate

Workers start as generic processes with no backend installed. When the SmartRouter needs to load a model on a worker, it calls POST /v1/control/backend/install through that worker's tunnel with the backend name and model ID. The worker:

1. Installs the backend from the gallery (if not already installed)
2. Starts a new gRPC backend process on a dynamic port (each model gets its own process)
3. Replies with the allocated gRPC address
4. The SmartRouter calls LoadModel via direct gRPC to that address

Workers can run multiple models concurrently - each model gets its own gRPC process on a separate port. For example, an embedding model on port 50051 and a chat model on port 50052 can run simultaneously on the same worker.

When the SmartRouter needs to free capacity, it can unload models with zero in-flight requests without affecting other models on the same worker.

Node Management API

The API is split into two prefixes with distinct auth:

/api/node/ - Node self-service

Used by workers themselves (registration, heartbeat, etc.). Authenticated via the registration token, exempt from global auth.

Method	Path	Description
POST	/api/node/register	Register a new worker
POST	/api/node/:id/heartbeat	Update heartbeat timestamp
POST	/api/node/:id/drain	Mark self as draining
GET	/api/node/:id/models	Query own loaded models
DELETE	/api/node/:id	Deregister self

The worker tunnel at GET /api/cluster/connect is also worker-facing but is authenticated differently: against the node's own stored token rather than the shared registration token. See Worker tunnels.

/api/nodes/ - Admin management

Used by the WebUI and admin API consumers. Requires admin authentication.

Method	Path	Description
GET	/api/nodes	List all registered workers
GET	/api/nodes/:id	Get a single worker by ID
GET	/api/nodes/:id/models	List models loaded on a worker
DELETE	/api/nodes/:id	Admin-delete a worker
POST	/api/nodes/:id/drain	Admin-drain a worker
POST	/api/nodes/:id/approve	Approve a pending worker node
POST	/api/nodes/:id/backends/install	Install a backend on a worker
POST	/api/nodes/:id/backends/upgrade	Upgrade (force-reinstall) a backend on a worker
POST	/api/nodes/:id/backends/delete	Delete a backend from a worker
POST	/api/nodes/:id/models/unload	Unload a model from a worker
POST	/api/nodes/:id/models/delete	Delete model files from a worker
PUT	/api/nodes/:id/vram-budget	Set a VRAM budget for a worker ({"value":"80%"})
DELETE	/api/nodes/:id/vram-budget	Clear a worker's VRAM budget (revert to all detected VRAM)

The Nodes page in the React WebUI provides a visual overview of all registered workers, their statuses, and loaded models. The page opens with a one-line cluster pulse summarising node health and an attention callout that surfaces nodes needing action (for example pending approvals). Below that, a roster of node panels lists each worker with its inline model chips (no expand click needed), filtered by an All / Backend / Agent segmented control. Selecting a panel opens a dedicated node detail page at /app/nodes/:id with per-node metrics, models, and backend actions. Model scheduling lives on its own Scheduling page (separate nav item), not as a tab on the Nodes page.

Model sizing in the WebUI

The model gallery answers "will this model run here" against the cluster, not against the frontend. A distributed frontend is usually a GPU-less pod, so sizing models against its own memory would report that a fleet of GPU workers can only run the smallest CPU build.

The budget is the largest single healthy backend node, not the sum of the fleet: a model loads into one node, so four 16GB workers do not add up to a home for a 40GB model. A node's operator-set VRAM budget caps its contribution, since the scheduler would refuse a load above that ceiling anyway, and a GPU node wins over a CPU node holding more system RAM. The gallery names the node its verdict belongs to ("Fits on dgx-01").

GET /api/resources and GET /api/models carry this as an additional cluster object; their existing aggregate and ram* fields keep reporting the frontend's own hardware, which is what the resource monitor shows. The object is absent in single-node mode, and also whenever the registry cannot be read, in which case every sizing surface falls back to the local host:

{
  "cluster": {
    "enabled": true,
    "node_id": "a1b2c3",
    "node_name": "dgx-01",
    "total_memory": 85899345920,
    "is_gpu": true,
    "node_count": 4
  }
}

Variant selection (GET /api/models/variants/:id) uses the same reading, and judges backend compatibility against the union of the capabilities present in the cluster, so a CUDA-only build is offered when any worker can run it.

Model configuration revisions

Distributed mode assigns a config_revision to each validated model configuration. It hashes the persisted semantic configuration, including fields such as context_size and parallel settings. YAML formatting, comments, and map order do not change it.

The first request for a model establishes its current revision and replay information. The replica reconciler uses only replay information that matches the current revision. This lets min_replicas recover after an ordinary worker failure without restoring an old configuration.

When you save a valid model edit, LocalAI makes replicas from the old revision ineligible immediately. New requests cannot route to those replicas. This rule applies to raw YAML edits, structured patches, renames, disabled models, and changes from another frontend.

The edit response includes these fields:

• config_revision identifies the saved semantic configuration.
• pending_cleanup counts old replicas that still need cleanup when the response returns.

LocalAI sends an acknowledged stop request for each exact backend process, over that worker's tunnel. If the worker is unreachable, LocalAI keeps the replica in the unloading state and retries with durable backoff. The saved edit remains successful while cleanup is pending.

Workers must support the exact model-stop protocol. Upgrade all workers before you rely on revision cleanup. An older worker cannot acknowledge the request, so its stale replica remains unloading until cleanup succeeds or the worker re-registers.

Worker re-registration removes stale live-replica rows, but it preserves the current model revision and matching replay information. A temporary worker outage therefore does not make an old revision routable. The reconciler can restore the current revision after the worker becomes healthy.

The responses from GET /api/node/:id/models and GET /api/nodes/:id/models include these replica fields:

Field	Meaning
config_revision	Hash of the persisted semantic model configuration that created the replica. Routable replicas match the current revision.
effective_options_hash	Hash of the final node-specific load options after defaults and file staging have been applied. Different hashes can be valid on heterogeneous workers when config_revision matches.
state	Replica lifecycle state, such as staging, loading, loaded, or unloading. Only eligible loaded replicas receive requests.
cleanup_error	Last exact-stop error. This field appears while cleanup is pending.
cleanup_next_retry_at	Time of the next durable cleanup attempt. This field appears after a failed attempt.

model.unload releases model memory inside a running backend. It does not replace the exact process stop that configuration cleanup requires. The backend.stop operation remains an administrative backend operation.

Per-node VRAM budget

Each worker advertises its detected VRAM, and the SmartRouter uses that number when picking a node with enough free memory. You can cap the VRAM a node offers for placement so it never gets scheduled beyond a chosen limit, leaving headroom for other workloads on that machine.

There are two ways to set the cap:

• At the worker: start it with --vram-budget / LOCALAI_VRAM_BUDGET (see Worker Configuration).
• From the frontend, live: set it per node in the node capacity editor on the node detail page, or via the admin API:

# Cap node placement at 80% of its detected VRAM
curl -X PUT http://frontend:8080/api/nodes/<node-id>/vram-budget \
  -H "Authorization: Bearer <admin-token>" \
  -H "Content-Type: application/json" \
  -d '{"value":"80%"}'

# Or an absolute amount
curl -X PUT http://frontend:8080/api/nodes/<node-id>/vram-budget \
  -H "Authorization: Bearer <admin-token>" \
  -d '{"value":"12GB"}'

# Clear the budget (revert to all detected VRAM)
curl -X DELETE http://frontend:8080/api/nodes/<node-id>/vram-budget \
  -H "Authorization: Bearer <admin-token>"

The value accepts the same formats as the standalone budget: a percentage (80%) or an absolute amount (12GB, 12GiB, 12000MB, or raw bytes). It is a hard ceiling: the node's advertised VRAM becomes min(detected, budget), so a budget can only lower the number, never raise it above the hardware. An admin-set node budget is sticky across worker restarts: it is stored in the node registry and reapplied when the worker re-registers, so it wins over whatever the worker reports on reconnect. For the underlying semantics and the standalone equivalent, see [VRAM Budget]({{%relref "advanced/vram-management#vram-budget-allocation-ceiling" %}}).

Disk headroom

Model weights are staged onto the worker's disk before the backend loads them, so a node needs free space as well as free VRAM. Each worker reports the capacity of the filesystem backing its models directory (--models-path), not the root filesystem, on registration and on every heartbeat. Those figures appear as total_disk and available_disk in the nodes API and as Models disk free on the node detail page.

Before placing a model, the SmartRouter removes any node whose models filesystem cannot hold it. The requirement is derived from the model's actual on-disk size plus a small margin (5%, at least 1 GiB), rather than a fixed percentage of the node's disk — a fixed threshold would take a small-but-usable node out of rotation for models it could comfortably store. When the model's size cannot be determined locally (a bare HuggingFace repo id that the worker fetches itself), the node only has to clear a 2 GiB floor.

If no node has enough space, the request fails immediately with a capacity error naming the requirement and each node's free space, for example:

scheduling longcat-video-avatar-1.5: no node has enough free disk for the model:
need 73.5 GB free on the models filesystem, but nvidia-thor has 0 B free of 937.0 GB

This is deliberately a scheduling-time verdict. Without it, a worker with a full disk still reported status: healthy, accepted the staging request, transferred tens of gigabytes and only then failed with no space left on device — minutes after a decision that could never have succeeded.

Workers that predate this feature (or whose disk reading fails) report total_disk as 0. Such nodes are treated as unknown, not full, and stay in rotation, so a rolling upgrade never empties the candidate pool. A full disk is distinguishable because it reports a non-zero total_disk with available_disk at 0.

Low disk does not mark a node unhealthy. Disk is compared per model rather than against a global threshold, so a node that is too small for one model remains a valid target for smaller ones. The check is also skipped entirely in shared-models mode, where nothing is staged to the worker at all.

Turning the check off

The check is on by default. To disable it, start the frontend with --distributed-disk-headroom-check=false / LOCALAI_DISTRIBUTED_DISK_HEADROOM_CHECK=false, or toggle Settings → Distributed → Disk headroom check in the WebUI (distributed_disk_headroom_check via POST /api/settings). The runtime setting takes effect on the next placement, with no restart; the env/CLI flag only sets the value LocalAI boots with, and both write the same underlying value, so the last change wins.

Disabling means warn, do not block. Node selection goes back to ignoring free disk (the pre-check behaviour), but the check still runs, and when it would have rejected every node it logs a warning naming the shortfall:

WARN No node has room to store this model, but the disk-headroom check is DISABLED;
     scheduling anyway — staging will most likely fail with ENOSPC
     model=longcat-video-avatar-1.5 knob=distributed-disk-headroom-check

The alternative — skipping the check outright — was rejected because it reproduces the condition that made the original bug expensive: the cluster was doing something that could not work and said nothing about it. The escape hatch exists for setups where the size estimate is wrong (deduplicating or compressing filesystems, a backend that fetches its own weights rather than using the staged copy), and in exactly those cases the operator needs to see what LocalAI thought was wrong. Disabling is logged once at startup as well.

The LocalAI Assistant can also set a node budget conversationally through the set_node_vram_budget MCP tool.

Node Approval

By default, new worker nodes start in pending status and must be approved by an admin before they can receive traffic. This prevents unknown machines from joining the cluster.

To approve a pending node via the API:

curl -X POST http://frontend:8080/api/nodes/<node-id>/approve \
  -H "Authorization: Bearer <admin-token>"

The Nodes page in the WebUI also shows pending nodes with an Approve button.

To skip manual approval and let nodes join immediately, set --auto-approve-nodes (or LOCALAI_AUTO_APPROVE_NODES=true) on the frontend. This is convenient for development and trusted environments.

Node Statuses

Status	Meaning
pending	Node registered but waiting for admin approval (when --auto-approve-nodes is false)
healthy	Node is active and responding to heartbeats
unhealthy	Node has missed heartbeats beyond the threshold (detected by the HealthMonitor)
offline	Node is temporarily offline (graceful shutdown or stale heartbeat). The node row is preserved so re-registration restores the previous approval status without requiring re-approval
draining	Node is shutting down gracefully - no new requests are routed to it, existing in-flight requests are allowed to complete

Agent Workers

Agent workers are dedicated processes for executing agent chats and MCP CI jobs. Unlike backend workers (which run gRPC model inference), agent workers use cogito to orchestrate multi-step conversations with tool calls.

local-ai agent-worker \
  --register-to http://frontend:8080 \
  --registration-token changeme

Agent workers:

• Execute agent chat messages dispatched to it as streaming control verbs on its tunnel
• Run MCP CI jobs (with access to MCP servers via docker)
• Handle MCP tool discovery and execution requests, which the frontend sends over the worker's own tunnel
• Get auto-provisioned API keys during registration for calling the inference API

In the docker-compose setup, the agent worker mounts the Docker socket so it can run MCP stdio servers (e.g., docker run commands):

agent-worker-1:
  command: agent-worker
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

MCP in Distributed Mode

MCP servers configured in model configs work in distributed mode. The frontend holds no MCP sessions of its own - creating one usually means running docker, which is what an agent worker is for - so it asks an agent worker instead:

• MCP discovery (GET /v1/mcp/servers/:model): the frontend picks a connected agent worker and asks it over that worker's tunnel; the worker creates the sessions and returns server info
• MCP tool execution (during /v1/chat/completions): the same, per tool call
• MCP CI jobs: executed entirely on agent workers with access to docker for stdio-based MCP servers

How a frontend picks an agent worker

Discovery and tool execution used to be NATS request-reply onto a queue group, where the broker chose the worker and neither side could say which one had answered. They are now an ordinary control RPC plus a selection, because a queue group was only ever a way of choosing a subscriber, and choosing is a query:

1. The frontend lists the approved agent nodes that are not draining.
2. It asks the node_connections table, in one statement joined against live replicas, which of those tunnels a live frontend replica currently holds.
3. It prefers one this replica holds, so the call skips the relay hop entirely, and otherwise takes any connected one at random. A broker's hidden balancing could not make that choice.
4. It issues the control RPC over that worker's tunnel, relayed through the owning replica when another one holds it.

A worker that answers with an error - "no such tool", "that MCP server refused your arguments" - is the worker's own answer and is returned to you unchanged; it is never re-tried on a second worker, because that would run a tool twice. A call that never reached a worker is re-tried, against a different worker, at most three times.

If no agent worker in the deployment currently holds a tunnel, the request fails with a message saying so. That is a statement about this moment, not about any particular worker: nothing is marked unhealthy and no model is evicted because of it.

MCP prompts and resources are not available in distributed mode

GET /v1/mcp/prompts/:model, POST /v1/mcp/prompts/:model/:prompt, GET /v1/mcp/resources/:model and POST /v1/mcp/resources/:model/read are served only from MCP sessions held by the frontend process, and in distributed mode it holds none. There is no verb that carries prompts or resources to an agent worker.

In distributed mode these four endpoints answer 501 Not Implemented with the reason in the body. Earlier releases answered 200 with an empty list, which was indistinguishable from a model that genuinely has no prompts. This is a pre-existing gap rather than a consequence of moving MCP off the bus - tools and discovery had a carrier to an agent worker and these never did - and single-binary deployments are unaffected.

vLLM Multi-Node (Data-Parallel)

A single vLLM model can span multiple GPU nodes via data parallelism: the head node serves the OpenAI API and runs the local DP ranks, follower nodes run vanilla vllm serve --headless and speak ZMQ directly to the head. LocalAI's role is starting the follower processes and surfacing them in the admin UI; the cross-rank tensor traffic is vLLM's own.

This mode is operator-launched - the head config and each follower's invocation must agree on the topology (data_parallel_size, data_parallel_size_local, data_parallel_address, data_parallel_rpc_port). The SmartRouter does not place follower ranks automatically.

Head node configuration

The head runs the existing single-node vLLM gRPC backend. Set engine_args to publish the DP topology vLLM expects:

backend: vllm
parameters:
  model: moonshotai/Kimi-K2.6-Instruct
engine_args:
  data_parallel_size: 4              # total ranks across all nodes
  data_parallel_size_local: 2        # ranks on the head node
  data_parallel_address: 10.0.0.1    # head's reachable IP
  data_parallel_rpc_port: 32100      # any free port; followers connect here
  enable_expert_parallel: true       # for MoE models

The head will start its 2 local ranks, listen on 10.0.0.1:32100, and wait for the remaining 2 ranks to handshake.

Follower nodes

Each follower runs local-ai p2p-worker vllm with matching topology, an explicit start rank, and the head's address:

local-ai p2p-worker vllm \
  moonshotai/Kimi-K2.6-Instruct \
  --data-parallel-size 4 \
  --data-parallel-size-local 2 \
  --start-rank 2 \
  --master-addr 10.0.0.1 \
  --master-port 32100 \
  --register-to http://frontend:8080 \
  --registration-token changeme

--register-to is optional but recommended - it makes the follower visible in the admin UI as an agent-type node tagged with node.role=vllm-follower. Without it the worker just runs vLLM and exits silently when vLLM does. The role label discourages SmartRouter from placing other models on the follower; pair it with model selectors like {"!node.role":"vllm-follower"} if you also run regular LocalAI models on the same fleet.

Worked example: 2-node Kimi-K2.6 deployment

Two A100 nodes (10.0.0.1, 10.0.0.2), 8 GPUs total, data_parallel_size=8 with 4 ranks per node:

# /models/kimi.yaml on the head (10.0.0.1)
name: kimi-k2-6
backend: vllm
parameters:
  model: moonshotai/Kimi-K2.6-Instruct
engine_args:
  data_parallel_size: 8
  data_parallel_size_local: 4
  data_parallel_address: 10.0.0.1
  data_parallel_rpc_port: 32100
  enable_expert_parallel: true
  all2all_backend: deepep_high_throughput

# On 10.0.0.2 (follower)
local-ai p2p-worker vllm moonshotai/Kimi-K2.6-Instruct \
  --data-parallel-size 8 --data-parallel-size-local 4 --start-rank 4 \
  --master-addr 10.0.0.1 --master-port 32100 \
  --register-to http://10.0.0.1:8080 --registration-token changeme

A curl http://10.0.0.1:8080/v1/chat/completions ... against the head will then dispatch across all 8 ranks.

Intel Arc / XPU notes

vLLM XPU supports DP (vllm/platforms/xpu.py:198 handles world_size_across_dp > 1; ranks bind to xpu:{local_rank} in xpu_worker.py:62, with xccl as the collective backend). Each rank still needs a distinct discrete GPU - the iGPU on a hybrid host is not a viable second device.

Older XE-HPG GPUs (e.g. Arc A770) need to bypass the cutlass attention path:

engine_args:
  attention_backend: TRITON_ATTN

docker-compose.vllm-multinode.intel.yaml at the repo root is the Intel equivalent of docker-compose.vllm-multinode.yaml - uses /dev/dri passthrough, ZE_AFFINITY_MASK to pin each rank to one device, and latest-gpu-intel images. Run via ./tests/e2e/vllm-multinode/smoke.sh --intel.

Caveats

• Tensor parallel within a node only. vLLM v1 does not support TP across nodes; combine tensor_parallel_size (within a node, via engine_args) with data_parallel_size (across nodes).
• Followers don't host LocalAI gRPC. The follower process is vanilla vLLM, so /api/backend-logs/<modelId> does not stream follower output. Use journalctl / kubectl logs / compose logs for the follower's stderr.
• Network reachability. The head's data_parallel_rpc_port plus a range of ZMQ ports (typically data_parallel_rpc_port..+N) must be reachable from every follower. Open them in your firewall / security group.
• Topology must match exactly. A mismatch in --data-parallel-size between head and any follower will hang the handshake. Check the head's vLLM logs for waiting for N DP ranks if startup stalls.

ds4 Layer-Split Distributed Inference

The ds4 backend (DeepSeek V4 Flash) supports layer-parallel distributed inference: a single model that is too large for one machine is split by transformer layer across several machines. Each machine must have the GGUF present locally, but loads only its own slice of the layers. This lets you run a model whose weights exceed any single host's memory.

This is not routed through the SmartRouter: it is a model-internal split, configured manually (Phase 1). It is unrelated to the PostgreSQL-backed distributed mode described above.

Topology

ds4 uses a coordinator/worker split:

• The coordinator owns tokenization, sampling, the prompt, and a low layer range (e.g. 0:19). It is LocalAI's ds4 backend and listens on a host/port. Workers dial into it.
• One or more workers own higher layer ranges (e.g. 20:output). Each worker loads only its slice and dials the coordinator to register the range it can serve. The last worker normally owns the output head.
• Activations flow through the connected slices and back to the coordinator. The route is "ready" only once the coordinator plus all connected workers cover every layer.

This dial direction is the inverse of the llama.cpp RPC model, where the main server dials out to a list of rpc-server workers. With ds4 the workers dial in to the coordinator.

Coordinator setup

The coordinator is a normal LocalAI ds4 model whose YAML carries distributed options::

name: ds4flash
backend: ds4
options:
  - "ds4_role:coordinator"
  - "ds4_layers:0:19"
  - "ds4_listen:0.0.0.0:1234"

Option	Meaning
ds4_role:coordinator	Enables distributed coordinator mode. Without ds4_role, the backend behaves as a normal single-node ds4 model.
ds4_layers:0:19	The coordinator's own layer slice (inclusive).
ds4_listen:0.0.0.0:1234	Address that workers dial into.
ds4_route_timeout:60	Optional. Seconds the coordinator waits for the worker route to form before returning an error on a request. Defaults to 60.

{{% notice warning %}} Worker↔coordinator traffic is plaintext and unauthenticated: there is no TLS or auth on this channel. Bind ds4_listen to an address on a trusted/private network only; using 0.0.0.0 exposes the coordinator on every interface. Run the layer split exclusively over a network you control. {{% /notice %}}

Once the model is loaded, the coordinator serves requests exactly like a single-node ds4 model: generation goes through the ordinary inference path and is transparently routed across the layer slices.

Worker setup

On each worker machine (with the GGUF present locally), start a worker pointed at the coordinator:

local-ai worker ds4-distributed -- \
  --role worker \
  --model /models/ds4flash.gguf \
  --layers 20:output \
  --coordinator <coordinator-host> 1234

local-ai worker ds4-distributed resolves the ds4 backend and execs the packaged ds4-worker binary, passing everything after -- straight through.

Layer-range semantics

• Ranges are inclusive: 0:19 is layers 0 through 19.
• N:output means layer N through the final layer plus the output head. The last worker normally owns the output head.
• The coordinator and all connected workers together must cover every layer. Until they do, the coordinator returns a gRPC UNAVAILABLE error on inference requests (so a worker that starts slightly after the coordinator is tolerated: once it connects and the route is complete, requests succeed). The wait is tunable via ds4_route_timeout.

{{% notice note %}} ds4 layer-split inference is manual setup in this release (Phase 1): you place the coordinator config and launch each worker yourself, and the layer ranges must be partitioned by hand so they cover the whole model. P2P auto-discovery of the coordinator is planned for a later phase. {{% /notice %}}

Scaling

Adding worker capacity: Start additional worker instances pointing to the same frontend. They self-register automatically:

# Additional workers - no backend type needed
local-ai worker \
  --register-to http://frontend:8080 \
  --node-name worker-2 \
  --registration-token changeme

local-ai worker \
  --register-to http://frontend:8080 \
  --node-name worker-3 \
  --registration-token changeme

Multiple frontend replicas: Run multiple LocalAI frontends behind a load balancer. Since all state is in PostgreSQL and coordination is via PostgreSQL and the workers' own tunnels, frontends are fully stateless and interchangeable.

Model Scheduling

Model scheduling controls where models are placed and how many replicas are maintained. In the React WebUI it has its own Scheduling page (a top-level nav item, separate from the Nodes page). It combines two optional features:

Node Selectors

Pin models to nodes with specific labels. Only nodes matching all selector labels are eligible:

# Only schedule on NVIDIA nodes in the us-east zone
curl -X POST http://frontend:8080/api/nodes/scheduling \
  -H "Content-Type: application/json" \
  -d '{"model_name": "llama3", "node_selector": {"gpu.vendor": "nvidia", "zone": "us-east"}}'

Without a node selector, models can schedule on any healthy node (default behavior).

In the WebUI, the node selector field completes what you type against the labels your cluster actually reports: start typing a key and the matching label keys appear inline, then the value field offers only the values that key takes. A key no node reports yet is still accepted as typed, so you can write a rule before labelling the nodes for it.

Replica Auto-Scaling

Control the number of model replicas across the cluster:

Field	Description
min_replicas	Minimum replicas to maintain (0 = no minimum, single replica default)
max_replicas	Maximum replicas allowed (0 = unlimited)

Auto-scaling is only active when min_replicas > 0 or max_replicas > 0.

# Scale llama3 between 2 and 4 replicas on NVIDIA nodes
curl -X POST http://frontend:8080/api/nodes/scheduling \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "llama3",
    "node_selector": {"gpu.vendor": "nvidia"},
    "min_replicas": 2,
    "max_replicas": 4
  }'

The Replica Reconciler runs as a background process on the frontend:

• Scale up: Adds replicas when all existing replicas are busy (have in-flight requests)
• Scale down: Removes idle replicas after 5 minutes of inactivity
• Maintain minimum: Ensures min_replicas are always loaded (recovers from node failures)
• Eviction protection: Models with auto-scaling enabled are never evicted below min_replicas
• Restart-safe: Per-model load metadata (backend type + ModelOptions) is persisted in the model_load_infos PostgreSQL table on the first successful dispatch, so a frontend restart or rolling upgrade does not require a fresh inference request to repopulate state before the reconciler can scale up replacement replicas.

All fields are optional and composable:

• Node selector only: pin model to matching nodes, single replica
• Replicas only: auto-scale across all nodes
• Both: auto-scale on matching nodes only

Scheduling a model alias

model_name accepts a model alias as well as a model. A rule keyed by an alias governs whatever model that alias currently points at, and keeps governing it after you repoint the alias:

# "production" is an alias for llama3
curl -X POST http://frontend:8080/api/nodes/scheduling \
  -H "Content-Type: application/json" \
  -d '{"model_name": "production", "node_selector": {"tier": "gpu"}, "min_replicas": 2}'

# Repoint the alias at a new model: the rule follows, llama4 now runs
# two replicas on the GPU tier and llama3 falls back to on-demand placement.

This makes an alias a stable deployment slot: the placement policy belongs to the slot, and the model filling it can change without rewriting the rule. The WebUI lists aliases in the model picker on the Scheduling page, tagged with the model each one resolves to.

Two constraints follow from replicas being shared. A single load of llama3 serves both production and any request that names llama3 directly, so only one rule can decide where it runs: a rule whose target is already governed by another rule is rejected with 409 Conflict naming the rule that has it. And a rule keyed by an alias that resolves to nothing (its target was deleted, or it points at another alias) is rejected, since it would govern nothing loadable.

A rule can still end up inert if the pair is created some other way, for example by a declarative seed or by repointing an alias onto a model that already has a rule. The rule that governs is the one keyed by the model's own name, or failing that the oldest one; the rest are listed as Shadowed in the WebUI and carry "shadowed": true in GET /api/nodes/scheduling.

Declarative per-model scheduling (unattended installs)

In distributed mode you can declare per-model scheduling at startup, instead of using the WebUI/API. Config is authoritative: it is re-applied on every boot and overwrites the listed models (models not listed are left untouched).

Variable	Description
LOCALAI_MODEL_SCHEDULING	Inline JSON list of scheduling entries
LOCALAI_MODEL_SCHEDULING_CONFIG	Path to a YAML file with the same list

Entry fields: model_name (required), node_selector (a label map; omit it to match every node), and then one of two replica modes (they are mutually exclusive):

• replicas: all - static spread: place exactly one replica on every matching node, proactively, regardless of load, and keep it in sync as nodes join and leave. Use this for "run model X everywhere (with this label)".
• min_replicas / max_replicas - elastic auto-scaling: keep at least min_replicas running, and burst up to max_replicas only when all replicas are busy, scaling back down to the minimum when idle. max_replicas: 0 means no upper bound (grow to cluster capacity). To enable this mode you must set min_replicas >= 1 or max_replicas >= 1 - an entry with only max_replicas: 0 (and no replicas: all) does nothing.

Net effect at a glance:

Config	Behavior
replicas: all	One replica per matching node, placed immediately, tracks join/leave
min_replicas: 1, max_replicas: 0	Always >=1, bursts to cluster capacity under load, back to 1 when idle
min_replicas: 2, max_replicas: 4	Always >=2, bursts to at most 4 under load

node_selector constrains which nodes a model may use; with no selector the model may use all healthy nodes. So "spread model X across all nodes" is just replicas: all with no node_selector. replicas: all targets one replica per matching node; with the default per-node cap of one replica per model this lands exactly one on each node (see the note below about LOCALAI_MAX_REPLICAS_PER_MODEL).

YAML example (scheduling.yaml):

# One replica on every GPU-labelled node (static spread, tracks join/leave):
- model_name: gpt-oss
  node_selector:
    tier: gpu
  replicas: all

# One replica on EVERY node in the cluster (no selector = all nodes):
- model_name: embeddings
  replicas: all

# Elastic on CPU nodes: always >=1, burst to capacity under load, 0 = no cap:
- model_name: whisper
  node_selector:
    tier: cpu
  min_replicas: 1
  max_replicas: 0

LOCALAI_DISTRIBUTED=true \
LOCALAI_MODEL_SCHEDULING_CONFIG=/etc/localai/scheduling.yaml \
local-ai run

Inline equivalent:

LOCALAI_MODEL_SCHEDULING='[{"model_name":"gpt-oss","node_selector":{"tier":"gpu"},"replicas":"all"}]'

Notes:

• Because the config is authoritative, each listed model's entire scheduling row is replaced on every boot, including the optional prefix-cache routing overrides (route_policy, balance_abs_threshold, balance_rel_threshold, min_prefix_match). For a model you manage via this config, set those fields here too if you need non-default values; values set only through the API are reset on the next restart. Models not listed in the config are never touched.
• replicas: all places one replica per matching node by relying on the default per-node cap of one replica per model. If you raise LOCALAI_MAX_REPLICAS_PER_MODEL on a worker above 1, the target count can be met by stacking replicas on fewer nodes rather than spreading one to each.

Label Management API

Method	Path	Description
GET	/api/nodes/:id/labels	Get labels for a node
PUT	/api/nodes/:id/labels	Replace all labels (JSON object)
PATCH	/api/nodes/:id/labels	Merge labels (add/update)
DELETE	/api/nodes/:id/labels/:key	Remove a single label

Scheduling API

Method	Path	Description
GET	/api/nodes/scheduling	List all scheduling configs
GET	/api/nodes/scheduling/:model	Get config for a model
POST	/api/nodes/scheduling	Create/update config
DELETE	/api/nodes/scheduling/:model	Remove config

Comparison with P2P

	P2P / Federation	Distributed Mode
Discovery	Automatic via libp2p token	Self-registration to frontend URL
State storage	In-memory / ledger	PostgreSQL
Coordination	Gossip protocol	Each worker's own tunnel for every verb addressed to it, agent workers included; PostgreSQL LISTEN/NOTIFY for cross-replica frontend events
Node management	Automatic	REST API + WebUI
Health monitoring	Peer heartbeats	Centralized HealthMonitor
Backend management	Manual per node	Dynamic via the worker's backend.install control route
Best for	Ad-hoc clusters, community sharing	Production, Kubernetes, managed infrastructure
Setup complexity	Minimal (share a token)	Requires PostgreSQL on the frontend, and nothing else. Workers of either kind need only an outbound route to the frontend URL.

Troubleshooting

Worker not registering:

• Verify the frontend URL is reachable from the worker (curl http://frontend:8080/api/node/register)
• Check that --registration-token matches on both frontend and worker
• Ensure auth is enabled on the frontend (LOCALAI_AUTH=true)

Message-broker connection errors:

• Nothing in LocalAI connects to a broker any more, on any component. A release that logs such an error predates the tunnel migration; on this release, look at the failing component's tunnel and its --register-to instead. See Migrating off the message broker.

PostgreSQL connection errors:

• Verify the connection URL format: postgresql://user:password@host:5432/dbname?sslmode=disable
• Ensure the database exists and the user has CREATE TABLE permissions (for auto-migration)
• Check that pgvector extension is installed if using RAG features

Node shows as unhealthy or offline:

• The HealthMonitor marks nodes offline when heartbeats are missed. Check network connectivity between worker and frontend.
• Verify --heartbeat-interval is not set too high
• Offline nodes automatically restore to healthy when they re-register (no re-approval needed)

Backend not installing:

• Check the worker logs for backend.install events

Requests still report an old context size or another old load option:

• Query /api/nodes/:id/models for every worker that hosts the model.
• Confirm that every routable replica has state: loaded and the same current config_revision.
• Treat a different effective_options_hash as diagnostic information. Node-specific defaults can cause valid differences.
• Check cleanup_error and cleanup_next_retry_at on replicas in the unloading state.
• Check that the worker's tunnel is up when cleanup reports a timeout or no route.
• Upgrade the worker when it does not support the exact model-stop request.
• Stop and restart the stale backend only as an operational recovery action. LocalAI keeps it non-routable while durable cleanup is pending.

A model cannot be scheduled on a node that looks free (no replica slot ... all models busy, cannot evict):

• A replica row in staging or loading holds its slot: slot allocation counts every state except unloading. If a worker drops out mid-transfer, that row never reaches loaded, and eviction only ever considers loaded replicas, so on a node with one replica slot per model the model became unschedulable there.
• The reconciler now reclaims a replica row stuck before serving when no load job is still driving it, and the freed slot is immediately reusable.
• Liveness is decided by the load job's progress heartbeat, not by elapsed time. Staging a large checkpoint legitimately runs for a long time without touching the replica row, so a transfer that is still progressing is never reclaimed however long it takes.
• Reconciler: reclaimed a replica slot held by a load nobody is driving names each row reclaimed this way.

A request fails with this frontend has no route to that worker:

• The chosen worker's tunnel was not reachable from the replica that handled the request. A node's status comes from its HTTP heartbeat, which is a separate channel: a worker that stops stays healthy until that heartbeat ages out, and a worker that is very much alive can be unroutable for a moment while its tunnel re-homes between frontend replicas.
• It is not the same as the worker being gone, and nothing acts on it as if it were. A model on an unroutable worker is not reaped, its rows are left alone, and the node is not demoted: doing any of those on a lost route is how a rolling frontend restart turns into a fleet-wide eviction.
• Check the worker process is running and that it has an open tunnel (opened a tunnelled stream to a worker in the frontend log, and the worker's own dial/reconnect lines). A worker behind a load balancer that keeps reconnecting is usually an idle-timeout or WebSocket-upgrade problem at the proxy; see the tunnel section above.
• no route is not gone, and nothing in the frontend reads it as such. A worker is declared gone by one mechanism only: no live frontend replica holds its tunnel and its departure is older than --worker-reconnect-grace. That is a fact recorded in the shared database, so every replica answers it identically. "No route" is one replica failing to reach a worker right now, and it is not evidence about the worker at all.
• Older releases decided absence from nats: no responders available for request, which was one frontend's observation that nobody answered it within a request budget. Two replicas asking at the same moment could disagree and demote each other's workers. That signal is gone from the scheduler, and no component opens a bus connection to produce it.

A worker fills its own disk over time:

• A request that carries a file (an image, an audio clip, a video) stages that file to the worker under <models>/../staging/ephemeral/. The worker deletes these 6 hours after the request that needed them, and sweeps every 30 minutes plus once at startup, so a worker that crashed mid-request still reclaims the space.
• Releases before this sweep existed kept every staged input for the lifetime of the worker. Delete <models>/../staging/ephemeral/ on an affected worker once, as the user the worker runs as; the sweep keeps it bounded from then on.
• Staged model files are not touched by this. They live beside the ephemeral directory and are not per-request scratch.
• A worker whose volume is genuinely full reports creating backend process state directory under ...: no space left on device when a backend starts.

Requests fail with stale model config revision although nobody edited the model:

• A model's stored revision must describe its persisted configuration. Releases before this fix also hashed the per-request prediction parameters, so the first request after a restart pinned the revision to its own temperature, top_p, stop and similar values. Every later request that sent different values was then rejected.
• Upgrade the frontend replicas first. After the upgrade the revision is stamped when the configuration is loaded, so it no longer depends on the request body.
• Each frontend now reconciles the stored revisions against the configuration on disk at startup, and republishes any that disagree, so a drifted revision heals on the next restart. Only models that actually drifted are republished, because republishing quarantines the replicas loaded under the old revision.
• A model that has never been served has no stored revision and is left alone; its first request establishes one.
• On a release without that reconciliation, clear the row once per affected model so the next request establishes the correct revision: DELETE FROM model_config_states WHERE model_name = '<model>'; Saving any edit through the API or the WebUI has the same effect.

Port conflicts on workers:

• Each model gets its own gRPC process on an incrementing port (50051, 50052, ...)
• The HTTP file transfer server runs on the base port - 1 (default: 50050)
• All of those bind loopback, so a firewall cannot be the cause. What can is another service on the same host already holding a port in the range: move the worker's range with LOCALAI_ADDR (see Worker Port Configuration) or bound it with LOCALAI_GRPC_MAX_PORT
• Verify the backend gallery configuration is correct
• The worker needs OUTBOUND network access to the gallery and to LOCALAI_REGISTER_TO. It needs no inbound access at all

Roadmap: Routing and Caching Enhancements

The scheduling algorithm above is load-based (least in-flight, then least-recently-used). Work is underway to make routing prefix-cache-aware: bias each request toward the replica that already holds the relevant KV/prefix cache (multi-turn conversations and shared system prompts), so backends reuse cache instead of recomputing it. The first step is a router-side radix tree of prompt-prefix hashes mapped to nodes, with longest-prefix match, a load guard that preserves round-robin behavior under imbalance, and cross-frontend sync on the PostgreSQL broadcast carrier. It is purely a routing-layer hint (no backend changes) and never routes worse than today's round-robin.

Further enhancements, surfaced from a survey of SGLang, vLLM production-stack, Ray Serve, llm-d, AIBrix, and NVIDIA Dynamo, are tracked under the routing roadmap epic (#10063):

• Reported/precise KV-event mode (#10064): subscribe to actual backend KV-cache events for exact residency instead of inferring it from routing history.
• Multi-tier cache-overlap scoring (#10065): credit GPU/CPU/disk cache tiers separately.
• Pluggable scorer/filter/picker pipeline (#10066): composable multi-signal routing (cache, queue depth, KV utilization, latency).
• Load-shaping (#10067): anti-herding (softmax/temperature) and dispatch-time freshness.
• Prefill/decode disaggregation routing (#10068): route prefill and decode to separate pools with KV transfer.
• Per-user fairness (VTC) (#10069): balance per-user token usage against pod load.
• Minor tuning + MCP parity (#10070): per-model TTL override, probabilistic LRU updates, and MCP scheduling-config tool parity.