mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-22 06:04:55 -04:00
Ettore Di Giacinto
6ae1fe25dd
Review round 1 on the change that stopped workers listening. One blocking item
and seven notes.
LOCALAI_WORKER_TUNNEL=false was the blocking one, and the ruling was to make it
fatal rather than to correct the comment that still promised it fell back to the
advertised address. There is no fallback left: a worker on this branch
advertises nothing and binds only loopback, so turning the tunnel off leaves it
reachable by nothing while it registers, heartbeats and reports healthy, and the
scheduler keeps placing models on it. That is the worst available failure shape,
so a new Config.validateStartup refuses it before prefetch, registration and
NATS, while the worker is still invisible to the cluster. It absorbs the
pre-existing empty-registration-token check, which had the same shape and no
spec. The flag is kept rather than deleted so an operator who set it is told the
promise is gone instead of having the setting ignored, and the guard around
StartTunnel is removed, because a branch nothing can take reads as a supported
no-tunnel mode that does not exist.
The justification for erroring on an install that names no address was wrong,
and the review is right that this is the dangerous form of overclaiming, because
the conclusion holds and the mechanism does not. It said the resulting empty
target would be refused as an invalid stream and that the refusal would read as
the worker answering about its backend. Nothing in this repo branches on
cluster.ErrNoRoute, and nodes.unroutable treats any recorded dial error as
unroutable, so that refusal reaches every reap guard as ProbeUnknown and deletes
nothing. The site now stands on what holds, that an install naming no port
produced nothing routable and the failure belongs to the install rather than to
a later probe, and records the retracted claim so nobody re-derives it. This
retracts the same paragraph in the body of 1cf847f29.
The reviewer deleted the whole tryWarmPath unnamed-replica guard and the suite
stayed green, including the reservation release. It is specced now, and the
asymmetry the review asked about is decided at the site: the row stays, unlike
the sibling !alive branch which removes it. That branch has observed a backend
dead; this one has observed only that the row is unreadable, which says nothing
about whether a process is running, and the row is the last record that one
might be, since the acknowledged stop path refuses a stop whose ExpectedAddress
does not match and an empty one cannot be cleaned up through it either.
The cross-version wire claim rested on two struct tags nobody asserted:
renaming only the json keys survived mutation while the gorm column rename went
red through raw SQL. Both keys are pinned now, marshal and unmarshal, per
struct.
A worker-first upgrade showed the operator a status code and not the reason. The
registration client discarded the body, so "address is required for backend
workers" was read off the socket and thrown away, and the ladder then spent four
minutes on a verdict the frontend reached instantly. Refusals now quote the body
and carry ErrRegistrationRejected, and both the ladder and the credential
manager's Acquire stop on the first one. Acquire matters more than the ladder:
it is the default path and its bound is 100 attempts, not 10. 408 and 429 are
deliberately not refusals, since both are the frontend asking for the same
request again.
Also: the stale "not blocked by firewalls" troubleshooting line, which now names
the real cause and the knobs that move the port range; and the inert address
fields on the MCP Node DTO, which the Assistant was still being handed. The
review named http_address there and I removed address too, because it is inert
by the same argument and leaving one of a pair is arbitrary.
Five mutations, all red. Deleting the warm-path guard reddens four specs and
falsifying only its reservation release reddens one, so the two halves are
pinned separately. Renaming only the json keys reddens both wire suites.
Discarding the refusal body reddens two. Dropping the rejection classification
does not fail the suite, it hangs it, which is the operator-visible symptom, so
it is recorded red under a ginkgo timeout.
The verify list is now derived from the diff rather than from the brief, which
is what let the previous round ship a spec asserting 200 where the endpoint
returns 201: nine ginkgo suites, the e2e vet, route auth coverage, the leaf
check, build, the healthcheck shell suite and lint. The two jsx files have no
harness in this worktree and are recorded as the one unverified surface.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
152 lines
12 KiB
Go
152 lines
12 KiB
Go
package worker
|
||
|
||
import "fmt"
|
||
|
||
// Config is the configuration for the distributed agent worker.
|
||
//
|
||
// Field tags are kong/kong-env metadata read by core/cli/worker.go's WorkerCMD,
|
||
// which embeds Config; this package does NOT import kong and the tags are inert
|
||
// here.
|
||
//
|
||
// Workers are backend-agnostic — they wait for backend.install NATS events
|
||
// from the SmartRouter to install and start the required backend.
|
||
//
|
||
// NATS is required. The worker acts as a process supervisor:
|
||
// - Receives backend.install → installs backend from gallery, starts gRPC process, replies success
|
||
// - Receives backend.stop → stops the gRPC process
|
||
// - Receives stop → full shutdown (deregister + exit)
|
||
//
|
||
// Model loading (LoadModel) is always via direct gRPC — no NATS needed for that.
|
||
type Config struct {
|
||
// Addr and ServeAddr are read for their PORT only. A worker binds nothing
|
||
// on a routable interface: backend processes and the file-transfer server
|
||
// both listen on loopback and are reached through this worker's outbound
|
||
// tunnel. The port still matters because it is the base of the backend
|
||
// port range (and port-1 is the HTTP server), so an operator who needs a
|
||
// different range sets it here. The host half is ignored, and is kept
|
||
// accepted rather than rejected so an upgraded worker starts on the
|
||
// environment it already had.
|
||
Addr string `env:"LOCALAI_ADDR" help:"Base port for this worker, as host:port; only the port is used. Backends take ports upward from it, the HTTP file-transfer server takes port-1. Nothing binds a routable interface." group:"server"`
|
||
ServeAddr string `env:"LOCALAI_SERVE_ADDR" default:"0.0.0.0:50051" help:"(Advanced) gRPC base port; only the port is used" group:"server" hidden:""`
|
||
|
||
// GRPCMaxPort bounds the dynamic gRPC port allocator at [basePort, this].
|
||
// The width of that range is how many backend processes this worker can run
|
||
// concurrently; released ports also sit in a short quarantine before reuse,
|
||
// so a worker with heavy start/stop churn needs headroom above its true
|
||
// concurrency. 0 = up to 65535.
|
||
GRPCMaxPort int `env:"LOCALAI_GRPC_MAX_PORT" default:"0" help:"Highest port the worker may assign to a backend gRPC process. The range is [base port, this]; its width caps concurrent backends on this worker. 0 uses up to 65535." group:"server"`
|
||
|
||
BackendsPath string `env:"LOCALAI_BACKENDS_PATH,BACKENDS_PATH" type:"path" default:"${basepath}/backends" help:"Path containing backends" group:"server"`
|
||
BackendsSystemPath string `env:"LOCALAI_BACKENDS_SYSTEM_PATH" type:"path" default:"/var/lib/local-ai/backends" help:"Path containing system backends" group:"server"`
|
||
BackendGalleries string `env:"LOCALAI_BACKEND_GALLERIES,BACKEND_GALLERIES" help:"JSON list of backend galleries" group:"server" default:"${backends}"`
|
||
Galleries string `env:"LOCALAI_GALLERIES,GALLERIES" help:"JSON list of model galleries (used to resolve --prefetch-models on boot)" group:"server" default:"${galleries}"`
|
||
ModelsPath string `env:"LOCALAI_MODELS_PATH,MODELS_PATH" type:"path" default:"${basepath}/models" help:"Path containing models" group:"server"`
|
||
RequireBackendIntegrity bool `env:"LOCALAI_REQUIRE_BACKEND_INTEGRITY,REQUIRE_BACKEND_INTEGRITY" help:"If true, reject backend installs without a configured signature verification policy (OCI URIs) or SHA256 (tarball/HTTP URIs)." group:"hardening" default:"false"`
|
||
|
||
// PrefetchModels lets a worker download gallery model artifacts (GGUFs, etc.)
|
||
// from its own outbound internet at boot, instead of waiting for the master to
|
||
// stream them over the cluster network at first-inference time. Useful when the
|
||
// cluster-internal path is slow (slirp/circuit-relay, CGNAT) but outbound NAT
|
||
// works fine. Resolution reuses the same gallery installer the master uses, so
|
||
// the on-disk /models layout is identical. Errors are non-fatal — if the gallery
|
||
// is unreachable on boot, the worker logs a warning and starts the NATS loop
|
||
// anyway; the master can still push the file on demand (existing behaviour).
|
||
PrefetchModels []string `env:"LOCALAI_PREFETCH_MODELS,PREFETCH_MODELS" help:"Comma-separated gallery model IDs to download from LOCALAI_GALLERIES at worker boot (e.g. 'llama-3.2-1b-instruct,phi-3-mini-4k'). Skipped if already on disk and SHA matches." group:"server"`
|
||
|
||
// HTTPAddr binds the HTTP file-transfer server. Default is loopback on
|
||
// basePort-1; an explicit value is bound exactly as given.
|
||
HTTPAddr string `env:"LOCALAI_HTTP_ADDR" default:"" help:"HTTP file transfer server bind address (default: loopback on the gRPC base port - 1)" group:"server" hidden:""`
|
||
EphemeralStagingByteLimit int64 `env:"LOCALAI_EPHEMERAL_STAGING_BYTE_LIMIT" default:"0" help:"Maximum bytes used by worker request-input staging across HTTP and S3 caches. Zero or negative uses min(10 GiB, 10% of filesystem capacity)." group:"server"`
|
||
EphemeralStagingMinFreeBytes int64 `env:"LOCALAI_EPHEMERAL_STAGING_MIN_FREE_BYTES" default:"0" help:"Filesystem space kept free while staging request inputs. Zero or negative uses max(1 GiB, 5% of filesystem capacity)." group:"server"`
|
||
|
||
// Registration (required)
|
||
RegisterTo string `env:"LOCALAI_REGISTER_TO" required:"" help:"Frontend URL for registration" group:"registration"`
|
||
NodeName string `env:"LOCALAI_NODE_NAME" help:"Node name for registration (defaults to hostname)" group:"registration"`
|
||
RegistrationToken string `env:"LOCALAI_REGISTRATION_TOKEN" help:"Token for authenticating with the frontend" group:"registration"`
|
||
RegistrationRequireAuth bool `env:"LOCALAI_REGISTRATION_REQUIRE_AUTH" default:"false" help:"Refuse to start the HTTP file-transfer server when no registration token is set (otherwise it fails open and serves read/write to models/staging/data unauthenticated)" group:"registration"`
|
||
DistributedRequireAuth bool `env:"LOCALAI_DISTRIBUTED_REQUIRE_AUTH" default:"false" help:"Umbrella switch implying both --nats-require-auth and --registration-require-auth" group:"distributed"`
|
||
HeartbeatInterval string `env:"LOCALAI_HEARTBEAT_INTERVAL" default:"10s" help:"Interval between heartbeats" group:"registration"`
|
||
// WorkerTunnel holds one outbound multiplexed connection to the frontend
|
||
// and serves the frontend's requests over it, so the worker needs no
|
||
// inbound port.
|
||
//
|
||
// Turning it off is now a fatal misconfiguration and validateStartup
|
||
// refuses to boot on it, which is a behaviour change from when this flag
|
||
// had a working "off" position. It no longer has one: this worker
|
||
// advertises no address and binds only loopback, and no frontend path
|
||
// dials a worker's address, so a worker without its tunnel is reachable by
|
||
// nothing. Left running it would be the worst available failure shape,
|
||
// because it registers, heartbeats and reports healthy, so the scheduler
|
||
// keeps placing models on it and every one of them fails.
|
||
//
|
||
// The flag is kept rather than deleted so that an operator who set it, on
|
||
// the old promise that it fell back to the advertised address, is told
|
||
// exactly that the promise is gone instead of having their setting quietly
|
||
// ignored.
|
||
WorkerTunnel bool `env:"LOCALAI_WORKER_TUNNEL" default:"true" help:"Hold one outbound multiplexed tunnel to the frontend and serve its requests over it, so this worker needs no inbound port. Setting it false is refused: a worker has no other way to be reached." group:"distributed"`
|
||
NodeLabels string `env:"LOCALAI_NODE_LABELS" help:"Comma-separated key=value labels for this node (e.g. tier=fast,gpu=a100)" group:"registration"`
|
||
// MaxReplicasPerModel caps how many replicas of any one model can run on
|
||
// this worker concurrently. Default 1 = historical single-replica
|
||
// behavior. Set higher when a node has enough VRAM to host multiple
|
||
// copies of the same model (e.g. a fat 128 GiB box running 4× of a
|
||
// 24 GiB model for throughput). The auto-label `node.replica-slots=N`
|
||
// is published so model schedulers can target high-capacity nodes via
|
||
// the existing label selector.
|
||
MaxReplicasPerModel int `env:"LOCALAI_MAX_REPLICAS_PER_MODEL" default:"1" help:"Max replicas of any single model on this worker. Default 1 preserves single-replica behavior; set higher to allow stacking replicas on a fat node." group:"registration"`
|
||
|
||
// VRAMBudget optionally caps this node's VRAM for model allocation ("80%" or
|
||
// "12GB"). Reported to the server as a string; the server resolves and
|
||
// enforces it against the raw VRAM this worker reports. Empty = no cap.
|
||
VRAMBudget string `env:"LOCALAI_VRAM_BUDGET" help:"Cap VRAM used for model allocation on this worker node, as a percentage (e.g. 80%) or absolute amount (e.g. 12GB)." group:"registration"`
|
||
|
||
// NATS (required)
|
||
NatsURL string `env:"LOCALAI_NATS_URL" required:"" help:"NATS server URL" group:"distributed"`
|
||
NatsJWT string `env:"LOCALAI_NATS_JWT" help:"NATS user JWT override (normally from registration nats_jwt)" group:"distributed"`
|
||
NatsUserSeed string `env:"LOCALAI_NATS_USER_SEED" help:"NATS user signing seed override (normally from registration nats_user_seed)" group:"distributed"`
|
||
NatsRequireAuth bool `env:"LOCALAI_NATS_REQUIRE_AUTH" default:"false" help:"Require NATS JWT+seed from registration or env" group:"distributed"`
|
||
NatsTLSCA string `env:"LOCALAI_NATS_TLS_CA" type:"existingfile" help:"PEM file for NATS server CA (private PKI)" group:"distributed"`
|
||
NatsTLSCert string `env:"LOCALAI_NATS_TLS_CERT" type:"existingfile" help:"Client certificate for NATS mTLS" group:"distributed"`
|
||
NatsTLSKey string `env:"LOCALAI_NATS_TLS_KEY" type:"existingfile" help:"Client private key for NATS mTLS" group:"distributed"`
|
||
|
||
// S3 storage for distributed file transfer
|
||
StorageURL string `env:"LOCALAI_STORAGE_URL" help:"S3 endpoint URL" group:"distributed"`
|
||
StorageBucket string `env:"LOCALAI_STORAGE_BUCKET" help:"S3 bucket name" group:"distributed"`
|
||
StorageRegion string `env:"LOCALAI_STORAGE_REGION" help:"S3 region" group:"distributed"`
|
||
StorageAccessKey string `env:"LOCALAI_STORAGE_ACCESS_KEY" help:"S3 access key" group:"distributed"`
|
||
StorageSecretKey string `env:"LOCALAI_STORAGE_SECRET_KEY" help:"S3 secret key" group:"distributed"`
|
||
}
|
||
|
||
// NatsAuthRequired reports whether NATS JWT credentials must be present — the
|
||
// granular flag or the umbrella (LOCALAI_DISTRIBUTED_REQUIRE_AUTH).
|
||
func (c Config) NatsAuthRequired() bool {
|
||
return c.NatsRequireAuth || c.DistributedRequireAuth
|
||
}
|
||
|
||
// RegistrationAuthRequired reports whether a registration token must be set
|
||
// before the file-transfer server may start — the granular flag or the umbrella.
|
||
func (c Config) RegistrationAuthRequired() bool {
|
||
return c.RegistrationRequireAuth || c.DistributedRequireAuth
|
||
}
|
||
|
||
// validateStartup reports a configuration this worker must refuse to boot on,
|
||
// as opposed to one it can degrade under.
|
||
//
|
||
// It runs before prefetch, registration and NATS, so a refusal happens while
|
||
// the worker is still invisible to the cluster. That ordering is the point of
|
||
// checking here at all: both conditions below produce a worker that would
|
||
// register, heartbeat and be scheduled onto, so discovering them later means
|
||
// discovering them as failed inferences on a node the frontend believes is
|
||
// healthy.
|
||
func (c Config) validateStartup() error {
|
||
// The file-transfer server fails open on an empty token (see
|
||
// nodes.checkBearerToken), so enforcement plus no token is a request to
|
||
// serve the models directory unauthenticated.
|
||
if c.RegistrationAuthRequired() && c.RegistrationToken == "" {
|
||
return fmt.Errorf("registration auth is required (LOCALAI_REGISTRATION_REQUIRE_AUTH or LOCALAI_DISTRIBUTED_REQUIRE_AUTH) but LOCALAI_REGISTRATION_TOKEN is empty: refusing to start an unauthenticated file-transfer server")
|
||
}
|
||
if !c.WorkerTunnel {
|
||
return fmt.Errorf("LOCALAI_WORKER_TUNNEL is false, but this worker advertises no address and binds only loopback, and no frontend path dials a worker's address: without its tunnel nothing can reach it. Remove the setting, or run the pre-tunnel release on both the worker and the frontend")
|
||
}
|
||
return nil
|
||
}
|