4 Commits

Author	SHA1	Message	Date
LocalAI [bot]	d7d7721eae	feat(distributed): SyncedMap component + migrate finetune/quant/agent-tasks to cross-replica state (#10542) ... * feat(distributed): add SyncedMap cross-replica in-memory state component Introduce core/services/syncstate.SyncedMap[K,V]: a thread-safe in-memory map that keeps itself consistent across frontend replicas via NATS, with an optional pluggable durable Store and hydrate-from-source convergence. Several features keep process-local state surfaced to the API (finetune/quant jobs, agent tasks, model configs) and each hand-wired the same in-memory + NATS broadcast + read-through-store legs - or forgot to, reintroducing cross-replica staleness. SyncedMap makes that consistency a configuration choice: - local writes mutate the map, write through the Store, then broadcast a delta; - the apply path is memory-only and never re-publishes or re-writes the Store (structural echo-loop guard, mirroring galleryop.mergeStatus); - on Start and on NATS reconnect the map re-hydrates from the source (Store, else Loader); an optional periodic Reconcile repairs silent drift; - standalone mode (nil NATS client) is a strict in-memory no-op. Reconnect re-hydrate is wired via a new *messaging.Client.OnReconnect callback, consumed through an optional type-assertion so MessagingClient stays minimal. Adds messaging.SubjectSyncStateDelta and a reusable testutil.FakeBus (synchronous in-process MessagingClient with wildcard matching) for adopter tests. Component only; service migrations follow in subsequent commits. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(finetune): back jobs with SyncedMap for cross-replica consistency FineTuneService kept jobs in a process-local map and, although it wrote them to Postgres, ListJobs/GetJob never read the store back and the wired natsClient was never used - so in distributed mode a job created on one replica was invisible to the others. Replace the map and the dead client with a syncstate.SyncedMap keyed by job ID, value *schema.FineTuneJob (the exact REST shape, so responses are unchanged). - Add a Store adapter (core/services/finetune/syncstore.go) over FineTuneStore, plus FineTuneStore.ListAll (global hydrate; per-user List kept) and an idempotent Upsert (create-or-update; Create alone fails on dup key). - Writes go through SyncedMap.Set/Delete (write-through + broadcast); reads use List/Get. The on-disk state.json path becomes the standalone Loader, keeping single-node restart recovery (stale->stopped / exporting->failed fixups). - Fold SetNATSClient/SetFineTuneStore into NewFineTuneService; app.go passes the distributed NATS client + store when distributed, nil otherwise. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(agentpool): back agent tasks with SyncedMap for cross-replica consistency AgentJobService.ListTasks read the process-local tasks map only, while ListJobs already read through the DB persister + dispatcher NATS - so in distributed mode a task created on one replica was invisible to the others. Back tasks with a syncstate.SyncedMap keyed by task ID (value schema.Task, the exact REST shape); jobs are left untouched. - Store adapter (task_syncstore.go) over the existing JobPersister (LoadTasks/SaveTask/DeleteTask); reads svc.persister/userID live so a persister swap needs no rebuild. No new persister methods required. - Task reads -> SyncedMap.List/Get; create/update -> Set (write-through + broadcast); delete -> Delete. The file persister now owns its own task set so the write-through path does not re-enter the SyncedMap lock (deadlock guard). - The distributed NATS client is not available at construction (start() precedes initDistributed), so it is injected via SetTaskSyncNATS, which rebuilds the still-empty map before Start/hydrate. Wired at the main, restart, and per-user (UserServicesManager) distributed sites. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(quantization): back jobs with SyncedMap + durable QuantStore QuantizationService kept jobs in a process-local map persisted only to a local state.json, so in distributed mode jobs were neither visible across replicas nor durable cluster-wide. Back jobs with a syncstate.SyncedMap keyed by job ID (value *schema.QuantizationJob, the exact REST shape). - New distributed.QuantStore (GORM, table quantization_jobs) mirroring FineTuneStore: Create/Get/ListAll/Upsert(idempotent)/Delete, registered for AutoMigrate via distributed.InitStores (Stores.Quant). - New adapter (quantization/syncstore.go) over QuantStore implementing syncstate.Store, with record<->schema conversion. - Reads go through List/Get, writes through Set/Delete (write-through + broadcast); state.json is kept as the standalone Loader for single-node restart recovery (stale-job fixups preserved). - app.go passes the distributed NATS client + QuantStore when distributed, nil otherwise; Start/Close lifecycle mirrors finetune. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(syncstate): annotate gosec G118 false positive on lifeCtx gosec flagged the WithCancel in Start as "cancellation function not called" because the returned cancel is stored on the struct rather than called/deferred in scope. It is invoked in Close (covered by tests), and lifeCtx must outlive Start to drive the reconnect/reconcile goroutines. Suppress the verified false positive with a justified #nosec G118. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * test(distributed): e2e two-replica SyncedMap sync over real NATS + Postgres Adds the real-infrastructure counterpart to the fake-bus unit tests, in the existing distributed e2e suite (testcontainers NATS + PostgreSQL). Two SyncedMap instances stand in for two frontend replicas - each with its OWN NATS connection to a shared server and a SHARED Postgres store (the distributed-mode invariant) - and assert, over the wire: - a create on replica A is observed by replica B; - an update and a delete propagate A -> B (delete prunes, which a reload cannot); - a late-joining replica recovers a job it never received a delta for, via store hydrate on Start (the at-most-once gap a fake bus cannot exercise); - a local Set is written through to the shared Postgres store. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-27 23:23:51 +02:00
Richard Palethorpe	12d1f3a697	security(http): refuse redirects on outbound clients via hardened pkg/httpclient (#10087) ... LocalAI's outbound HTTP clients used Go's default redirect policy, which follows up to 10 redirects. On a cross-host redirect Go forwards custom request headers — including credential headers such as Anthropic's x-api-key — to the redirect target (Go strips Authorization, Cookie and WWW-Authenticate cross-host, but NOT arbitrary custom headers). An attacker able to elicit a redirect from an upstream (a hijacked or spoofed upstream, DNS trickery, or a malicious upstream_url) then harvests the operator's provider API key. This was first reported against the cloud-proxy / MITM PII path (GHSA-3mj3-57v2-4636); the same class affects every other outbound client. Rather than patch each call site, add pkg/httpclient as the one sanctioned constructor for outbound HTTP and route everything through it. pkg/httpclient: - New(...) refuses redirects, TLS 1.2 floor, no body deadline (streaming/SSE safe) - NewWithTimeout(d) simple request/response calls - WithFollowRedirects opt-in following that still strips credential headers on any cross-host hop; different scheme/host/port == different origin, guarding the curl CVE-2022-27774 port-confusion class - WithTransport(rt) keep a custom transport (IP-pin, HTTP/2, a credential-injecting RoundTripper) - HardenedTransport() base transport with the TLS floor + bounded setup - Harden(c) apply the policy to a library-supplied *http.Client - NoRedirect the CheckRedirect policy; wraps ErrRedirectBlocked Lint: a forbidigo rule flags http.DefaultClient and http.Get/Post/ PostForm/Head, pointing at pkg/httpclient (.golangci.yml, .agents/coding-style.md). forbidigo cannot match the &http.Client{} composite literal without also flagging legitimate *http.Client type references, so that form is enforced by review. Migrates every non-test outbound call site across core/, pkg/, cmd/, and the Go backend (backend/go/cloud-proxy). Credential-bearing and internal-RPC clients refuse redirects; download / CDN / registry clients use WithFollowRedirects so they keep working while stripping secrets cross-host. The only credential-bearing client that follows redirects is the gated-download path (pkg/downloader/uri.go), which strips the token on the cross-host hop to the CDN. Hardening this closes, in passing: - MCP remote-server bearer token leaking via a redirect (the RoundTripper re-injected Authorization on every hop) - agent multimedia/webhook clients leaking user-supplied auth headers - cors_proxy following redirects, bypassing its SSRF IP-pin - downloader's authorized read path leaking the token cross-host Fixes: GHSA-3mj3-57v2-4636 (cloud-proxy leaks operator provider API key (x-api-key) to attacker host on cross-host redirect) Reported-by: tonghuaroot Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-30 12:04:10 +02:00
LocalAI [bot]	ab01ed1a3e	fix(agentpool): close truncate-then-read race in agent_jobs.json persistence (#9811) ... * fix(agentpool): close truncate-then-read race in agent_jobs.json persistence Three call sites wrote and read agent_jobs.json (and agent_tasks.json) through three independent mutexes: - AgentJobService.ExecuteJob spawns go saveJobs(job) -> fileJobPersister holding p.mu - AgentJobService.SaveJobsToFile holding service.fileMutex - AgentJobService.LoadJobsFromFile on a separate service instance holding a different service.fileMutex Nothing serialized those mutexes, and both writers used os.WriteFile, which opens O_TRUNC. A reader landing between the truncate and the write saw a zero-byte file and surfaced as `unexpected end of JSON input` at offset 0. The macOS tests-apple job started hitting this consistently once the path filter was removed from .github/workflows/test.yml and the file-mode race test ran on every push (run 25823124797 was the first observed failure). Two changes close the window: 1. fileJobPersister.saveTasksToFile / saveJobsToFile now write to a same-directory temp file and os.Rename to the final path. rename(2) is atomic on POSIX, so concurrent readers see either the prior contents or the new contents and never a zero-byte window. The helper Syncs before close so a crash mid-write leaves either the old file intact or the temp behind (cleaned up on next save). 2. AgentJobService.{Load,Save}{Tasks,Jobs}{FromFile,ToFile} are collapsed to thin wrappers around fileJobPersister, removing the duplicate write path and the redundant service.fileMutex / service.tasksFile / service.jobsFile fields. Within a single service all task/job I/O now serializes on the persister's mutex; the atomic rename handles the cross-instance case the tests exercise. Adds a regression test that hammers SaveJobsToFile and LoadJobsFromFile concurrently for 500ms across two service instances on the same paths. On master this reproduces `unexpected end of JSON input` on Linux within ~500ms; with the fix the suite ran -until-it-fails for 30s (54 attempts, all green). Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(agentpool): route service flush/load through JobPersister interface The first cut of the race fix made AgentJobService.{Save,Load}{Tasks,Jobs}* type-assert s.persister to *fileJobPersister so they could reach the unexported saveTasksToFile / saveJobsToFile helpers. That defeats the JobPersister interface: the service is back to reasoning about a concrete implementation instead of an abstraction. Promote the bulk-flush operations to the interface as FlushTasks / FlushJobs: - fileJobPersister.FlushTasks/FlushJobs call the existing private helpers (atomic temp+rename writes from the prior commit). - dbJobPersister.FlushTasks/FlushJobs are no-ops because SaveTask/SaveJob are already write-through to the database. The service's four file-named methods now talk only to the interface: LoadTasks/LoadJobs read through s.persister.LoadTasks/LoadJobs, and the Save side calls FlushTasks/FlushJobs. The "FromFile"/"ToFile" suffixes stay for backward compat with user_services.go and the existing tests, but they no longer claim a file-only contract. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-13 23:58:43 +02:00
Ettore Di Giacinto	59108fbe32	feat: add distributed mode (#9124) ... * feat: add distributed mode (experimental) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix data races, mutexes, transactions Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactorings Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix events and tool stream in agent chat Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * use ginkgo Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(cron): compute correctly time boundaries avoiding re-triggering Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * enhancements, refactorings Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * do not flood of healthy checks Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * do not list obvious backends as text backends Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * tests fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Drop redundant healthcheck Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * enhancements, refactorings Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-30 00:47:27 +02:00