From d11b202dd2c7e26fc850aa13388bd58862d021d3 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sat, 27 Jun 2026 14:07:56 +0200 Subject: [PATCH 01/17] fix(backends): whisper darwin run.sh loads whichever fallback lib exists (.so/.dylib) (#10553) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fix(backends): whisper darwin run.sh loads whichever fallback lib exists The macOS branch hardcoded WHISPER_LIBRARY=$CURDIR/libgowhisper-fallback.dylib, but the cmake build emits a Mach-O named libgowhisper-fallback.so on darwin, so the Go loader panicked at runtime ("dlopen ...dylib: no such file") and the backend exited ("grpc service not ready") — breaking e.g. the silero-vad-ggml VAD on darwin. Pick whichever of .dylib/.so is present so it is robust to the build's naming either way. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- backend/go/whisper/run.sh | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/backend/go/whisper/run.sh b/backend/go/whisper/run.sh index 444a247c7..0cb05fe8b 100755 --- a/backend/go/whisper/run.sh +++ b/backend/go/whisper/run.sh @@ -13,8 +13,14 @@ if [ "$(uname)" != "Darwin" ]; then fi if [ "$(uname)" = "Darwin" ]; then - # macOS: single dylib variant (Metal or Accelerate) - LIBRARY="$CURDIR/libgowhisper-fallback.dylib" + # macOS: single fallback variant (Metal/Accelerate). The cmake build emits a + # Mach-O named .so, but tolerate .dylib too — pick whichever exists so the Go + # loader doesn't panic on a hardcoded name that isn't on disk. + if [ -e "$CURDIR/libgowhisper-fallback.dylib" ]; then + LIBRARY="$CURDIR/libgowhisper-fallback.dylib" + else + LIBRARY="$CURDIR/libgowhisper-fallback.so" + fi export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH else LIBRARY="$CURDIR/libgowhisper-fallback.so" From ec26b86dd481265335697412797436050847baac Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sat, 27 Jun 2026 22:36:02 +0200 Subject: [PATCH 02/17] docs: :arrow_up: update docs version mudler/LocalAI (#10560) :arrow_up: Update docs version mudler/LocalAI Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --- docs/data/version.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/data/version.json b/docs/data/version.json index 944cb9836..d7031c71b 100644 --- a/docs/data/version.json +++ b/docs/data/version.json @@ -1,3 +1,3 @@ { - "version": "v4.5.2" + "version": "v4.5.5" } From c548150f9915c6650e6d9e8d86d7ccb4d7a61de0 Mon Sep 17 00:00:00 2001 From: Nicholas Ciechanowski Date: Sun, 28 Jun 2026 07:10:12 +1000 Subject: [PATCH 03/17] fix(distributed): missing agent NATS permission (#10549) Signed-off-by: Nicholas Ciechanowski --- pkg/natsauth/permissions.go | 1 + 1 file changed, 1 insertion(+) diff --git a/pkg/natsauth/permissions.go b/pkg/natsauth/permissions.go index 8fdb11ef9..03a5a79c8 100644 --- a/pkg/natsauth/permissions.go +++ b/pkg/natsauth/permissions.go @@ -19,6 +19,7 @@ func WorkerPermissions(nodeID, nodeType string) (pubAllow, subAllow []string) { // Keep this list in sync with the subscriptions in core/cli/agent_worker.go. subAllow = []string{ "agent.execute", + "agent.*.cancel", "jobs.*.cancel", "jobs.*.progress", "jobs.*.result", From d7d7721eae1d5730b45d1f617e2cf6735e2cabfb Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sat, 27 Jun 2026 23:23:51 +0200 Subject: [PATCH 04/17] feat(distributed): SyncedMap component + migrate finetune/quant/agent-tasks to cross-replica state (#10542) * feat(distributed): add SyncedMap cross-replica in-memory state component Introduce core/services/syncstate.SyncedMap[K,V]: a thread-safe in-memory map that keeps itself consistent across frontend replicas via NATS, with an optional pluggable durable Store and hydrate-from-source convergence. Several features keep process-local state surfaced to the API (finetune/quant jobs, agent tasks, model configs) and each hand-wired the same in-memory + NATS broadcast + read-through-store legs - or forgot to, reintroducing cross-replica staleness. SyncedMap makes that consistency a configuration choice: - local writes mutate the map, write through the Store, then broadcast a delta; - the apply path is memory-only and never re-publishes or re-writes the Store (structural echo-loop guard, mirroring galleryop.mergeStatus); - on Start and on NATS reconnect the map re-hydrates from the source (Store, else Loader); an optional periodic Reconcile repairs silent drift; - standalone mode (nil NATS client) is a strict in-memory no-op. Reconnect re-hydrate is wired via a new *messaging.Client.OnReconnect callback, consumed through an optional type-assertion so MessagingClient stays minimal. Adds messaging.SubjectSyncStateDelta and a reusable testutil.FakeBus (synchronous in-process MessagingClient with wildcard matching) for adopter tests. Component only; service migrations follow in subsequent commits. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(finetune): back jobs with SyncedMap for cross-replica consistency FineTuneService kept jobs in a process-local map and, although it wrote them to Postgres, ListJobs/GetJob never read the store back and the wired natsClient was never used - so in distributed mode a job created on one replica was invisible to the others. Replace the map and the dead client with a syncstate.SyncedMap keyed by job ID, value *schema.FineTuneJob (the exact REST shape, so responses are unchanged). - Add a Store adapter (core/services/finetune/syncstore.go) over FineTuneStore, plus FineTuneStore.ListAll (global hydrate; per-user List kept) and an idempotent Upsert (create-or-update; Create alone fails on dup key). - Writes go through SyncedMap.Set/Delete (write-through + broadcast); reads use List/Get. The on-disk state.json path becomes the standalone Loader, keeping single-node restart recovery (stale->stopped / exporting->failed fixups). - Fold SetNATSClient/SetFineTuneStore into NewFineTuneService; app.go passes the distributed NATS client + store when distributed, nil otherwise. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(agentpool): back agent tasks with SyncedMap for cross-replica consistency AgentJobService.ListTasks read the process-local tasks map only, while ListJobs already read through the DB persister + dispatcher NATS - so in distributed mode a task created on one replica was invisible to the others. Back tasks with a syncstate.SyncedMap keyed by task ID (value schema.Task, the exact REST shape); jobs are left untouched. - Store adapter (task_syncstore.go) over the existing JobPersister (LoadTasks/SaveTask/DeleteTask); reads svc.persister/userID live so a persister swap needs no rebuild. No new persister methods required. - Task reads -> SyncedMap.List/Get; create/update -> Set (write-through + broadcast); delete -> Delete. The file persister now owns its own task set so the write-through path does not re-enter the SyncedMap lock (deadlock guard). - The distributed NATS client is not available at construction (start() precedes initDistributed), so it is injected via SetTaskSyncNATS, which rebuilds the still-empty map before Start/hydrate. Wired at the main, restart, and per-user (UserServicesManager) distributed sites. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(quantization): back jobs with SyncedMap + durable QuantStore QuantizationService kept jobs in a process-local map persisted only to a local state.json, so in distributed mode jobs were neither visible across replicas nor durable cluster-wide. Back jobs with a syncstate.SyncedMap keyed by job ID (value *schema.QuantizationJob, the exact REST shape). - New distributed.QuantStore (GORM, table quantization_jobs) mirroring FineTuneStore: Create/Get/ListAll/Upsert(idempotent)/Delete, registered for AutoMigrate via distributed.InitStores (Stores.Quant). - New adapter (quantization/syncstore.go) over QuantStore implementing syncstate.Store, with record<->schema conversion. - Reads go through List/Get, writes through Set/Delete (write-through + broadcast); state.json is kept as the standalone Loader for single-node restart recovery (stale-job fixups preserved). - app.go passes the distributed NATS client + QuantStore when distributed, nil otherwise; Start/Close lifecycle mirrors finetune. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(syncstate): annotate gosec G118 false positive on lifeCtx gosec flagged the WithCancel in Start as "cancellation function not called" because the returned cancel is stored on the struct rather than called/deferred in scope. It is invoked in Close (covered by tests), and lifeCtx must outlive Start to drive the reconnect/reconcile goroutines. Suppress the verified false positive with a justified #nosec G118. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * test(distributed): e2e two-replica SyncedMap sync over real NATS + Postgres Adds the real-infrastructure counterpart to the fake-bus unit tests, in the existing distributed e2e suite (testcontainers NATS + PostgreSQL). Two SyncedMap instances stand in for two frontend replicas - each with its OWN NATS connection to a shared server and a SHARED Postgres store (the distributed-mode invariant) - and assert, over the wire: - a create on replica A is observed by replica B; - an update and a delete propagate A -> B (delete prunes, which a reload cannot); - a late-joining replica recovers a job it never received a delta for, via store hydrate on Start (the at-most-once gap a fake bus cannot exercise); - a local Set is written through to the shared Postgres store. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- core/application/agent_jobs.go | 2 + core/application/application.go | 4 + core/application/startup.go | 3 + core/http/app.go | 34 +- core/services/agentpool/agent_jobs.go | 150 +++++---- core/services/agentpool/job_persister_file.go | 57 +++- core/services/agentpool/job_persister_test.go | 18 +- core/services/agentpool/task_sync_test.go | 152 +++++++++ core/services/agentpool/task_syncstore.go | 47 +++ core/services/agentpool/user_services.go | 14 + core/services/distributed/finetune.go | 29 ++ .../distributed/finetune_suite_test.go | 13 + core/services/distributed/finetune_test.go | 61 ++++ core/services/distributed/init.go | 9 +- core/services/distributed/quant.go | 105 +++++++ core/services/distributed/quant_test.go | 57 ++++ core/services/finetune/finetune_suite_test.go | 13 + core/services/finetune/service.go | 168 +++++----- core/services/finetune/service_test.go | 185 +++++++++++ core/services/finetune/syncstore.go | 114 +++++++ core/services/messaging/client.go | 41 ++- core/services/messaging/subjects.go | 14 + .../quantization/quantization_suite_test.go | 13 + core/services/quantization/service.go | 120 ++++++-- core/services/quantization/service_test.go | 187 +++++++++++ core/services/quantization/syncstore.go | 114 +++++++ core/services/syncstate/syncstate.go | 289 +++++++++++++++++ .../syncstate/syncstate_suite_test.go | 13 + core/services/syncstate/syncstate_test.go | 291 ++++++++++++++++++ core/services/testutil/fakebus.go | 160 ++++++++++ .../distributed/syncstate_distributed_test.go | 161 ++++++++++ 31 files changed, 2450 insertions(+), 188 deletions(-) create mode 100644 core/services/agentpool/task_sync_test.go create mode 100644 core/services/agentpool/task_syncstore.go create mode 100644 core/services/distributed/finetune_suite_test.go create mode 100644 core/services/distributed/finetune_test.go create mode 100644 core/services/distributed/quant.go create mode 100644 core/services/distributed/quant_test.go create mode 100644 core/services/finetune/finetune_suite_test.go create mode 100644 core/services/finetune/service_test.go create mode 100644 core/services/finetune/syncstore.go create mode 100644 core/services/quantization/quantization_suite_test.go create mode 100644 core/services/quantization/service_test.go create mode 100644 core/services/quantization/syncstore.go create mode 100644 core/services/syncstate/syncstate.go create mode 100644 core/services/syncstate/syncstate_suite_test.go create mode 100644 core/services/syncstate/syncstate_test.go create mode 100644 core/services/testutil/fakebus.go create mode 100644 tests/e2e/distributed/syncstate_distributed_test.go diff --git a/core/application/agent_jobs.go b/core/application/agent_jobs.go index b7cfb20a3..f380b0750 100644 --- a/core/application/agent_jobs.go +++ b/core/application/agent_jobs.go @@ -37,6 +37,8 @@ func (a *Application) RestartAgentJobService() error { if d.JobStore != nil { agentJobService.SetDistributedJobStore(d.JobStore) } + // Keep agent tasks consistent across replicas (same client the dispatcher uses). + agentJobService.SetTaskSyncNATS(d.Nats) } // Start the service diff --git a/core/application/application.go b/core/application/application.go index 9bbf26bb8..52f8618f1 100644 --- a/core/application/application.go +++ b/core/application/application.go @@ -604,6 +604,10 @@ func (a *Application) StartAgentPool() { usm.SetJobDBStore(s) } } + // Keep per-user agent tasks consistent across replicas (nil in standalone). + if d := a.Distributed(); d != nil { + usm.SetJobSyncNATS(d.Nats) + } aps.SetUserServicesManager(usm) a.agentPoolService.Store(aps) diff --git a/core/application/startup.go b/core/application/startup.go index a71f8d0ea..25d965834 100644 --- a/core/application/startup.go +++ b/core/application/startup.go @@ -280,6 +280,9 @@ func New(opts ...config.AppOption) (*Application, error) { if application.agentJobService != nil { application.agentJobService.SetDistributedBackends(distSvc.Dispatcher) application.agentJobService.SetDistributedJobStore(distSvc.JobStore) + // Keep agent tasks consistent across replicas (jobs already sync via the + // dispatcher + DB read-through). Same NATS client the dispatcher uses. + application.agentJobService.SetTaskSyncNATS(distSvc.Nats) } // Wire skill store into AgentPoolService (wired at pool start time via closure) // The actual wiring happens in StartAgentPool since the pool doesn't exist yet. diff --git a/core/http/app.go b/core/http/app.go index ee5cd99eb..fff8a3468 100644 --- a/core/http/app.go +++ b/core/http/app.go @@ -23,8 +23,10 @@ import ( "github.com/mudler/LocalAI/core/application" "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/core/services/distributed" "github.com/mudler/LocalAI/core/services/finetune" "github.com/mudler/LocalAI/core/services/galleryop" + "github.com/mudler/LocalAI/core/services/messaging" "github.com/mudler/LocalAI/core/services/nodes" "github.com/mudler/LocalAI/core/services/quantization" @@ -400,25 +402,45 @@ func API(application *application.Application) (*echo.Echo, error) { routes.RegisterAgentPoolRoutes(e, application, agentsMw, skillsMw, collectionsMw) // Fine-tuning routes fineTuningMw := auth.RequireFeature(application.AuthDB(), auth.FeatureFineTuning) + // In distributed mode pass the shared NATS client + PostgreSQL store so + // fine-tune jobs stay consistent across replicas (the SyncedMap broadcasts + // mutations and hydrates from the DB); standalone passes nil for both. + var ftNats messaging.MessagingClient + var ftStore *distributed.FineTuneStore + if d := application.Distributed(); d != nil { + ftNats = d.Nats + if d.DistStores != nil && d.DistStores.FineTune != nil { + ftStore = d.DistStores.FineTune + } + } ftService := finetune.NewFineTuneService( application.ApplicationConfig(), application.ModelLoader(), application.ModelConfigLoader(), + ftNats, + ftStore, ) - if d := application.Distributed(); d != nil { - ftService.SetNATSClient(d.Nats) - if d.DistStores != nil && d.DistStores.FineTune != nil { - ftService.SetFineTuneStore(d.DistStores.FineTune) - } - } routes.RegisterFineTuningRoutes(e, ftService, application.ApplicationConfig(), fineTuningMw) // Quantization routes quantizationMw := auth.RequireFeature(application.AuthDB(), auth.FeatureQuantization) + // In distributed mode pass the shared NATS client + PostgreSQL store so + // quantization jobs stay consistent across replicas (the SyncedMap broadcasts + // mutations and hydrates from the DB); standalone passes nil for both. + var quantNats messaging.MessagingClient + var quantStore *distributed.QuantStore + if d := application.Distributed(); d != nil { + quantNats = d.Nats + if d.DistStores != nil && d.DistStores.Quant != nil { + quantStore = d.DistStores.Quant + } + } qService := quantization.NewQuantizationService( application.ApplicationConfig(), application.ModelLoader(), application.ModelConfigLoader(), + quantNats, + quantStore, ) routes.RegisterQuantizationRoutes(e, qService, application.ApplicationConfig(), quantizationMw) diff --git a/core/services/agentpool/agent_jobs.go b/core/services/agentpool/agent_jobs.go index 8d9e82b8e..59850981a 100644 --- a/core/services/agentpool/agent_jobs.go +++ b/core/services/agentpool/agent_jobs.go @@ -30,6 +30,8 @@ import ( mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp" "github.com/mudler/LocalAI/core/schema" "github.com/mudler/LocalAI/core/services/jobs" + "github.com/mudler/LocalAI/core/services/messaging" + "github.com/mudler/LocalAI/core/services/syncstate" "github.com/mudler/LocalAI/core/templates" "github.com/mudler/LocalAI/pkg/httpclient" "github.com/mudler/LocalAI/pkg/model" @@ -43,8 +45,18 @@ type AgentJobService struct { configLoader *config.ModelConfigLoader evaluator *templates.Evaluator + // tasks is the cross-replica task store: an in-memory map kept consistent + // across replicas via NATS, with read-through to the configured persister + // (file in standalone, PostgreSQL in distributed). Unlike jobs - which already + // converge via the dispatcher + DB read-through - tasks previously read + // in-memory only, so ListTasks went stale on non-originating replicas. + tasks *syncstate.SyncedMap[string, schema.Task] + // taskNats is the distributed NATS client backing the tasks SyncedMap. It is + // not available at construction time, so it is injected via SetTaskSyncNATS + // during distributed wiring; nil keeps tasks in-memory-only (standalone). + taskNats messaging.MessagingClient + // Storage (in-memory primary, persister for secondary persistence) - tasks *xsync.SyncedMap[string, schema.Task] jobs *xsync.SyncedMap[string, schema.Job] persister JobPersister userID string // Scoping: empty for global (main service), set for per-user instances @@ -96,6 +108,31 @@ func (s *AgentJobService) SetDistributedJobStore(store *jobs.JobStore) { s.persister = &dbJobPersister{store: store} } +// SetTaskSyncNATS wires the distributed NATS client used to keep agent *tasks* +// consistent across replicas (jobs already converge via the dispatcher + DB +// read-through, so they are left untouched). The client is not available when the +// service is constructed, so it is injected here during distributed wiring and the +// tasks SyncedMap is rebuilt to pick it up. It is always called before Start / +// hydrate, while the map is still empty, so rebuilding loses no state. Passing nil +// (standalone) keeps the map in-memory-only with no broadcast. +func (s *AgentJobService) SetTaskSyncNATS(nats messaging.MessagingClient) { + s.taskNats = nats + s.buildTasksMap() +} + +// buildTasksMap (re)constructs the cross-replica tasks SyncedMap from the current +// taskNats. The Store adapter reads s.persister/s.userID live, so a persister swap +// (SetDistributedJobStore) needs no rebuild; only the NATS client, fixed at +// New-time, forces one - hence SetTaskSyncNATS calls this. +func (s *AgentJobService) buildTasksMap() { + s.tasks = syncstate.New(syncstate.Config[string, schema.Task]{ + Name: "agent.tasks", + Key: func(t schema.Task) string { return t.ID }, + Nats: s.taskNats, + Store: &taskStoreAdapter{svc: s}, + }) +} + // Dispatcher returns the distributed dispatcher (nil if not in distributed mode). func (s *AgentJobService) Dispatcher() DistributedDispatcher { return s.dispatcher @@ -106,13 +143,6 @@ func (s *AgentJobService) DBStore() *jobs.JobStore { return s.rawDBStore } -// saveTasks persists tasks via the configured persister (file or DB). -func (s *AgentJobService) saveTasks(task schema.Task) { - if err := s.persister.SaveTask(s.userID, task); err != nil { - xlog.Warn("Failed to persist task", "error", err, "task_id", task.ID) - } -} - // saveJobs persists jobs via the configured persister (file or DB). func (s *AgentJobService) saveJobs(job schema.Job) { if err := s.persister.SaveJob(s.userID, job); err != nil { @@ -129,18 +159,8 @@ func (s *AgentJobService) LoadFromDB() { // loadFromPersister loads tasks and jobs from the configured persister into memory. func (s *AgentJobService) loadFromPersister() { - if tasks, err := s.persister.LoadTasks(s.userID); err != nil { + if err := s.hydrateTasks(s.appConfig.Context); err != nil { xlog.Warn("Failed to load tasks from persister", "error", err) - } else { - for _, task := range tasks { - s.tasks.Set(task.ID, task) - if task.Enabled && task.Cron != "" { - if err := s.ScheduleCronTask(task); err != nil { - xlog.Warn("Failed to schedule cron task on load", "error", err, "task_id", task.ID) - } - } - } - xlog.Info("Loaded tasks from persister", "count", len(tasks)) } if loadedJobs, err := s.persister.LoadJobs(s.userID); err != nil { @@ -153,6 +173,27 @@ func (s *AgentJobService) loadFromPersister() { } } +// hydrateTasks loads tasks into the cross-replica SyncedMap and (re)schedules +// cron entries for enabled tasks. Hydration goes through the SyncedMap's Store +// read-through (Start), not Set, so it neither re-persists nor re-broadcasts the +// loaded tasks. Each service instance hydrates exactly once: the main service via +// Start -> loadFromPersister, per-user services via LoadFromDB or LoadTasksFromFile. +func (s *AgentJobService) hydrateTasks(ctx context.Context) error { + if err := s.tasks.Start(ctx); err != nil { + return err + } + tasks := s.tasks.List() + for _, task := range tasks { + if task.Enabled && task.Cron != "" { + if err := s.ScheduleCronTask(task); err != nil { + xlog.Warn("Failed to schedule cron task on load", "error", err, "task_id", task.ID) + } + } + } + xlog.Info("Loaded tasks from persister", "count", len(tasks)) + return nil +} + // JobExecution represents a job to be executed type JobExecution struct { Job schema.Job @@ -200,21 +241,19 @@ func NewAgentJobServiceWithPaths( ) *AgentJobService { retentionDays := cmp.Or(appConfig.AgentJobRetentionDays, 30) - tasks := xsync.NewSyncedMap[string, schema.Task]() jobsMap := xsync.NewSyncedMap[string, schema.Job]() - return &AgentJobService{ + s := &AgentJobService{ appConfig: appConfig, modelLoader: modelLoader, configLoader: configLoader, evaluator: evaluator, - tasks: tasks, jobs: jobsMap, persister: &fileJobPersister{ - tasks: tasks, jobs: jobsMap, tasksFile: tasksFile, jobsFile: jobsFile, + taskSet: make(map[string]schema.Task), }, jobQueue: make(chan JobExecution, 100), // Buffer for 100 jobs cancellations: xsync.NewSyncedMap[string, context.CancelFunc](), @@ -222,25 +261,17 @@ func NewAgentJobServiceWithPaths( cronEntries: xsync.NewSyncedMap[string, cron.EntryID](), retentionDays: retentionDays, } + // Build the cross-replica tasks map standalone (nil NATS); SetTaskSyncNATS + // rebuilds it with the distributed client once that is available, before Start. + s.buildTasksMap() + return s } // LoadTasksFromFile loads tasks from the persister into the in-memory map // and schedules cron entries. Named "FromFile" for backward compat; in DB // mode it loads from the database. func (s *AgentJobService) LoadTasksFromFile() error { - tasks, err := s.persister.LoadTasks(s.userID) - if err != nil { - return err - } - for _, task := range tasks { - s.tasks.Set(task.ID, task) - if task.Enabled && task.Cron != "" { - if err := s.ScheduleCronTask(task); err != nil { - xlog.Warn("Failed to schedule cron task on load", "error", err, "task_id", task.ID) - } - } - } - return nil + return s.hydrateTasks(s.appConfig.Context) } // SaveTasksToFile flushes the current tasks map via the persister. File @@ -293,8 +324,12 @@ func (s *AgentJobService) CreateTask(task schema.Task) (string, error) { task.Enabled = true // Default to enabled } - // Store task - s.tasks.Set(id, task) + // Store task: Set updates the in-memory map, write-throughs to the persister + // (file or DB), and broadcasts the create to peer replicas. Background ctx + // because CreateTask carries no request ctx (mirrors the finetune service). + if err := s.tasks.Set(context.Background(), task); err != nil { + return "", fmt.Errorf("failed to persist task: %w", err) + } // Schedule cron if enabled and has cron expression if task.Enabled && task.Cron != "" { @@ -303,16 +338,15 @@ func (s *AgentJobService) CreateTask(task schema.Task) (string, error) { } } - s.saveTasks(task) return id, nil } // UpdateTask updates an existing task func (s *AgentJobService) UpdateTask(id string, task schema.Task) error { - if !s.tasks.Exists(id) { + existing, ok := s.tasks.Get(id) + if !ok { return fmt.Errorf("%w: %s", ErrTaskNotFound, id) } - existing := s.tasks.Get(id) // Preserve ID and CreatedAt task.ID = id @@ -324,8 +358,10 @@ func (s *AgentJobService) UpdateTask(id string, task schema.Task) error { s.UnscheduleCronTask(id) } - // Store updated task - s.tasks.Set(id, task) + // Store updated task: write-through + broadcast (see CreateTask). + if err := s.tasks.Set(context.Background(), task); err != nil { + return fmt.Errorf("failed to persist task: %w", err) + } // Schedule new cron if enabled and has cron expression if task.Enabled && task.Cron != "" { @@ -334,24 +370,22 @@ func (s *AgentJobService) UpdateTask(id string, task schema.Task) error { } } - s.saveTasks(task) return nil } // DeleteTask deletes a task func (s *AgentJobService) DeleteTask(id string) error { - if !s.tasks.Exists(id) { + if _, ok := s.tasks.Get(id); !ok { return fmt.Errorf("%w: %s", ErrTaskNotFound, id) } // Unschedule cron s.UnscheduleCronTask(id) - // Remove from memory - s.tasks.Delete(id) - - if err := s.persister.DeleteTask(id); err != nil { - xlog.Warn("Failed to delete task from persister", "error", err, "task_id", id) + // Delete removes from the in-memory map, deletes from the persister, and + // broadcasts the removal to peer replicas. + if err := s.tasks.Delete(context.Background(), id); err != nil { + xlog.Warn("Failed to delete task from store", "error", err, "task_id", id) } return nil @@ -359,8 +393,8 @@ func (s *AgentJobService) DeleteTask(id string) error { // GetTask retrieves a task by ID func (s *AgentJobService) GetTask(id string) (*schema.Task, error) { - task := s.tasks.Get(id) - if task.ID == "" { + task, ok := s.tasks.Get(id) + if !ok { return nil, fmt.Errorf("%w: %s", ErrTaskNotFound, id) } return &task, nil @@ -368,7 +402,7 @@ func (s *AgentJobService) GetTask(id string) (*schema.Task, error) { // ListTasks returns all tasks, sorted by creation date (newest first) func (s *AgentJobService) ListTasks() []schema.Task { - tasks := s.tasks.Values() + tasks := s.tasks.List() // Sort by CreatedAt descending (newest first), then by Name for stability slices.SortFunc(tasks, func(a, b schema.Task) int { if a.CreatedAt.Equal(b.CreatedAt) { @@ -397,8 +431,8 @@ func (s *AgentJobService) buildPrompt(templateStr string, params map[string]stri // ExecuteJob creates and queues a job for execution // multimedia can be nil for backward compatibility func (s *AgentJobService) ExecuteJob(taskID string, params map[string]string, triggeredBy string, multimedia *schema.MultimediaAttachment) (string, error) { - task := s.tasks.Get(taskID) - if task.ID == "" { + task, ok := s.tasks.Get(taskID) + if !ok { return "", fmt.Errorf("%w: %s", ErrTaskNotFound, taskID) } @@ -1451,6 +1485,12 @@ func (s *AgentJobService) Stop() error { if s.cronScheduler != nil { s.cronScheduler.Stop() } + // Release the tasks SyncedMap subscription / background workers. + if s.tasks != nil { + if err := s.tasks.Close(); err != nil { + xlog.Warn("Error closing tasks sync map", "error", err) + } + } xlog.Info("AgentJobService stopped") return nil } diff --git a/core/services/agentpool/job_persister_file.go b/core/services/agentpool/job_persister_file.go index 3087a2524..b161c442b 100644 --- a/core/services/agentpool/job_persister_file.go +++ b/core/services/agentpool/job_persister_file.go @@ -14,24 +14,38 @@ import ( ) // fileJobPersister persists tasks and jobs to JSON files. -// It holds references to the service's syncmaps and serializes the entire -// map contents on each save (bulk write). Reads at runtime return nil -// (the in-memory map is the authoritative source); LoadTasks/LoadJobs -// are used only at startup to bootstrap the syncmaps. +// +// Jobs serialize the service's in-memory jobs syncmap on each save (bulk write). +// Tasks are kept in this persister's own taskSet map instead: the tasks SyncedMap +// calls SaveTask/DeleteTask while holding its internal lock (write-through), so +// reading back the SyncedMap here would re-enter that lock and deadlock. The +// self-contained taskSet, seeded by LoadTasks, lets a per-task write rewrite the +// whole bulk file without touching the SyncedMap. +// +// Runtime reads (GetJob/ListJobs) return nil (the in-memory state is the +// authoritative source); LoadTasks/LoadJobs bootstrap state at startup. type fileJobPersister struct { - tasks *xsync.SyncedMap[string, schema.Task] jobs *xsync.SyncedMap[string, schema.Job] tasksFile string jobsFile string mu sync.Mutex + // taskSet is the persister's own view of all tasks, seeded by LoadTasks and + // updated by SaveTask/DeleteTask. The bulk JSON file is rewritten from it. + taskSet map[string]schema.Task } -func (p *fileJobPersister) SaveTask(_ string, _ schema.Task) error { - return p.saveTasksToFile() +func (p *fileJobPersister) SaveTask(_ string, task schema.Task) error { + p.mu.Lock() + defer p.mu.Unlock() + p.taskSet[task.ID] = task + return p.writeTasksLocked() } -func (p *fileJobPersister) DeleteTask(_ string) error { - return p.saveTasksToFile() +func (p *fileJobPersister) DeleteTask(taskID string) error { + p.mu.Lock() + defer p.mu.Unlock() + delete(p.taskSet, taskID) + return p.writeTasksLocked() } func (p *fileJobPersister) SaveJob(_ string, _ schema.Job) error { @@ -43,7 +57,9 @@ func (p *fileJobPersister) DeleteJob(_ string) error { } func (p *fileJobPersister) FlushTasks() error { - return p.saveTasksToFile() + p.mu.Lock() + defer p.mu.Unlock() + return p.writeTasksLocked() } func (p *fileJobPersister) FlushJobs() error { @@ -83,6 +99,12 @@ func (p *fileJobPersister) LoadTasks(_ string) ([]schema.Task, error) { return nil, fmt.Errorf("failed to parse tasks file: %w", err) } + // Seed the in-memory set so subsequent per-task SaveTask/DeleteTask merge into + // (rather than overwrite) the persisted tasks when the bulk file is rewritten. + for _, t := range tf.Tasks { + p.taskSet[t.ID] = t + } + xlog.Info("Loaded tasks from file", "count", len(tf.Tasks)) return tf.Tasks, nil } @@ -118,19 +140,20 @@ func (p *fileJobPersister) CleanupOldJobs(_ time.Duration) (int64, error) { return 0, nil // cleanup handled via in-memory filtering } -// saveTasksToFile serializes the entire tasks map to the JSON file. -func (p *fileJobPersister) saveTasksToFile() error { +// writeTasksLocked serializes the persister's task set to the JSON file. Callers +// must hold p.mu. +func (p *fileJobPersister) writeTasksLocked() error { if p.tasksFile == "" { return nil } - p.mu.Lock() - defer p.mu.Unlock() - - tf := schema.TasksFile{ - Tasks: p.tasks.Values(), + tasks := make([]schema.Task, 0, len(p.taskSet)) + for _, t := range p.taskSet { + tasks = append(tasks, t) } + tf := schema.TasksFile{Tasks: tasks} + data, err := json.MarshalIndent(tf, "", " ") if err != nil { return fmt.Errorf("failed to marshal tasks: %w", err) diff --git a/core/services/agentpool/job_persister_test.go b/core/services/agentpool/job_persister_test.go index 919eb4a66..646104db6 100644 --- a/core/services/agentpool/job_persister_test.go +++ b/core/services/agentpool/job_persister_test.go @@ -20,28 +20,26 @@ var _ = Describe("JobPersister", func() { Context("fileJobPersister", func() { var ( p *fileJobPersister - tasks *xsync.SyncedMap[string, schema.Task] jobsMap *xsync.SyncedMap[string, schema.Job] tmpDir string ) BeforeEach(func() { tmpDir = GinkgoT().TempDir() - tasks = xsync.NewSyncedMap[string, schema.Task]() jobsMap = xsync.NewSyncedMap[string, schema.Job]() p = &fileJobPersister{ - tasks: tasks, jobs: jobsMap, tasksFile: filepath.Join(tmpDir, "tasks.json"), jobsFile: filepath.Join(tmpDir, "jobs.json"), + // taskSet is the persister's own task view (decoupled from the tasks + // SyncedMap to avoid re-entering its lock during write-through). + taskSet: make(map[string]schema.Task), } }) It("SaveTask writes all tasks to file", func() { - tasks.Set("t1", schema.Task{ID: "t1", Name: "Task One", Model: "m", Prompt: "p"}) - tasks.Set("t2", schema.Task{ID: "t2", Name: "Task Two", Model: "m", Prompt: "p"}) - - Expect(p.SaveTask("", schema.Task{})).To(Succeed()) + Expect(p.SaveTask("", schema.Task{ID: "t1", Name: "Task One", Model: "m", Prompt: "p"})).To(Succeed()) + Expect(p.SaveTask("", schema.Task{ID: "t2", Name: "Task Two", Model: "m", Prompt: "p"})).To(Succeed()) // Verify file contents data, err := os.ReadFile(p.tasksFile) @@ -52,11 +50,9 @@ var _ = Describe("JobPersister", func() { }) It("DeleteTask writes updated tasks to file", func() { - tasks.Set("t1", schema.Task{ID: "t1", Name: "Keep"}) - tasks.Set("t2", schema.Task{ID: "t2", Name: "Delete"}) + Expect(p.SaveTask("", schema.Task{ID: "t1", Name: "Keep"})).To(Succeed()) + Expect(p.SaveTask("", schema.Task{ID: "t2", Name: "Delete"})).To(Succeed()) - // Simulate deletion from memory (caller does this before calling persister) - tasks.Delete("t2") Expect(p.DeleteTask("t2")).To(Succeed()) data, err := os.ReadFile(p.tasksFile) diff --git a/core/services/agentpool/task_sync_test.go b/core/services/agentpool/task_sync_test.go new file mode 100644 index 000000000..d42a197f6 --- /dev/null +++ b/core/services/agentpool/task_sync_test.go @@ -0,0 +1,152 @@ +package agentpool + +// White-box tests (package agentpool) so a spec can build two AgentJobService +// instances sharing one in-memory bus and assert that agent *tasks* converge +// across replicas - the bug this migration fixes (ListTasks used to read +// in-memory only, so a task created on replica A was invisible on replica B). +// Jobs are deliberately untouched here: they already converge via the dispatcher +// + DB read-through. + +import ( + "context" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/core/services/messaging" + "github.com/mudler/LocalAI/core/services/syncstate" + "github.com/mudler/LocalAI/core/services/testutil" + "github.com/mudler/LocalAI/pkg/system" +) + +// newTaskSyncService builds an AgentJobService wired to the given bus and a +// throwaway data dir (so the file persister has somewhere to write). Model/config +// loaders are nil because the task sync paths under test never touch them. +func newTaskSyncService(bus messaging.MessagingClient) *AgentJobService { + tmpDir := GinkgoT().TempDir() + sysState := &system.SystemState{} + sysState.Model.ModelsPath = tmpDir + appConfig := config.NewApplicationConfig( + config.WithDynamicConfigDir(tmpDir), + config.WithContext(context.Background()), + ) + appConfig.SystemState = sysState + + svc := NewAgentJobServiceWithPaths(appConfig, nil, nil, nil, + // Distinct per-replica files so the file persister write-through never + // crosses replicas: convergence here must be proven via the bus alone. + tmpDir+"/tasks.json", tmpDir+"/jobs.json") + svc.SetTaskSyncNATS(bus) + return svc +} + +var _ = Describe("AgentJobService task cross-replica sync", func() { + Describe("two replicas sharing one bus", func() { + var ( + bus *testutil.FakeBus + a, b *AgentJobService + ) + + BeforeEach(func() { + // One shared bus, two replicas: exactly the distributed topology where a + // round-robin request may land on a replica that did not originate the + // change. + bus = testutil.NewFakeBus() + a = newTaskSyncService(bus) + b = newTaskSyncService(bus) + // Start hydrates (empty here) and subscribes both replicas to deltas. + Expect(a.Start(context.Background())).To(Succeed()) + Expect(b.Start(context.Background())).To(Succeed()) + }) + + AfterEach(func() { + Expect(a.Stop()).To(Succeed()) + Expect(b.Stop()).To(Succeed()) + }) + + It("makes a task created on A visible via B's GetTask and ListTasks", func() { + id, err := a.CreateTask(schema.Task{Name: "Shared", Model: "m", Prompt: "p"}) + Expect(err).NotTo(HaveOccurred()) + + got, err := b.GetTask(id) + Expect(err).NotTo(HaveOccurred(), "B must see a task A just created") + Expect(got.Name).To(Equal("Shared")) + + listed := b.ListTasks() + Expect(listed).To(HaveLen(1)) + Expect(listed[0].ID).To(Equal(id)) + }) + + It("propagates a task update from A to B", func() { + id, err := a.CreateTask(schema.Task{Name: "Before", Model: "m", Prompt: "p"}) + Expect(err).NotTo(HaveOccurred()) + + Expect(a.UpdateTask(id, schema.Task{Name: "After", Model: "m", Prompt: "p"})).To(Succeed()) + + got, err := b.GetTask(id) + Expect(err).NotTo(HaveOccurred()) + Expect(got.Name).To(Equal("After"), "an update on A must be visible on B") + }) + + It("removes a task from B when it is deleted on A", func() { + id, err := a.CreateTask(schema.Task{Name: "Doomed", Model: "m", Prompt: "p"}) + Expect(err).NotTo(HaveOccurred()) + _, err = b.GetTask(id) + Expect(err).NotTo(HaveOccurred(), "precondition: B must have the task before the delete") + + Expect(a.DeleteTask(id)).To(Succeed()) + + _, err = b.GetTask(id) + Expect(err).To(HaveOccurred(), "a delete on A must remove the task from B") + Expect(b.ListTasks()).To(BeEmpty()) + }) + + It("does not re-broadcast a delta it received (echo-loop guard)", func() { + subject := messaging.SubjectSyncStateDelta("agent.tasks") + + _, err := a.CreateTask(schema.Task{Name: "Once", Model: "m", Prompt: "p"}) + Expect(err).NotTo(HaveOccurred()) + + // Exactly one publish: A's create. B applies it without re-publishing, + // otherwise this would be 2+ and a real bus would storm. + Expect(bus.PublishCount(subject)).To(Equal(1)) + }) + }) + + Describe("ListTasks ordering and scoping", func() { + var svc *AgentJobService + + BeforeEach(func() { + svc = newTaskSyncService(testutil.NewFakeBus()) + Expect(svc.Start(context.Background())).To(Succeed()) + }) + AfterEach(func() { Expect(svc.Stop()).To(Succeed()) }) + + It("sorts newest-first, breaking ties by name", func() { + // CreateTask stamps CreatedAt with time.Now(); space them out so ordering + // is deterministic rather than relying on the sub-millisecond gap. + oldID, err := svc.CreateTask(schema.Task{Name: "Old", Model: "m", Prompt: "p"}) + Expect(err).NotTo(HaveOccurred()) + time.Sleep(5 * time.Millisecond) + newID, err := svc.CreateTask(schema.Task{Name: "New", Model: "m", Prompt: "p"}) + Expect(err).NotTo(HaveOccurred()) + + listed := svc.ListTasks() + Expect(listed).To(HaveLen(2)) + Expect(listed[0].ID).To(Equal(newID), "newest first") + Expect(listed[1].ID).To(Equal(oldID)) + }) + }) + + Describe("compile-time adapter contract", func() { + It("satisfies syncstate.Store for tasks", func() { + // Mirrors the var assertion in task_syncstore.go; keeps the type + // referenced from a spec so drift surfaces here too. + var _ syncstate.Store[string, schema.Task] = (*taskStoreAdapter)(nil) + Expect(&taskStoreAdapter{}).ToNot(BeNil()) + }) + }) +}) diff --git a/core/services/agentpool/task_syncstore.go b/core/services/agentpool/task_syncstore.go new file mode 100644 index 000000000..ef8f3f8cc --- /dev/null +++ b/core/services/agentpool/task_syncstore.go @@ -0,0 +1,47 @@ +package agentpool + +import ( + "context" + + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/core/services/syncstate" +) + +// taskStoreAdapter bridges the existing JobPersister (file- or DB-backed) to the +// generic syncstate.Store the tasks SyncedMap consumes. Only tasks are migrated: +// jobs already converge across replicas via the dispatcher (NATS) plus the DB +// read-through in ListJobs/GetJob, whereas ListTasks read in-memory only and so +// went stale on replicas that did not originate the change. +// +// The adapter reads svc.persister and svc.userID live (rather than capturing +// them) because both are configured by setters - SetDistributedJobStore swaps the +// file persister for the DB one, SetUserID scopes per-user queries - AFTER the +// service, and thus this adapter, is constructed. Reading them at call time means +// the SyncedMap never has to be rebuilt when the persister is swapped. +// +// The SyncedMap value type is schema.Task: the exact shape ListTasks returns, so +// reads need no conversion and REST responses are provably unchanged. +type taskStoreAdapter struct { + svc *AgentJobService +} + +// compile-time assertion that the adapter satisfies the component's Store. +var _ syncstate.Store[string, schema.Task] = (*taskStoreAdapter)(nil) + +// List hydrates the map from durable storage on Start/reconnect: the file's task +// list (standalone) or every task row (DB / distributed). +func (a *taskStoreAdapter) List(_ context.Context) ([]schema.Task, error) { + return a.svc.persister.LoadTasks(a.svc.userID) +} + +// Upsert write-through persists a single task created/updated locally; the +// SyncedMap then broadcasts the delta to peers. +func (a *taskStoreAdapter) Upsert(_ context.Context, task schema.Task) error { + return a.svc.persister.SaveTask(a.svc.userID, task) +} + +// Delete write-through removes a task locally; the SyncedMap then broadcasts the +// removal to peers. +func (a *taskStoreAdapter) Delete(_ context.Context, id string) error { + return a.svc.persister.DeleteTask(id) +} diff --git a/core/services/agentpool/user_services.go b/core/services/agentpool/user_services.go index db30e25ad..56d19e0fc 100644 --- a/core/services/agentpool/user_services.go +++ b/core/services/agentpool/user_services.go @@ -7,6 +7,7 @@ import ( "github.com/mudler/LocalAGI/webui/collections" "github.com/mudler/LocalAI/core/config" "github.com/mudler/LocalAI/core/services/jobs" + "github.com/mudler/LocalAI/core/services/messaging" "github.com/mudler/LocalAI/core/templates" "github.com/mudler/LocalAI/pkg/model" "github.com/mudler/xlog" @@ -28,6 +29,9 @@ type UserServicesManager struct { // Shared distributed backends (set once, inherited by per-user job services) jobDispatcher DistributedDispatcher jobDBStore *jobs.JobStore + // jobNats keeps per-user agent tasks consistent across replicas (nil in + // standalone). Inherited by each per-user AgentJobService. + jobNats messaging.MessagingClient } // NewUserServicesManager creates a new UserServicesManager. @@ -162,6 +166,10 @@ func (m *UserServicesManager) GetJobs(userID string) (*AgentJobService, error) { if m.jobDispatcher != nil { svc.SetDistributedBackends(m.jobDispatcher) } + // Inherit the NATS client so per-user tasks broadcast across replicas. Must be + // set before the hydrate below (LoadFromDB / LoadTasksFromFile) so the tasks + // SyncedMap is rebuilt with the client while it is still empty. + svc.SetTaskSyncNATS(m.jobNats) if m.jobDBStore != nil { svc.SetDistributedJobStore(m.jobDBStore) // Load tasks/jobs from DB immediately (per-user services skip Start()) @@ -189,6 +197,12 @@ func (m *UserServicesManager) SetJobDBStore(s *jobs.JobStore) { m.jobDBStore = s } +// SetJobSyncNATS sets the NATS client used to keep per-user agent tasks consistent +// across replicas. +func (m *UserServicesManager) SetJobSyncNATS(nats messaging.MessagingClient) { + m.jobNats = nats +} + // ListAllUserIDs returns all user IDs that have scoped data directories. func (m *UserServicesManager) ListAllUserIDs() ([]string, error) { return m.storage.ListUserDirs() diff --git a/core/services/distributed/finetune.go b/core/services/distributed/finetune.go index 49144c32d..1f4cfeb8f 100644 --- a/core/services/distributed/finetune.go +++ b/core/services/distributed/finetune.go @@ -8,6 +8,7 @@ import ( "github.com/google/uuid" "github.com/mudler/LocalAI/core/services/advisorylock" "gorm.io/gorm" + "gorm.io/gorm/clause" ) // FineTuneJobRecord tracks fine-tune jobs in PostgreSQL. @@ -80,6 +81,34 @@ func (s *FineTuneStore) List(userID string) ([]FineTuneJobRecord, error) { return jobs, q.Find(&jobs).Error } +// ListAll returns every fine-tune job across all users. The SyncedMap that backs +// FineTuneService is a single global map (the REST API filters by user at read +// time), so hydrate needs the full set rather than the per-user List above. +func (s *FineTuneStore) ListAll() ([]FineTuneJobRecord, error) { + var jobs []FineTuneJobRecord + return jobs, s.db.Order("created_at DESC").Find(&jobs).Error +} + +// Upsert idempotently inserts or fully replaces a job row by primary key. The +// SyncedMap write-through path issues a single Set per mutation regardless of +// whether the job already exists, so it needs one create-or-update primitive +// (Create alone fails on a duplicate key, UpdateStatus alone misses new rows and +// only touches a few columns). +func (s *FineTuneStore) Upsert(job *FineTuneJobRecord) error { + if job.ID == "" { + job.ID = uuid.New().String() + } + now := time.Now() + if job.CreatedAt.IsZero() { + job.CreatedAt = now + } + job.UpdatedAt = now + return s.db.Clauses(clause.OnConflict{ + Columns: []clause.Column{{Name: "id"}}, + UpdateAll: true, + }).Create(job).Error +} + // UpdateStatus updates the status and message of a fine-tune job. func (s *FineTuneStore) UpdateStatus(id, status, message string) error { return s.db.Model(&FineTuneJobRecord{}).Where("id = ?", id).Updates(map[string]any{ diff --git a/core/services/distributed/finetune_suite_test.go b/core/services/distributed/finetune_suite_test.go new file mode 100644 index 000000000..87add73fd --- /dev/null +++ b/core/services/distributed/finetune_suite_test.go @@ -0,0 +1,13 @@ +package distributed_test + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestDistributed(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Distributed Suite") +} diff --git a/core/services/distributed/finetune_test.go b/core/services/distributed/finetune_test.go new file mode 100644 index 000000000..cf92b5cf4 --- /dev/null +++ b/core/services/distributed/finetune_test.go @@ -0,0 +1,61 @@ +package distributed_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/services/distributed" + "github.com/mudler/LocalAI/core/services/testutil" +) + +var _ = Describe("FineTuneStore", func() { + var store *distributed.FineTuneStore + + BeforeEach(func() { + db := testutil.SetupTestDB() + var err error + store, err = distributed.NewFineTuneStore(db) + Expect(err).ToNot(HaveOccurred()) + }) + + Describe("ListAll", func() { + It("returns jobs across all users (unlike per-user List)", func() { + Expect(store.Create(&distributed.FineTuneJobRecord{ID: "j1", UserID: "u1", Status: "queued"})).To(Succeed()) + Expect(store.Create(&distributed.FineTuneJobRecord{ID: "j2", UserID: "u2", Status: "queued"})).To(Succeed()) + + all, err := store.ListAll() + Expect(err).ToNot(HaveOccurred()) + Expect(all).To(HaveLen(2)) + + perUser, err := store.List("u1") + Expect(err).ToNot(HaveOccurred()) + Expect(perUser).To(HaveLen(1), "List stays per-user") + }) + }) + + Describe("Upsert", func() { + It("inserts a new row", func() { + Expect(store.Upsert(&distributed.FineTuneJobRecord{ID: "up-1", UserID: "u1", Status: "queued"})).To(Succeed()) + + got, err := store.Get("up-1") + Expect(err).ToNot(HaveOccurred()) + Expect(got.Status).To(Equal("queued")) + }) + + It("idempotently updates an existing row on a repeated key", func() { + Expect(store.Upsert(&distributed.FineTuneJobRecord{ID: "up-2", UserID: "u1", Status: "queued"})).To(Succeed()) + // Second Upsert with the same primary key must update, not error on a + // duplicate-key violation (this is the SyncedMap write-through contract). + Expect(store.Upsert(&distributed.FineTuneJobRecord{ID: "up-2", UserID: "u1", Status: "completed", Message: "done"})).To(Succeed()) + + got, err := store.Get("up-2") + Expect(err).ToNot(HaveOccurred()) + Expect(got.Status).To(Equal("completed")) + Expect(got.Message).To(Equal("done")) + + all, err := store.ListAll() + Expect(err).ToNot(HaveOccurred()) + Expect(all).To(HaveLen(1), "upsert must not create a duplicate") + }) + }) +}) diff --git a/core/services/distributed/init.go b/core/services/distributed/init.go index 0ccbe5969..ac28441e2 100644 --- a/core/services/distributed/init.go +++ b/core/services/distributed/init.go @@ -11,6 +11,7 @@ import ( type Stores struct { Gallery *GalleryStore FineTune *FineTuneStore + Quant *QuantStore Skills *SkillStore } @@ -26,15 +27,21 @@ func InitStores(db *gorm.DB) (*Stores, error) { return nil, fmt.Errorf("fine-tune store: %w", err) } + quant, err := NewQuantStore(db) + if err != nil { + return nil, fmt.Errorf("quantization store: %w", err) + } + skills, err := NewSkillStore(db) if err != nil { return nil, fmt.Errorf("skills store: %w", err) } - xlog.Info("Distributed stores initialized (Gallery, FineTune, Skills)") + xlog.Info("Distributed stores initialized (Gallery, FineTune, Quant, Skills)") return &Stores{ Gallery: gallery, FineTune: ft, + Quant: quant, Skills: skills, }, nil } diff --git a/core/services/distributed/quant.go b/core/services/distributed/quant.go new file mode 100644 index 000000000..cba032f4d --- /dev/null +++ b/core/services/distributed/quant.go @@ -0,0 +1,105 @@ +package distributed + +import ( + "context" + "fmt" + "time" + + "github.com/google/uuid" + "github.com/mudler/LocalAI/core/services/advisorylock" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +// QuantJobRecord tracks quantization jobs in PostgreSQL. The columns mirror the +// API shape (schema.QuantizationJob); the structured Config and ExtraOptions are +// serialized into JSON text columns so a record fully reconstructs the job. +type QuantJobRecord struct { + ID string `gorm:"primaryKey;size:36" json:"id"` + UserID string `gorm:"index;size:36" json:"user_id,omitempty"` + Model string `gorm:"size:255" json:"model"` + Backend string `gorm:"size:64" json:"backend"` + ModelID string `gorm:"size:255" json:"model_id,omitempty"` + QuantizationType string `gorm:"size:32" json:"quantization_type"` + Status string `gorm:"index;size:32;default:queued" json:"status"` // queued, downloading, converting, quantizing, completed, failed, stopped + Message string `gorm:"type:text" json:"message,omitempty"` + OutputDir string `gorm:"size:512" json:"output_dir,omitempty"` + OutputFile string `gorm:"size:512" json:"output_file,omitempty"` + ConfigJSON string `gorm:"column:config;type:text" json:"-"` + ExtraOptsJSON string `gorm:"column:extra_options;type:text" json:"-"` + ImportStatus string `gorm:"size:32" json:"import_status,omitempty"` + ImportMessage string `gorm:"type:text" json:"import_message,omitempty"` + ImportModelName string `gorm:"size:255" json:"import_model_name,omitempty"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` +} + +func (QuantJobRecord) TableName() string { return "quantization_jobs" } + +// QuantStore manages quantization job state in PostgreSQL. +type QuantStore struct { + db *gorm.DB +} + +// NewQuantStore creates a new QuantStore and auto-migrates. +// Uses a PostgreSQL advisory lock to prevent concurrent migration races +// when multiple instances (frontend + workers) start at the same time. +func NewQuantStore(db *gorm.DB) (*QuantStore, error) { + if err := advisorylock.WithLockCtx(context.Background(), db, advisorylock.KeySchemaMigrate, func() error { + return db.AutoMigrate(&QuantJobRecord{}) + }); err != nil { + return nil, fmt.Errorf("migrating quantization_jobs: %w", err) + } + return &QuantStore{db: db}, nil +} + +// Create stores a new quantization job. +func (s *QuantStore) Create(job *QuantJobRecord) error { + if job.ID == "" { + job.ID = uuid.New().String() + } + job.CreatedAt = time.Now() + job.UpdatedAt = job.CreatedAt + return s.db.Create(job).Error +} + +// Get retrieves a quantization job by ID. +func (s *QuantStore) Get(id string) (*QuantJobRecord, error) { + var job QuantJobRecord + if err := s.db.First(&job, "id = ?", id).Error; err != nil { + return nil, err + } + return &job, nil +} + +// ListAll returns every quantization job across all users. The SyncedMap that +// backs QuantizationService is a single global map (the REST API filters by user +// at read time), so hydrate needs the full set. +func (s *QuantStore) ListAll() ([]QuantJobRecord, error) { + var jobs []QuantJobRecord + return jobs, s.db.Order("created_at DESC").Find(&jobs).Error +} + +// Upsert idempotently inserts or fully replaces a job row by primary key. The +// SyncedMap write-through path issues a single Set per mutation regardless of +// whether the job already exists, so it needs one create-or-update primitive +// (Create alone fails on a duplicate key). +func (s *QuantStore) Upsert(job *QuantJobRecord) error { + if job.ID == "" { + job.ID = uuid.New().String() + } + now := time.Now() + if job.CreatedAt.IsZero() { + job.CreatedAt = now + } + job.UpdatedAt = now + return s.db.Clauses(clause.OnConflict{ + Columns: []clause.Column{{Name: "id"}}, + UpdateAll: true, + }).Create(job).Error +} + +// Delete removes a quantization job. +func (s *QuantStore) Delete(id string) error { + return s.db.Where("id = ?", id).Delete(&QuantJobRecord{}).Error +} diff --git a/core/services/distributed/quant_test.go b/core/services/distributed/quant_test.go new file mode 100644 index 000000000..49ae483f9 --- /dev/null +++ b/core/services/distributed/quant_test.go @@ -0,0 +1,57 @@ +package distributed_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/services/distributed" + "github.com/mudler/LocalAI/core/services/testutil" +) + +var _ = Describe("QuantStore", func() { + var store *distributed.QuantStore + + BeforeEach(func() { + db := testutil.SetupTestDB() + var err error + store, err = distributed.NewQuantStore(db) + Expect(err).ToNot(HaveOccurred()) + }) + + Describe("ListAll", func() { + It("returns jobs across all users", func() { + Expect(store.Create(&distributed.QuantJobRecord{ID: "j1", UserID: "u1", Status: "queued"})).To(Succeed()) + Expect(store.Create(&distributed.QuantJobRecord{ID: "j2", UserID: "u2", Status: "queued"})).To(Succeed()) + + all, err := store.ListAll() + Expect(err).ToNot(HaveOccurred()) + Expect(all).To(HaveLen(2)) + }) + }) + + Describe("Upsert", func() { + It("inserts a new row", func() { + Expect(store.Upsert(&distributed.QuantJobRecord{ID: "up-1", UserID: "u1", Status: "queued"})).To(Succeed()) + + got, err := store.Get("up-1") + Expect(err).ToNot(HaveOccurred()) + Expect(got.Status).To(Equal("queued")) + }) + + It("idempotently updates an existing row on a repeated key", func() { + Expect(store.Upsert(&distributed.QuantJobRecord{ID: "up-2", UserID: "u1", Status: "queued"})).To(Succeed()) + // Second Upsert with the same primary key must update, not error on a + // duplicate-key violation (this is the SyncedMap write-through contract). + Expect(store.Upsert(&distributed.QuantJobRecord{ID: "up-2", UserID: "u1", Status: "completed", Message: "done"})).To(Succeed()) + + got, err := store.Get("up-2") + Expect(err).ToNot(HaveOccurred()) + Expect(got.Status).To(Equal("completed")) + Expect(got.Message).To(Equal("done")) + + all, err := store.ListAll() + Expect(err).ToNot(HaveOccurred()) + Expect(all).To(HaveLen(1), "upsert must not create a duplicate") + }) + }) +}) diff --git a/core/services/finetune/finetune_suite_test.go b/core/services/finetune/finetune_suite_test.go new file mode 100644 index 000000000..fe7deb994 --- /dev/null +++ b/core/services/finetune/finetune_suite_test.go @@ -0,0 +1,13 @@ +package finetune + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestFinetune(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Finetune Suite") +} diff --git a/core/services/finetune/service.go b/core/services/finetune/service.go index 84d50d80e..3e2431df2 100644 --- a/core/services/finetune/service.go +++ b/core/services/finetune/service.go @@ -19,6 +19,7 @@ import ( "github.com/mudler/LocalAI/core/schema" "github.com/mudler/LocalAI/core/services/distributed" "github.com/mudler/LocalAI/core/services/messaging" + "github.com/mudler/LocalAI/core/services/syncstate" pb "github.com/mudler/LocalAI/pkg/grpc/proto" "github.com/mudler/LocalAI/pkg/model" "github.com/mudler/LocalAI/pkg/utils" @@ -32,44 +33,63 @@ type FineTuneService struct { modelLoader *model.ModelLoader configLoader *config.ModelConfigLoader - mu sync.Mutex - jobs map[string]*schema.FineTuneJob + // mu serializes the read-modify-write of job values. The SyncedMap guards its + // own map structure, but a job is a pointer mutated in place (e.g. the export + // goroutine), so the service still needs a lock to keep those field updates + // and the subsequent Set atomic with respect to readers. + mu sync.Mutex - // Distributed mode (nil when not in distributed mode) - natsClient messaging.Publisher - fineTuneStore *distributed.FineTuneStore + // jobs is the cross-replica job store: an in-memory map kept consistent across + // replicas via NATS, optionally read-through to PostgreSQL in distributed mode. + jobs *syncstate.SyncedMap[string, *schema.FineTuneJob] } -// SetNATSClient sets the NATS client for distributed progress publishing. -func (s *FineTuneService) SetNATSClient(nc messaging.Publisher) { - s.mu.Lock() - defer s.mu.Unlock() - s.natsClient = nc -} - -// SetFineTuneStore sets the PostgreSQL fine-tune store for distributed persistence. -func (s *FineTuneService) SetFineTuneStore(store *distributed.FineTuneStore) { - s.mu.Lock() - defer s.mu.Unlock() - s.fineTuneStore = store -} - -// NewFineTuneService creates a new FineTuneService. +// NewFineTuneService creates a new FineTuneService. In distributed mode pass the +// shared NATS client and PostgreSQL store so jobs stay consistent across +// replicas; pass nil for both in standalone mode, where the disk Loader hydrates +// the map and there is nothing to broadcast. func NewFineTuneService( appConfig *config.ApplicationConfig, modelLoader *model.ModelLoader, configLoader *config.ModelConfigLoader, + nats messaging.MessagingClient, + store *distributed.FineTuneStore, ) *FineTuneService { s := &FineTuneService{ appConfig: appConfig, modelLoader: modelLoader, configLoader: configLoader, - jobs: make(map[string]*schema.FineTuneJob), } - s.loadAllJobs() + + // Only attach a Store interface when a concrete store exists, otherwise the + // SyncedMap would see a non-nil interface wrapping a nil pointer and try to + // hydrate/write through a nil DB. + var syncStore syncstate.Store[string, *schema.FineTuneJob] + if store != nil { + syncStore = &fineTuneStoreAdapter{store: store} + } + + s.jobs = syncstate.New(syncstate.Config[string, *schema.FineTuneJob]{ + Name: "finetune.jobs", + Key: func(j *schema.FineTuneJob) string { return j.ID }, + Nats: nats, + Store: syncStore, + Loader: s.loadJobsFromDisk, // ignored when Store is set (distributed mode) + }) + + // Hydrate + subscribe. A hydrate failure must not take the server down: log + // and continue degraded (standalone), mirroring the OpCache wiring. + if err := s.jobs.Start(appConfig.Context); err != nil { + xlog.Warn("FineTune SyncedMap start failed; running degraded", "error", err) + } return s } +// Close releases the SyncedMap subscription and background workers. +func (s *FineTuneService) Close() error { + return s.jobs.Close() +} + // fineTuneBaseDir returns the base directory for fine-tune job data. func (s *FineTuneService) fineTuneBaseDir() string { return filepath.Join(s.appConfig.DataPath, "fine-tune") @@ -100,15 +120,18 @@ func (s *FineTuneService) saveJobState(job *schema.FineTuneJob) { } } -// loadAllJobs scans the fine-tune directory for persisted jobs and loads them. -func (s *FineTuneService) loadAllJobs() { +// loadJobsFromDisk scans the fine-tune directory for persisted jobs and returns +// them. It is the SyncedMap Loader used in standalone mode (no DB); the returned +// slice hydrates the map on Start. +func (s *FineTuneService) loadJobsFromDisk(_ context.Context) ([]*schema.FineTuneJob, error) { baseDir := s.fineTuneBaseDir() entries, err := os.ReadDir(baseDir) if err != nil { - // Directory doesn't exist yet — that's fine - return + // Directory doesn't exist yet — that's fine, start empty. + return nil, nil } + var jobs []*schema.FineTuneJob for _, entry := range entries { if !entry.IsDir() { continue @@ -137,12 +160,13 @@ func (s *FineTuneService) loadAllJobs() { job.ExportMessage = "Server restarted while export was running" } - s.jobs[job.ID] = &job + jobs = append(jobs, &job) } - if len(s.jobs) > 0 { - xlog.Info("Loaded persisted fine-tune jobs", "count", len(s.jobs)) + if len(jobs) > 0 { + xlog.Info("Loaded persisted fine-tune jobs", "count", len(jobs)) } + return jobs, nil } // StartJob starts a new fine-tuning job. @@ -236,27 +260,13 @@ func (s *FineTuneService) StartJob(ctx context.Context, userID string, req schem CreatedAt: time.Now().UTC().Format(time.RFC3339), Config: &req, } - s.jobs[jobID] = job - s.saveJobState(job) - - // Persist to PostgreSQL in distributed mode - if s.fineTuneStore != nil { - configJSON, _ := json.Marshal(req) - extraJSON, _ := json.Marshal(req.ExtraOptions) - s.fineTuneStore.Create(&distributed.FineTuneJobRecord{ - ID: jobID, - UserID: userID, - Model: req.Model, - Backend: backendName, - ModelID: modelID, - TrainingType: req.TrainingType, - TrainingMethod: req.TrainingMethod, - Status: "queued", - OutputDir: outputDir, - ConfigJSON: string(configJSON), - ExtraOptsJSON: string(extraJSON), - }) + // Set write-through persists to PostgreSQL (distributed) and broadcasts to + // peer replicas; the disk state.json is written separately for restart + // recovery / standalone hydrate. + if err := s.jobs.Set(ctx, job); err != nil { + return nil, fmt.Errorf("failed to persist job: %w", err) } + s.saveJobState(job) return &schema.FineTuneJobResponse{ ID: jobID, @@ -270,7 +280,7 @@ func (s *FineTuneService) GetJob(userID, jobID string) (*schema.FineTuneJob, err s.mu.Lock() defer s.mu.Unlock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { return nil, fmt.Errorf("job not found: %s", jobID) } @@ -286,7 +296,7 @@ func (s *FineTuneService) ListJobs(userID string) []*schema.FineTuneJob { defer s.mu.Unlock() var result []*schema.FineTuneJob - for _, job := range s.jobs { + for _, job := range s.jobs.List() { if userID == "" || job.UserID == userID { result = append(result, job) } @@ -302,7 +312,7 @@ func (s *FineTuneService) ListJobs(userID string) []*schema.FineTuneJob { // StopJob stops a running fine-tuning job. func (s *FineTuneService) StopJob(ctx context.Context, userID, jobID string, saveCheckpoint bool) error { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return fmt.Errorf("job not found: %s", jobID) @@ -323,10 +333,10 @@ func (s *FineTuneService) StopJob(ctx context.Context, userID, jobID string, sav s.mu.Lock() job.Status = "stopped" job.Message = "Training stopped by user" - s.saveJobState(job) - if s.fineTuneStore != nil { - s.fineTuneStore.UpdateStatus(jobID, "stopped", "Training stopped by user") + if err := s.jobs.Set(ctx, job); err != nil { + xlog.Warn("Failed to persist stopped job", "job_id", jobID, "error", err) } + s.saveJobState(job) s.mu.Unlock() return nil @@ -335,7 +345,7 @@ func (s *FineTuneService) StopJob(ctx context.Context, userID, jobID string, sav // DeleteJob removes a fine-tuning job and its associated data from disk. func (s *FineTuneService) DeleteJob(userID, jobID string) error { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return fmt.Errorf("job not found: %s", jobID) @@ -360,9 +370,10 @@ func (s *FineTuneService) DeleteJob(userID, jobID string) error { } exportModelName := job.ExportModelName - delete(s.jobs, jobID) - if s.fineTuneStore != nil { - s.fineTuneStore.Delete(jobID) + // Delete write-through removes the DB row (distributed) and broadcasts the + // removal to peer replicas. DeleteJob has no ctx, so use Background. + if err := s.jobs.Delete(context.Background(), jobID); err != nil { + xlog.Warn("Failed to delete job from store", "job_id", jobID, "error", err) } s.mu.Unlock() @@ -398,7 +409,7 @@ func (s *FineTuneService) DeleteJob(userID, jobID string) error { // StreamProgress opens a gRPC progress stream and calls the callback for each update. func (s *FineTuneService) StreamProgress(ctx context.Context, userID, jobID string, callback func(event *schema.FineTuneProgressEvent)) error { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return fmt.Errorf("job not found: %s", jobID) @@ -427,7 +438,7 @@ func (s *FineTuneService) StreamProgress(ctx context.Context, userID, jobID stri }, func(update *pb.FineTuneProgressUpdate) { // Update job status and persist s.mu.Lock() - if j, ok := s.jobs[jobID]; ok { + if j, ok := s.jobs.Get(jobID); ok { // Don't let progress updates overwrite terminal states isTerminal := j.Status == "stopped" || j.Status == "completed" || j.Status == "failed" if !isTerminal { @@ -436,10 +447,10 @@ func (s *FineTuneService) StreamProgress(ctx context.Context, userID, jobID stri if update.Message != "" { j.Message = update.Message } - s.saveJobState(j) - if s.fineTuneStore != nil { - s.fineTuneStore.UpdateStatus(jobID, j.Status, j.Message) + if err := s.jobs.Set(ctx, j); err != nil { + xlog.Warn("Failed to persist progress update", "job_id", jobID, "error", err) } + s.saveJobState(j) } s.mu.Unlock() @@ -474,7 +485,7 @@ func (s *FineTuneService) StreamProgress(ctx context.Context, userID, jobID stri // ListCheckpoints lists checkpoints for a job. func (s *FineTuneService) ListCheckpoints(ctx context.Context, userID, jobID string) ([]*pb.CheckpointInfo, error) { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return nil, fmt.Errorf("job not found: %s", jobID) @@ -520,7 +531,7 @@ func sanitizeModelName(s string) string { // ExportModel starts an async model export from a checkpoint and returns the intended model name immediately. func (s *FineTuneService) ExportModel(ctx context.Context, userID, jobID string, req schema.ExportRequest) (string, error) { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return "", fmt.Errorf("job not found: %s", jobID) @@ -572,6 +583,9 @@ func (s *FineTuneService) ExportModel(ctx context.Context, userID, jobID string, job.ExportStatus = "exporting" job.ExportMessage = "" job.ExportModelName = "" + if err := s.jobs.Set(ctx, job); err != nil { + xlog.Warn("Failed to persist export start", "job_id", jobID, "error", err) + } s.saveJobState(job) s.mu.Unlock() @@ -662,24 +676,30 @@ func (s *FineTuneService) ExportModel(ctx context.Context, userID, jobID string, xlog.Info("Model exported and registered", "job_id", jobID, "model_name", modelName, "format", req.ExportFormat) + // Runs after the HTTP request returns, so use Background rather than the + // (now likely cancelled) request ctx for the write-through. s.mu.Lock() job.ExportStatus = "completed" job.ExportModelName = modelName job.ExportMessage = "" - s.saveJobState(job) - if s.fineTuneStore != nil { - s.fineTuneStore.UpdateExportStatus(jobID, "completed", "", modelName) + if err := s.jobs.Set(context.Background(), job); err != nil { + xlog.Warn("Failed to persist export completion", "job_id", jobID, "error", err) } + s.saveJobState(job) s.mu.Unlock() }() return modelName, nil } -// setExportMessage updates the export message and persists the job state. +// setExportMessage updates the export message and persists the job state. Called +// from the background export goroutine, so it uses Background for write-through. func (s *FineTuneService) setExportMessage(job *schema.FineTuneJob, msg string) { s.mu.Lock() job.ExportMessage = msg + if err := s.jobs.Set(context.Background(), job); err != nil { + xlog.Warn("Failed to persist export message", "job_id", job.ID, "error", err) + } s.saveJobState(job) s.mu.Unlock() } @@ -687,7 +707,7 @@ func (s *FineTuneService) setExportMessage(job *schema.FineTuneJob, msg string) // GetExportedModelPath returns the path to the exported model directory and its name. func (s *FineTuneService) GetExportedModelPath(userID, jobID string) (string, string, error) { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return "", "", fmt.Errorf("job not found: %s", jobID) @@ -723,10 +743,10 @@ func (s *FineTuneService) setExportFailed(job *schema.FineTuneJob, message strin s.mu.Lock() job.ExportStatus = "failed" job.ExportMessage = message - s.saveJobState(job) - if s.fineTuneStore != nil { - s.fineTuneStore.UpdateExportStatus(job.ID, "failed", message, "") + if err := s.jobs.Set(context.Background(), job); err != nil { + xlog.Warn("Failed to persist export failure", "job_id", job.ID, "error", err) } + s.saveJobState(job) s.mu.Unlock() } diff --git a/core/services/finetune/service_test.go b/core/services/finetune/service_test.go new file mode 100644 index 000000000..dc7c53290 --- /dev/null +++ b/core/services/finetune/service_test.go @@ -0,0 +1,185 @@ +package finetune + +// White-box tests (package finetune) so a spec can drive the service's internal +// SyncedMap the same way StartJob does (via jobs.Set) without standing up a +// training backend, then assert the cross-replica reads (GetJob/ListJobs) and +// the adapter conversions that keep REST responses byte-for-byte unchanged. + +import ( + "context" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/core/services/distributed" + "github.com/mudler/LocalAI/core/services/testutil" +) + +// newTestService builds a standalone FineTuneService wired to the given bus. The +// model/config loaders are nil because the read/sync paths under test never touch +// them; the data dir is a throwaway temp dir so the disk Loader finds nothing. +func newTestService(bus *testutil.FakeBus) *FineTuneService { + appConfig := &config.ApplicationConfig{ + Context: context.Background(), + DataPath: GinkgoT().TempDir(), + } + return NewFineTuneService(appConfig, nil, nil, bus, nil) +} + +var _ = Describe("FineTuneService", func() { + ctx := context.Background() + + Describe("cross-replica job visibility", func() { + var ( + bus *testutil.FakeBus + a, b *FineTuneService + ) + + BeforeEach(func() { + // One shared bus, two replicas: exactly the distributed topology where + // a round-robin request may land on a replica that did not originate + // the change. + bus = testutil.NewFakeBus() + a = newTestService(bus) + b = newTestService(bus) + }) + + AfterEach(func() { + Expect(a.Close()).To(Succeed()) + Expect(b.Close()).To(Succeed()) + }) + + It("makes a job created on A visible via B's GetJob and ListJobs", func() { + job := &schema.FineTuneJob{ID: "job-1", UserID: "user-1", Status: "queued", CreatedAt: "2026-06-27T10:00:00Z"} + // StartJob persists via jobs.Set; drive that directly to avoid a backend. + Expect(a.jobs.Set(ctx, job)).To(Succeed()) + + got, err := b.GetJob("user-1", "job-1") + Expect(err).ToNot(HaveOccurred(), "B must see a job A just created") + Expect(got.Status).To(Equal("queued")) + + listed := b.ListJobs("user-1") + Expect(listed).To(HaveLen(1)) + Expect(listed[0].ID).To(Equal("job-1")) + }) + + It("removes a job from B when it is deleted on A", func() { + job := &schema.FineTuneJob{ID: "job-2", UserID: "user-1", Status: "completed", CreatedAt: "2026-06-27T10:00:00Z"} + Expect(a.jobs.Set(ctx, job)).To(Succeed()) + _, err := b.GetJob("user-1", "job-2") + Expect(err).ToNot(HaveOccurred(), "precondition: B must have the job before the delete") + + Expect(a.jobs.Delete(ctx, "job-2")).To(Succeed()) + + _, err = b.GetJob("user-1", "job-2") + Expect(err).To(HaveOccurred(), "a delete on A must remove the job from B") + }) + + It("propagates a status update from A to B", func() { + job := &schema.FineTuneJob{ID: "job-3", UserID: "user-1", Status: "training", CreatedAt: "2026-06-27T10:00:00Z"} + Expect(a.jobs.Set(ctx, job)).To(Succeed()) + + updated := &schema.FineTuneJob{ID: "job-3", UserID: "user-1", Status: "completed", CreatedAt: "2026-06-27T10:00:00Z"} + Expect(a.jobs.Set(ctx, updated)).To(Succeed()) + + got, err := b.GetJob("user-1", "job-3") + Expect(err).ToNot(HaveOccurred()) + Expect(got.Status).To(Equal("completed")) + }) + }) + + Describe("ListJobs", func() { + var svc *FineTuneService + + BeforeEach(func() { + svc = newTestService(testutil.NewFakeBus()) + }) + AfterEach(func() { Expect(svc.Close()).To(Succeed()) }) + + It("filters by user and sorts newest-first", func() { + Expect(svc.jobs.Set(ctx, &schema.FineTuneJob{ID: "old", UserID: "u1", CreatedAt: "2026-06-25T10:00:00Z"})).To(Succeed()) + Expect(svc.jobs.Set(ctx, &schema.FineTuneJob{ID: "new", UserID: "u1", CreatedAt: "2026-06-27T10:00:00Z"})).To(Succeed()) + Expect(svc.jobs.Set(ctx, &schema.FineTuneJob{ID: "other", UserID: "u2", CreatedAt: "2026-06-26T10:00:00Z"})).To(Succeed()) + + jobs := svc.ListJobs("u1") + Expect(jobs).To(HaveLen(2), "only u1's jobs") + Expect(jobs[0].ID).To(Equal("new"), "newest first") + Expect(jobs[1].ID).To(Equal("old")) + }) + + It("returns every user's jobs when the userID filter is empty", func() { + Expect(svc.jobs.Set(ctx, &schema.FineTuneJob{ID: "a", UserID: "u1", CreatedAt: "2026-06-25T10:00:00Z"})).To(Succeed()) + Expect(svc.jobs.Set(ctx, &schema.FineTuneJob{ID: "b", UserID: "u2", CreatedAt: "2026-06-26T10:00:00Z"})).To(Succeed()) + + Expect(svc.ListJobs("")).To(HaveLen(2)) + }) + + It("rejects GetJob for a job owned by another user", func() { + Expect(svc.jobs.Set(ctx, &schema.FineTuneJob{ID: "x", UserID: "owner", CreatedAt: "2026-06-25T10:00:00Z"})).To(Succeed()) + + _, err := svc.GetJob("intruder", "x") + Expect(err).To(HaveOccurred(), "a different user must not read someone else's job") + }) + }) + + Describe("store adapter conversion", func() { + // The SyncedMap value type is *schema.FineTuneJob (the exact REST shape). + // These specs prove the DB adapter round-trips it losslessly, so hydrate + // and write-through in distributed mode keep responses unchanged. + It("round-trips a job through jobToRecord/recordToJob preserving the API shape", func() { + original := &schema.FineTuneJob{ + ID: "rt-1", + UserID: "user-1", + Model: "base-model", + Backend: "trl", + ModelID: "trl-finetune-rt-1", + TrainingType: "lora", + TrainingMethod: "sft", + Status: "completed", + Message: "done", + OutputDir: "/data/fine-tune/rt-1", + ExtraOptions: map[string]string{"hf_token": "secret"}, + CreatedAt: "2026-06-27T10:00:00Z", + ExportStatus: "completed", + ExportMessage: "", + ExportModelName: "base-model-ft-rt-1", + Config: &schema.FineTuneJobRequest{Model: "base-model", Backend: "trl", DatasetSource: "data.jsonl"}, + } + + rec := jobToRecord(original) + Expect(rec.ID).To(Equal("rt-1")) + Expect(rec.ConfigJSON).ToNot(BeEmpty(), "structured config must serialize into the JSON column") + Expect(rec.ExtraOptsJSON).ToNot(BeEmpty()) + + back := recordToJob(rec) + Expect(back.ID).To(Equal(original.ID)) + Expect(back.UserID).To(Equal(original.UserID)) + Expect(back.Model).To(Equal(original.Model)) + Expect(back.Backend).To(Equal(original.Backend)) + Expect(back.ModelID).To(Equal(original.ModelID)) + Expect(back.TrainingType).To(Equal(original.TrainingType)) + Expect(back.TrainingMethod).To(Equal(original.TrainingMethod)) + Expect(back.Status).To(Equal(original.Status)) + Expect(back.Message).To(Equal(original.Message)) + Expect(back.OutputDir).To(Equal(original.OutputDir)) + Expect(back.ExportStatus).To(Equal(original.ExportStatus)) + Expect(back.ExportModelName).To(Equal(original.ExportModelName)) + Expect(back.CreatedAt).To(Equal(original.CreatedAt)) + Expect(back.ExtraOptions).To(Equal(original.ExtraOptions)) + Expect(back.Config).ToNot(BeNil()) + Expect(back.Config.DatasetSource).To(Equal("data.jsonl")) + }) + }) + + Describe("compile-time adapter contract", func() { + It("satisfies syncstate.Store for *distributed.FineTuneStore", func() { + // Guards against drift between the adapter and the component interface; + // the var assertion in syncstore.go covers it at build time, this keeps + // the type referenced from a spec too. + var _ *distributed.FineTuneStore + Expect(&fineTuneStoreAdapter{}).ToNot(BeNil()) + }) + }) +}) diff --git a/core/services/finetune/syncstore.go b/core/services/finetune/syncstore.go new file mode 100644 index 000000000..e5bd8239c --- /dev/null +++ b/core/services/finetune/syncstore.go @@ -0,0 +1,114 @@ +package finetune + +import ( + "context" + "encoding/json" + "time" + + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/core/services/distributed" + "github.com/mudler/LocalAI/core/services/syncstate" +) + +// fineTuneStoreAdapter bridges the distributed PostgreSQL FineTuneStore to the +// generic syncstate.Store the SyncedMap consumes. It is only wired in distributed +// mode; standalone leaves Store nil and hydrates from disk via a Loader instead. +// +// The SyncedMap value type is *schema.FineTuneJob (the exact shape the REST API +// returns) so reads need no conversion and the response JSON is provably +// unchanged. The adapter is the single place that translates between that API +// shape and the DB FineTuneJobRecord. +type fineTuneStoreAdapter struct { + store *distributed.FineTuneStore +} + +// compile-time assertion that the adapter satisfies the component's Store. +var _ syncstate.Store[string, *schema.FineTuneJob] = (*fineTuneStoreAdapter)(nil) + +func (a *fineTuneStoreAdapter) List(_ context.Context) ([]*schema.FineTuneJob, error) { + records, err := a.store.ListAll() + if err != nil { + return nil, err + } + jobs := make([]*schema.FineTuneJob, 0, len(records)) + for i := range records { + jobs = append(jobs, recordToJob(&records[i])) + } + return jobs, nil +} + +func (a *fineTuneStoreAdapter) Upsert(_ context.Context, job *schema.FineTuneJob) error { + return a.store.Upsert(jobToRecord(job)) +} + +func (a *fineTuneStoreAdapter) Delete(_ context.Context, id string) error { + return a.store.Delete(id) +} + +// recordToJob maps a persisted DB record back to the API shape, reconstructing +// the structured Config / ExtraOptions from their JSON columns. +func recordToJob(r *distributed.FineTuneJobRecord) *schema.FineTuneJob { + job := &schema.FineTuneJob{ + ID: r.ID, + UserID: r.UserID, + Model: r.Model, + Backend: r.Backend, + ModelID: r.ModelID, + TrainingType: r.TrainingType, + TrainingMethod: r.TrainingMethod, + Status: r.Status, + Message: r.Message, + OutputDir: r.OutputDir, + ExportStatus: r.ExportStatus, + ExportMessage: r.ExportMessage, + ExportModelName: r.ExportModelName, + CreatedAt: r.CreatedAt.UTC().Format(time.RFC3339), + } + if r.ExtraOptsJSON != "" { + // Best-effort: a malformed column must not drop the whole job from the API. + _ = json.Unmarshal([]byte(r.ExtraOptsJSON), &job.ExtraOptions) + } + if r.ConfigJSON != "" { + var cfg schema.FineTuneJobRequest + if err := json.Unmarshal([]byte(r.ConfigJSON), &cfg); err == nil { + job.Config = &cfg + } + } + return job +} + +// jobToRecord maps the API shape to a DB record for write-through, serializing +// the structured Config / ExtraOptions into their JSON columns. CreatedAt is +// parsed back from the RFC3339 string the service stamps; an unparseable value +// is left zero so FineTuneStore.Upsert stamps "now". +func jobToRecord(job *schema.FineTuneJob) *distributed.FineTuneJobRecord { + rec := &distributed.FineTuneJobRecord{ + ID: job.ID, + UserID: job.UserID, + Model: job.Model, + Backend: job.Backend, + ModelID: job.ModelID, + TrainingType: job.TrainingType, + TrainingMethod: job.TrainingMethod, + Status: job.Status, + Message: job.Message, + OutputDir: job.OutputDir, + ExportStatus: job.ExportStatus, + ExportMessage: job.ExportMessage, + ExportModelName: job.ExportModelName, + } + if job.Config != nil { + if data, err := json.Marshal(job.Config); err == nil { + rec.ConfigJSON = string(data) + } + } + if job.ExtraOptions != nil { + if data, err := json.Marshal(job.ExtraOptions); err == nil { + rec.ExtraOptsJSON = string(data) + } + } + if t, err := time.Parse(time.RFC3339, job.CreatedAt); err == nil { + rec.CreatedAt = t + } + return rec +} diff --git a/core/services/messaging/client.go b/core/services/messaging/client.go index 31257f1fd..e01c7d9ca 100644 --- a/core/services/messaging/client.go +++ b/core/services/messaging/client.go @@ -22,6 +22,14 @@ const subscribeConfirmTimeout = 5 * time.Second type Client struct { conn *nats.Conn mu sync.RWMutex + + // reconnectCbs are invoked after the underlying connection is + // re-established. nats.go transparently resubscribes existing + // subscriptions on reconnect, but it cannot know that a consumer kept + // derived in-memory state (e.g. syncstate.SyncedMap) that may have drifted + // while the link was down — these callbacks let such consumers re-hydrate. + cbMu sync.Mutex + reconnectCbs []func() } // New creates a new NATS client with auto-reconnect. @@ -31,6 +39,10 @@ func New(url string, opts ...Option) (*Client, error) { o(&cfg) } + // Allocate the client up front so the reconnect handler closure can reach + // it; conn is populated after nats.Connect succeeds below. + c := &Client{} + natsOpts := []nats.Option{ nats.RetryOnFailedConnect(true), nats.MaxReconnects(-1), @@ -41,6 +53,7 @@ func New(url string, opts ...Option) (*Client, error) { }), nats.ReconnectHandler(func(_ *nats.Conn) { xlog.Info("NATS reconnected") + c.runReconnectCallbacks() }), nats.ClosedHandler(func(_ *nats.Conn) { xlog.Info("NATS connection closed") @@ -103,7 +116,33 @@ func New(url string, opts ...Option) (*Client, error) { return nil, fmt.Errorf("connecting to NATS at %s: %w", sanitize.URL(url), err) } - return &Client{conn: nc}, nil + c.conn = nc + return c, nil +} + +// OnReconnect registers a callback invoked after the NATS connection is +// re-established. It is consumed via an optional interface type-assertion +// (interface{ OnReconnect(func()) }) rather than being added to MessagingClient, +// so the messaging abstraction stays minimal and standalone/test clients are not +// forced to implement reconnect semantics. A nil callback is ignored. +func (c *Client) OnReconnect(cb func()) { + if cb == nil { + return + } + c.cbMu.Lock() + c.reconnectCbs = append(c.reconnectCbs, cb) + c.cbMu.Unlock() +} + +// runReconnectCallbacks invokes registered reconnect callbacks. It copies the +// slice under the lock so a callback that (re)registers cannot deadlock. +func (c *Client) runReconnectCallbacks() { + c.cbMu.Lock() + cbs := append([]func(){}, c.reconnectCbs...) + c.cbMu.Unlock() + for _, cb := range cbs { + cb() + } } // Publish marshals data as JSON and publishes it to the given subject. diff --git a/core/services/messaging/subjects.go b/core/services/messaging/subjects.go index 7d099460c..d2b11f535 100644 --- a/core/services/messaging/subjects.go +++ b/core/services/messaging/subjects.go @@ -380,6 +380,20 @@ func SubjectCacheInvalidateCollection(name string) string { return "cache.invalidate.collections." + sanitizeSubjectToken(name) } +// SyncedMap State Sync (Pub/Sub — broadcast to all frontends) +// +// The reusable syncstate.SyncedMap component publishes a {op,key,value} delta on +// this subject whenever a replica mutates a piece of cross-replica in-memory +// state. Peers subscribe and apply the delta to their own map, so a round-robin +// API request that lands on a replica which did not originate the change still +// sees it. Convergence on (re)connect is done by re-hydrating from the durable +// source, so no request/reply snapshot subject is needed here. +func SubjectSyncStateDelta(name string) string { + return subjectSyncStatePrefix + sanitizeSubjectToken(name) + ".delta" +} + +const subjectSyncStatePrefix = "state." + // Prefix-Cache Routing Sync (Pub/Sub - broadcast to all frontends) // // Frontends share prefix-cache observations so a request routed to any replica diff --git a/core/services/quantization/quantization_suite_test.go b/core/services/quantization/quantization_suite_test.go new file mode 100644 index 000000000..6fabcd2f5 --- /dev/null +++ b/core/services/quantization/quantization_suite_test.go @@ -0,0 +1,13 @@ +package quantization + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestQuantization(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Quantization Suite") +} diff --git a/core/services/quantization/service.go b/core/services/quantization/service.go index 64325b97c..cd9cbcead 100644 --- a/core/services/quantization/service.go +++ b/core/services/quantization/service.go @@ -17,6 +17,9 @@ import ( "github.com/mudler/LocalAI/core/config" "github.com/mudler/LocalAI/core/gallery/importers" "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/core/services/distributed" + "github.com/mudler/LocalAI/core/services/messaging" + "github.com/mudler/LocalAI/core/services/syncstate" pb "github.com/mudler/LocalAI/pkg/grpc/proto" "github.com/mudler/LocalAI/pkg/model" "github.com/mudler/LocalAI/pkg/utils" @@ -30,26 +33,63 @@ type QuantizationService struct { modelLoader *model.ModelLoader configLoader *config.ModelConfigLoader - mu sync.Mutex - jobs map[string]*schema.QuantizationJob + // mu serializes the read-modify-write of job values. The SyncedMap guards its + // own map structure, but a job is a pointer mutated in place (e.g. the import + // goroutine), so the service still needs a lock to keep those field updates and + // the subsequent Set atomic with respect to readers. + mu sync.Mutex + + // jobs is the cross-replica job store: an in-memory map kept consistent across + // replicas via NATS, optionally read-through to PostgreSQL in distributed mode. + jobs *syncstate.SyncedMap[string, *schema.QuantizationJob] } -// NewQuantizationService creates a new QuantizationService. +// NewQuantizationService creates a new QuantizationService. In distributed mode +// pass the shared NATS client and PostgreSQL store so jobs stay consistent across +// replicas; pass nil for both in standalone mode, where the disk Loader hydrates +// the map and there is nothing to broadcast. func NewQuantizationService( appConfig *config.ApplicationConfig, modelLoader *model.ModelLoader, configLoader *config.ModelConfigLoader, + nats messaging.MessagingClient, + store *distributed.QuantStore, ) *QuantizationService { s := &QuantizationService{ appConfig: appConfig, modelLoader: modelLoader, configLoader: configLoader, - jobs: make(map[string]*schema.QuantizationJob), } - s.loadAllJobs() + + // Only attach a Store interface when a concrete store exists, otherwise the + // SyncedMap would see a non-nil interface wrapping a nil pointer and try to + // hydrate/write through a nil DB. + var syncStore syncstate.Store[string, *schema.QuantizationJob] + if store != nil { + syncStore = &quantStoreAdapter{store: store} + } + + s.jobs = syncstate.New(syncstate.Config[string, *schema.QuantizationJob]{ + Name: "quant.jobs", + Key: func(j *schema.QuantizationJob) string { return j.ID }, + Nats: nats, + Store: syncStore, + Loader: s.loadJobsFromDisk, // ignored when Store is set (distributed mode) + }) + + // Hydrate + subscribe. A hydrate failure must not take the server down: log and + // continue degraded (standalone), mirroring the FineTune/OpCache wiring. + if err := s.jobs.Start(appConfig.Context); err != nil { + xlog.Warn("Quantization SyncedMap start failed; running degraded", "error", err) + } return s } +// Close releases the SyncedMap subscription and background workers. +func (s *QuantizationService) Close() error { + return s.jobs.Close() +} + // quantizationBaseDir returns the base directory for quantization job data. func (s *QuantizationService) quantizationBaseDir() string { return filepath.Join(s.appConfig.DataPath, "quantization") @@ -80,15 +120,18 @@ func (s *QuantizationService) saveJobState(job *schema.QuantizationJob) { } } -// loadAllJobs scans the quantization directory for persisted jobs and loads them. -func (s *QuantizationService) loadAllJobs() { +// loadJobsFromDisk scans the quantization directory for persisted jobs and +// returns them. It is the SyncedMap Loader used in standalone mode (no DB); the +// returned slice hydrates the map on Start. +func (s *QuantizationService) loadJobsFromDisk(_ context.Context) ([]*schema.QuantizationJob, error) { baseDir := s.quantizationBaseDir() entries, err := os.ReadDir(baseDir) if err != nil { - // Directory doesn't exist yet — that's fine - return + // Directory doesn't exist yet — that's fine, start empty. + return nil, nil } + var jobs []*schema.QuantizationJob for _, entry := range entries { if !entry.IsDir() { continue @@ -117,12 +160,13 @@ func (s *QuantizationService) loadAllJobs() { job.ImportMessage = "Server restarted while import was running" } - s.jobs[job.ID] = &job + jobs = append(jobs, &job) } - if len(s.jobs) > 0 { - xlog.Info("Loaded persisted quantization jobs", "count", len(s.jobs)) + if len(jobs) > 0 { + xlog.Info("Loaded persisted quantization jobs", "count", len(jobs)) } + return jobs, nil } // StartJob starts a new quantization job. @@ -188,7 +232,12 @@ func (s *QuantizationService) StartJob(ctx context.Context, userID string, req s CreatedAt: time.Now().UTC().Format(time.RFC3339), Config: &req, } - s.jobs[jobID] = job + // Set write-through persists to PostgreSQL (distributed) and broadcasts to + // peer replicas; the disk state.json is written separately for restart + // recovery / standalone hydrate. + if err := s.jobs.Set(ctx, job); err != nil { + return nil, fmt.Errorf("failed to persist job: %w", err) + } s.saveJobState(job) return &schema.QuantizationJobResponse{ @@ -203,7 +252,7 @@ func (s *QuantizationService) GetJob(userID, jobID string) (*schema.Quantization s.mu.Lock() defer s.mu.Unlock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { return nil, fmt.Errorf("job not found: %s", jobID) } @@ -219,7 +268,7 @@ func (s *QuantizationService) ListJobs(userID string) []*schema.QuantizationJob defer s.mu.Unlock() var result []*schema.QuantizationJob - for _, job := range s.jobs { + for _, job := range s.jobs.List() { if userID == "" || job.UserID == userID { result = append(result, job) } @@ -235,7 +284,7 @@ func (s *QuantizationService) ListJobs(userID string) []*schema.QuantizationJob // StopJob stops a running quantization job. func (s *QuantizationService) StopJob(ctx context.Context, userID, jobID string) error { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return fmt.Errorf("job not found: %s", jobID) @@ -256,6 +305,9 @@ func (s *QuantizationService) StopJob(ctx context.Context, userID, jobID string) s.mu.Lock() job.Status = "stopped" job.Message = "Quantization stopped by user" + if err := s.jobs.Set(ctx, job); err != nil { + xlog.Warn("Failed to persist stopped job", "job_id", jobID, "error", err) + } s.saveJobState(job) s.mu.Unlock() @@ -265,7 +317,7 @@ func (s *QuantizationService) StopJob(ctx context.Context, userID, jobID string) // DeleteJob removes a quantization job and its associated data from disk. func (s *QuantizationService) DeleteJob(userID, jobID string) error { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return fmt.Errorf("job not found: %s", jobID) @@ -289,7 +341,11 @@ func (s *QuantizationService) DeleteJob(userID, jobID string) error { } importModelName := job.ImportModelName - delete(s.jobs, jobID) + // Delete write-through removes the DB row (distributed) and broadcasts the + // removal to peer replicas. DeleteJob has no ctx, so use Background. + if err := s.jobs.Delete(context.Background(), jobID); err != nil { + xlog.Warn("Failed to delete job from store", "job_id", jobID, "error", err) + } s.mu.Unlock() // Remove job directory (state.json, output files) @@ -324,7 +380,7 @@ func (s *QuantizationService) DeleteJob(userID, jobID string) error { // StreamProgress opens a gRPC progress stream and calls the callback for each update. func (s *QuantizationService) StreamProgress(ctx context.Context, userID, jobID string, callback func(event *schema.QuantizationProgressEvent)) error { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return fmt.Errorf("job not found: %s", jobID) @@ -353,7 +409,7 @@ func (s *QuantizationService) StreamProgress(ctx context.Context, userID, jobID }, func(update *pb.QuantizationProgressUpdate) { // Update job status and persist s.mu.Lock() - if j, ok := s.jobs[jobID]; ok { + if j, ok := s.jobs.Get(jobID); ok { // Don't let progress updates overwrite terminal states isTerminal := j.Status == "stopped" || j.Status == "completed" || j.Status == "failed" if !isTerminal { @@ -365,6 +421,9 @@ func (s *QuantizationService) StreamProgress(ctx context.Context, userID, jobID if update.OutputFile != "" { j.OutputFile = update.OutputFile } + if err := s.jobs.Set(ctx, j); err != nil { + xlog.Warn("Failed to persist progress update", "job_id", jobID, "error", err) + } s.saveJobState(j) } s.mu.Unlock() @@ -399,7 +458,7 @@ func sanitizeQuantModelName(s string) string { // ImportModel imports a quantized model into LocalAI asynchronously. func (s *QuantizationService) ImportModel(ctx context.Context, userID, jobID string, req schema.QuantizationImportRequest) (string, error) { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return "", fmt.Errorf("job not found: %s", jobID) @@ -459,6 +518,9 @@ func (s *QuantizationService) ImportModel(ctx context.Context, userID, jobID str job.ImportStatus = "importing" job.ImportMessage = "" job.ImportModelName = "" + if err := s.jobs.Set(ctx, job); err != nil { + xlog.Warn("Failed to persist import start", "job_id", jobID, "error", err) + } s.saveJobState(job) s.mu.Unlock() @@ -514,10 +576,15 @@ func (s *QuantizationService) ImportModel(ctx context.Context, userID, jobID str xlog.Info("Quantized model imported and registered", "job_id", jobID, "model_name", modelName) + // Runs after the HTTP request returns, so use Background rather than the + // (now likely cancelled) request ctx for the write-through. s.mu.Lock() job.ImportStatus = "completed" job.ImportModelName = modelName job.ImportMessage = "" + if err := s.jobs.Set(context.Background(), job); err != nil { + xlog.Warn("Failed to persist import completion", "job_id", jobID, "error", err) + } s.saveJobState(job) s.mu.Unlock() }() @@ -525,10 +592,14 @@ func (s *QuantizationService) ImportModel(ctx context.Context, userID, jobID str return modelName, nil } -// setImportMessage updates the import message and persists the job state. +// setImportMessage updates the import message and persists the job state. Called +// from the background import goroutine, so it uses Background for write-through. func (s *QuantizationService) setImportMessage(job *schema.QuantizationJob, msg string) { s.mu.Lock() job.ImportMessage = msg + if err := s.jobs.Set(context.Background(), job); err != nil { + xlog.Warn("Failed to persist import message", "job_id", job.ID, "error", err) + } s.saveJobState(job) s.mu.Unlock() } @@ -539,6 +610,9 @@ func (s *QuantizationService) setImportFailed(job *schema.QuantizationJob, messa s.mu.Lock() job.ImportStatus = "failed" job.ImportMessage = message + if err := s.jobs.Set(context.Background(), job); err != nil { + xlog.Warn("Failed to persist import failure", "job_id", job.ID, "error", err) + } s.saveJobState(job) s.mu.Unlock() } @@ -546,7 +620,7 @@ func (s *QuantizationService) setImportFailed(job *schema.QuantizationJob, messa // GetOutputPath returns the path to the quantized model file and a download name. func (s *QuantizationService) GetOutputPath(userID, jobID string) (string, string, error) { s.mu.Lock() - job, ok := s.jobs[jobID] + job, ok := s.jobs.Get(jobID) if !ok { s.mu.Unlock() return "", "", fmt.Errorf("job not found: %s", jobID) diff --git a/core/services/quantization/service_test.go b/core/services/quantization/service_test.go new file mode 100644 index 000000000..665728614 --- /dev/null +++ b/core/services/quantization/service_test.go @@ -0,0 +1,187 @@ +package quantization + +// White-box tests (package quantization) so a spec can drive the service's +// internal SyncedMap the same way StartJob does (via jobs.Set) without standing +// up a quantization backend, then assert the cross-replica reads +// (GetJob/ListJobs) and the adapter conversions that keep REST responses +// byte-for-byte unchanged. + +import ( + "context" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/core/services/distributed" + "github.com/mudler/LocalAI/core/services/testutil" +) + +// newTestService builds a standalone QuantizationService wired to the given bus. +// The model/config loaders are nil because the read/sync paths under test never +// touch them; the data dir is a throwaway temp dir so the disk Loader finds +// nothing. +func newTestService(bus *testutil.FakeBus) *QuantizationService { + appConfig := &config.ApplicationConfig{ + Context: context.Background(), + DataPath: GinkgoT().TempDir(), + } + return NewQuantizationService(appConfig, nil, nil, bus, nil) +} + +var _ = Describe("QuantizationService", func() { + ctx := context.Background() + + Describe("cross-replica job visibility", func() { + var ( + bus *testutil.FakeBus + a, b *QuantizationService + ) + + BeforeEach(func() { + // One shared bus, two replicas: exactly the distributed topology where a + // round-robin request may land on a replica that did not originate the + // change. + bus = testutil.NewFakeBus() + a = newTestService(bus) + b = newTestService(bus) + }) + + AfterEach(func() { + Expect(a.Close()).To(Succeed()) + Expect(b.Close()).To(Succeed()) + }) + + It("makes a job created on A visible via B's GetJob and ListJobs", func() { + job := &schema.QuantizationJob{ID: "job-1", UserID: "user-1", Status: "queued", CreatedAt: "2026-06-27T10:00:00Z"} + // StartJob persists via jobs.Set; drive that directly to avoid a backend. + Expect(a.jobs.Set(ctx, job)).To(Succeed()) + + got, err := b.GetJob("user-1", "job-1") + Expect(err).ToNot(HaveOccurred(), "B must see a job A just created") + Expect(got.Status).To(Equal("queued")) + + listed := b.ListJobs("user-1") + Expect(listed).To(HaveLen(1)) + Expect(listed[0].ID).To(Equal("job-1")) + }) + + It("removes a job from B when it is deleted on A", func() { + job := &schema.QuantizationJob{ID: "job-2", UserID: "user-1", Status: "completed", CreatedAt: "2026-06-27T10:00:00Z"} + Expect(a.jobs.Set(ctx, job)).To(Succeed()) + _, err := b.GetJob("user-1", "job-2") + Expect(err).ToNot(HaveOccurred(), "precondition: B must have the job before the delete") + + Expect(a.jobs.Delete(ctx, "job-2")).To(Succeed()) + + _, err = b.GetJob("user-1", "job-2") + Expect(err).To(HaveOccurred(), "a delete on A must remove the job from B") + }) + + It("propagates a status update from A to B", func() { + job := &schema.QuantizationJob{ID: "job-3", UserID: "user-1", Status: "quantizing", CreatedAt: "2026-06-27T10:00:00Z"} + Expect(a.jobs.Set(ctx, job)).To(Succeed()) + + updated := &schema.QuantizationJob{ID: "job-3", UserID: "user-1", Status: "completed", CreatedAt: "2026-06-27T10:00:00Z"} + Expect(a.jobs.Set(ctx, updated)).To(Succeed()) + + got, err := b.GetJob("user-1", "job-3") + Expect(err).ToNot(HaveOccurred()) + Expect(got.Status).To(Equal("completed")) + }) + }) + + Describe("ListJobs", func() { + var svc *QuantizationService + + BeforeEach(func() { + svc = newTestService(testutil.NewFakeBus()) + }) + AfterEach(func() { Expect(svc.Close()).To(Succeed()) }) + + It("filters by user and sorts newest-first", func() { + Expect(svc.jobs.Set(ctx, &schema.QuantizationJob{ID: "old", UserID: "u1", CreatedAt: "2026-06-25T10:00:00Z"})).To(Succeed()) + Expect(svc.jobs.Set(ctx, &schema.QuantizationJob{ID: "new", UserID: "u1", CreatedAt: "2026-06-27T10:00:00Z"})).To(Succeed()) + Expect(svc.jobs.Set(ctx, &schema.QuantizationJob{ID: "other", UserID: "u2", CreatedAt: "2026-06-26T10:00:00Z"})).To(Succeed()) + + jobs := svc.ListJobs("u1") + Expect(jobs).To(HaveLen(2), "only u1's jobs") + Expect(jobs[0].ID).To(Equal("new"), "newest first") + Expect(jobs[1].ID).To(Equal("old")) + }) + + It("returns every user's jobs when the userID filter is empty", func() { + Expect(svc.jobs.Set(ctx, &schema.QuantizationJob{ID: "a", UserID: "u1", CreatedAt: "2026-06-25T10:00:00Z"})).To(Succeed()) + Expect(svc.jobs.Set(ctx, &schema.QuantizationJob{ID: "b", UserID: "u2", CreatedAt: "2026-06-26T10:00:00Z"})).To(Succeed()) + + Expect(svc.ListJobs("")).To(HaveLen(2)) + }) + + It("rejects GetJob for a job owned by another user", func() { + Expect(svc.jobs.Set(ctx, &schema.QuantizationJob{ID: "x", UserID: "owner", CreatedAt: "2026-06-25T10:00:00Z"})).To(Succeed()) + + _, err := svc.GetJob("intruder", "x") + Expect(err).To(HaveOccurred(), "a different user must not read someone else's job") + }) + }) + + Describe("store adapter conversion", func() { + // The SyncedMap value type is *schema.QuantizationJob (the exact REST shape). + // These specs prove the DB adapter round-trips it losslessly, so hydrate and + // write-through in distributed mode keep responses unchanged. + It("round-trips a job through jobToRecord/recordToJob preserving the API shape", func() { + original := &schema.QuantizationJob{ + ID: "rt-1", + UserID: "user-1", + Model: "base-model", + Backend: "llama-cpp-quantization", + ModelID: "llama-cpp-quantization-quantize-rt-1", + QuantizationType: "q4_k_m", + Status: "completed", + Message: "done", + OutputDir: "/data/quantization/rt-1", + OutputFile: "/data/quantization/rt-1/model.gguf", + ExtraOptions: map[string]string{"hf_token": "secret"}, + CreatedAt: "2026-06-27T10:00:00Z", + ImportStatus: "completed", + ImportMessage: "", + ImportModelName: "base-model-q4_k_m-rt-1", + Config: &schema.QuantizationJobRequest{Model: "base-model", Backend: "llama-cpp-quantization", QuantizationType: "q4_k_m"}, + } + + rec := jobToRecord(original) + Expect(rec.ID).To(Equal("rt-1")) + Expect(rec.ConfigJSON).ToNot(BeEmpty(), "structured config must serialize into the JSON column") + Expect(rec.ExtraOptsJSON).ToNot(BeEmpty()) + + back := recordToJob(rec) + Expect(back.ID).To(Equal(original.ID)) + Expect(back.UserID).To(Equal(original.UserID)) + Expect(back.Model).To(Equal(original.Model)) + Expect(back.Backend).To(Equal(original.Backend)) + Expect(back.ModelID).To(Equal(original.ModelID)) + Expect(back.QuantizationType).To(Equal(original.QuantizationType)) + Expect(back.Status).To(Equal(original.Status)) + Expect(back.Message).To(Equal(original.Message)) + Expect(back.OutputDir).To(Equal(original.OutputDir)) + Expect(back.OutputFile).To(Equal(original.OutputFile)) + Expect(back.ImportStatus).To(Equal(original.ImportStatus)) + Expect(back.ImportModelName).To(Equal(original.ImportModelName)) + Expect(back.CreatedAt).To(Equal(original.CreatedAt)) + Expect(back.ExtraOptions).To(Equal(original.ExtraOptions)) + Expect(back.Config).ToNot(BeNil()) + Expect(back.Config.QuantizationType).To(Equal("q4_k_m")) + }) + }) + + Describe("compile-time adapter contract", func() { + It("satisfies syncstate.Store for *distributed.QuantStore", func() { + // Guards against drift between the adapter and the component interface; + // the var assertion in syncstore.go covers it at build time, this keeps + // the type referenced from a spec too. + var _ *distributed.QuantStore + Expect(&quantStoreAdapter{}).ToNot(BeNil()) + }) + }) +}) diff --git a/core/services/quantization/syncstore.go b/core/services/quantization/syncstore.go new file mode 100644 index 000000000..4201c5ca6 --- /dev/null +++ b/core/services/quantization/syncstore.go @@ -0,0 +1,114 @@ +package quantization + +import ( + "context" + "encoding/json" + "time" + + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/core/services/distributed" + "github.com/mudler/LocalAI/core/services/syncstate" +) + +// quantStoreAdapter bridges the distributed PostgreSQL QuantStore to the generic +// syncstate.Store the SyncedMap consumes. It is only wired in distributed mode; +// standalone leaves Store nil and hydrates from disk via a Loader instead. +// +// The SyncedMap value type is *schema.QuantizationJob (the exact shape the REST +// API returns) so reads need no conversion and the response JSON is provably +// unchanged. The adapter is the single place that translates between that API +// shape and the DB QuantJobRecord. +type quantStoreAdapter struct { + store *distributed.QuantStore +} + +// compile-time assertion that the adapter satisfies the component's Store. +var _ syncstate.Store[string, *schema.QuantizationJob] = (*quantStoreAdapter)(nil) + +func (a *quantStoreAdapter) List(_ context.Context) ([]*schema.QuantizationJob, error) { + records, err := a.store.ListAll() + if err != nil { + return nil, err + } + jobs := make([]*schema.QuantizationJob, 0, len(records)) + for i := range records { + jobs = append(jobs, recordToJob(&records[i])) + } + return jobs, nil +} + +func (a *quantStoreAdapter) Upsert(_ context.Context, job *schema.QuantizationJob) error { + return a.store.Upsert(jobToRecord(job)) +} + +func (a *quantStoreAdapter) Delete(_ context.Context, id string) error { + return a.store.Delete(id) +} + +// recordToJob maps a persisted DB record back to the API shape, reconstructing +// the structured Config / ExtraOptions from their JSON columns. +func recordToJob(r *distributed.QuantJobRecord) *schema.QuantizationJob { + job := &schema.QuantizationJob{ + ID: r.ID, + UserID: r.UserID, + Model: r.Model, + Backend: r.Backend, + ModelID: r.ModelID, + QuantizationType: r.QuantizationType, + Status: r.Status, + Message: r.Message, + OutputDir: r.OutputDir, + OutputFile: r.OutputFile, + ImportStatus: r.ImportStatus, + ImportMessage: r.ImportMessage, + ImportModelName: r.ImportModelName, + CreatedAt: r.CreatedAt.UTC().Format(time.RFC3339), + } + if r.ExtraOptsJSON != "" { + // Best-effort: a malformed column must not drop the whole job from the API. + _ = json.Unmarshal([]byte(r.ExtraOptsJSON), &job.ExtraOptions) + } + if r.ConfigJSON != "" { + var cfg schema.QuantizationJobRequest + if err := json.Unmarshal([]byte(r.ConfigJSON), &cfg); err == nil { + job.Config = &cfg + } + } + return job +} + +// jobToRecord maps the API shape to a DB record for write-through, serializing +// the structured Config / ExtraOptions into their JSON columns. CreatedAt is +// parsed back from the RFC3339 string the service stamps; an unparseable value is +// left zero so QuantStore.Upsert stamps "now". +func jobToRecord(job *schema.QuantizationJob) *distributed.QuantJobRecord { + rec := &distributed.QuantJobRecord{ + ID: job.ID, + UserID: job.UserID, + Model: job.Model, + Backend: job.Backend, + ModelID: job.ModelID, + QuantizationType: job.QuantizationType, + Status: job.Status, + Message: job.Message, + OutputDir: job.OutputDir, + OutputFile: job.OutputFile, + ImportStatus: job.ImportStatus, + ImportMessage: job.ImportMessage, + ImportModelName: job.ImportModelName, + } + if job.Config != nil { + if data, err := json.Marshal(job.Config); err == nil { + rec.ConfigJSON = string(data) + } + } + if job.ExtraOptions != nil { + if data, err := json.Marshal(job.ExtraOptions); err == nil { + rec.ExtraOptsJSON = string(data) + } + } + if t, err := time.Parse(time.RFC3339, job.CreatedAt); err == nil { + rec.CreatedAt = t + } + return rec +} diff --git a/core/services/syncstate/syncstate.go b/core/services/syncstate/syncstate.go new file mode 100644 index 000000000..809177d40 --- /dev/null +++ b/core/services/syncstate/syncstate.go @@ -0,0 +1,289 @@ +// Package syncstate provides SyncedMap, a reusable cross-replica in-memory map. +// +// LocalAI in distributed mode runs multiple frontend replicas behind a +// round-robin load balancer. Several features keep process-local in-memory state +// that is surfaced to the HTTP/UI API; without cross-replica sync a poll that +// lands on a replica which did not originate a change sees stale or missing data. +// SyncedMap collapses the three legs each feature otherwise hand-wires - an +// in-memory map, a NATS broadcast/apply path, and optional durable read-through - +// into one well-tested component so cross-replica consistency is a configuration +// choice rather than a bespoke re-implementation. +package syncstate + +import ( + "context" + "sync" + "time" + + "github.com/mudler/LocalAI/core/services/messaging" + "github.com/mudler/xlog" +) + +// Op values carried on the wire and passed to OnApply. +const ( + opSet = "set" + opDelete = "delete" +) + +// Store is optional durable backing for a SyncedMap. In distributed mode it is a +// single shared DB, so the apply path (a delta received from a peer) updates +// memory only and never re-writes the Store. +type Store[K comparable, V any] interface { + List(ctx context.Context) ([]V, error) + Upsert(ctx context.Context, v V) error + Delete(ctx context.Context, k K) error +} + +// Config configures a SyncedMap. +type Config[K comparable, V any] struct { + Name string // subject namespace, e.g. "finetune.jobs" + Key func(V) K // extract the key from a value + Nats messaging.MessagingClient // nil => standalone: in-memory only, no broadcast/subscribe + Store Store[K, V] // optional read-through persistence + Loader func(ctx context.Context) ([]V, error) // source when there is no Store (e.g. disk reload) + OnApply func(op string, k K, v V) // optional hook after an applied change (e.g. ShutdownModel) + Reconcile time.Duration // optional periodic re-hydrate; 0 = off +} + +// delta is the JSON wire envelope broadcast on every local mutation. Value is +// omitempty so a delete carries only op+key. +type delta[K comparable, V any] struct { + Op string `json:"op"` + Key K `json:"key"` + Value V `json:"value,omitempty"` +} + +// SyncedMap is a cross-replica in-memory map. A local write (Set/Delete) updates +// memory, the optional durable Store, then broadcasts a delta to peers. A peer's +// delta updates memory only and fires OnApply - it never re-broadcasts and never +// writes the Store. That structural split is the echo-loop guard (same pattern as +// galleryop.mergeStatus / OpCache.applyStart): receiving your own broadcast just +// re-applies an idempotent value to memory, so there is no storm and no +// double-write. +type SyncedMap[K comparable, V any] struct { + cfg Config[K, V] + + mu sync.RWMutex + data map[K]V + + sub Subscription + + // lifeCtx outlives Start's argument: a reconnect callback or reconcile tick + // can fire long after Start returns, so they must not be tied to a ctx the + // caller may cancel. Close cancels it. + lifeCtx context.Context + cancel context.CancelFunc + wg sync.WaitGroup +} + +// Subscription is the subset of messaging.Subscription the component holds onto. +type Subscription = messaging.Subscription + +// New constructs a SyncedMap. Call Start to hydrate and begin syncing. +func New[K comparable, V any](cfg Config[K, V]) *SyncedMap[K, V] { + return &SyncedMap[K, V]{cfg: cfg, data: make(map[K]V)} +} + +func (m *SyncedMap[K, V]) subject() string { + return messaging.SubjectSyncStateDelta(m.cfg.Name) +} + +// Start hydrates from the source, subscribes for peer deltas, registers a +// reconnect re-hydrate (when the client supports it), and starts the optional +// reconcile ticker. +func (m *SyncedMap[K, V]) Start(ctx context.Context) error { + if err := m.hydrate(ctx); err != nil { + return err + } + + // The cancel func is stored on the struct and invoked in Close (covered by + // tests); lifeCtx must outlive Start to drive the reconnect/reconcile + // goroutines, so it cannot be cancelled or deferred within this scope. + m.lifeCtx, m.cancel = context.WithCancel(context.Background()) // #nosec G118 -- cancel is invoked in Close() + + if m.cfg.Nats != nil { + sub, err := messaging.SubscribeJSON(m.cfg.Nats, m.subject(), m.apply) + if err != nil { + return err + } + m.sub = sub + + // nats.go transparently resubscribes on reconnect, but it cannot know we + // kept derived in-memory state that may have drifted while the link was + // down, so re-hydrate from the durable source. Detected via an optional + // interface so MessagingClient itself stays minimal; standalone/test + // clients without the method simply fall back to the reconcile ticker. + if r, ok := m.cfg.Nats.(interface{ OnReconnect(func()) }); ok { + r.OnReconnect(func() { + if err := m.hydrate(m.lifeCtx); err != nil { + xlog.Warn("syncstate: reconnect re-hydrate failed", "name", m.cfg.Name, "error", err) + } + }) + } + } + + if m.cfg.Reconcile > 0 { + m.wg.Add(1) + go m.reconcileLoop() + } + return nil +} + +// Close unsubscribes and stops the reconcile ticker. +func (m *SyncedMap[K, V]) Close() error { + if m.cancel != nil { + m.cancel() + } + m.wg.Wait() + if m.sub != nil { + return m.sub.Unsubscribe() + } + return nil +} + +// Set updates the value locally, writes through the Store, then broadcasts. +// Per the data-flow contract the Store write happens under the lock so memory and +// durable state move together; the broadcast is best-effort after unlocking. +func (m *SyncedMap[K, V]) Set(ctx context.Context, v V) error { + k := m.cfg.Key(v) + m.mu.Lock() + m.data[k] = v + if m.cfg.Store != nil { + if err := m.cfg.Store.Upsert(ctx, v); err != nil { + m.mu.Unlock() + return err + } + } + m.mu.Unlock() + m.publish(opSet, k, v) + return nil +} + +// Delete removes the key locally, deletes it from the Store, then broadcasts. +func (m *SyncedMap[K, V]) Delete(ctx context.Context, k K) error { + m.mu.Lock() + delete(m.data, k) + if m.cfg.Store != nil { + if err := m.cfg.Store.Delete(ctx, k); err != nil { + m.mu.Unlock() + return err + } + } + m.mu.Unlock() + var zero V + m.publish(opDelete, k, zero) + return nil +} + +// Get returns the value for k and whether it was present. +func (m *SyncedMap[K, V]) Get(k K) (V, bool) { + m.mu.RLock() + defer m.mu.RUnlock() + v, ok := m.data[k] + return v, ok +} + +// List returns a snapshot slice of all values. +func (m *SyncedMap[K, V]) List() []V { + m.mu.RLock() + defer m.mu.RUnlock() + out := make([]V, 0, len(m.data)) + for _, v := range m.data { + out = append(out, v) + } + return out +} + +// Snapshot returns a copy of the underlying map. +func (m *SyncedMap[K, V]) Snapshot() map[K]V { + m.mu.RLock() + defer m.mu.RUnlock() + out := make(map[K]V, len(m.data)) + for k, v := range m.data { + out[k] = v + } + return out +} + +// publish broadcasts a delta. Standalone (nil Nats) is a strict no-op. +func (m *SyncedMap[K, V]) publish(op string, k K, v V) { + if m.cfg.Nats == nil { + return + } + if err := m.cfg.Nats.Publish(m.subject(), delta[K, V]{Op: op, Key: k, Value: v}); err != nil { + xlog.Warn("syncstate: failed to broadcast delta", "name", m.cfg.Name, "op", op, "error", err) + } +} + +// apply handles a peer's delta: memory-only update plus OnApply. It deliberately +// never writes the Store nor re-publishes - that is the echo-loop guard. +func (m *SyncedMap[K, V]) apply(d delta[K, V]) { + switch d.Op { + case opSet: + m.mu.Lock() + m.data[d.Key] = d.Value + m.mu.Unlock() + case opDelete: + m.mu.Lock() + delete(m.data, d.Key) + m.mu.Unlock() + default: + xlog.Warn("syncstate: ignoring delta with unknown op", "name", m.cfg.Name, "op", d.Op) + return + } + if m.cfg.OnApply != nil { + m.cfg.OnApply(d.Op, d.Key, d.Value) + } +} + +// hydrate replaces the whole map from the durable source: Store if present, else +// Loader. With neither, a late joiner starts empty and catches up via deltas +// (acceptable only for ephemeral state). +func (m *SyncedMap[K, V]) hydrate(ctx context.Context) error { + var ( + vals []V + err error + ) + switch { + case m.cfg.Store != nil: + vals, err = m.cfg.Store.List(ctx) + case m.cfg.Loader != nil: + vals, err = m.cfg.Loader(ctx) + default: + return nil + } + if err != nil { + return err + } + m.replaceAll(vals) + return nil +} + +// replaceAll atomically swaps the map contents for the given values, keyed via +// cfg.Key. +func (m *SyncedMap[K, V]) replaceAll(vals []V) { + next := make(map[K]V, len(vals)) + for _, v := range vals { + next[m.cfg.Key(v)] = v + } + m.mu.Lock() + m.data = next + m.mu.Unlock() +} + +// reconcileLoop periodically re-hydrates to repair silent drift (missed deltas). +func (m *SyncedMap[K, V]) reconcileLoop() { + defer m.wg.Done() + t := time.NewTicker(m.cfg.Reconcile) + defer t.Stop() + for { + select { + case <-m.lifeCtx.Done(): + return + case <-t.C: + if err := m.hydrate(m.lifeCtx); err != nil { + xlog.Warn("syncstate: reconcile re-hydrate failed", "name", m.cfg.Name, "error", err) + } + } + } +} diff --git a/core/services/syncstate/syncstate_suite_test.go b/core/services/syncstate/syncstate_suite_test.go new file mode 100644 index 000000000..b4f025c9e --- /dev/null +++ b/core/services/syncstate/syncstate_suite_test.go @@ -0,0 +1,13 @@ +package syncstate_test + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestSyncstate(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Syncstate Suite") +} diff --git a/core/services/syncstate/syncstate_test.go b/core/services/syncstate/syncstate_test.go new file mode 100644 index 000000000..1e31db41b --- /dev/null +++ b/core/services/syncstate/syncstate_test.go @@ -0,0 +1,291 @@ +package syncstate_test + +import ( + "context" + "sync" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/services/messaging" + "github.com/mudler/LocalAI/core/services/syncstate" + "github.com/mudler/LocalAI/core/services/testutil" +) + +// job is a minimal JSON-serializable value stand-in for the real cross-replica +// records (finetune/quant/agent jobs) the component is built for. +type job struct { + ID string `json:"id"` + Status string `json:"status"` +} + +func jobKey(j *job) string { return j.ID } + +const stateName = "test.jobs" + +func deltaSubject() string { return messaging.SubjectSyncStateDelta(stateName) } + +// fakeStore is an in-memory Store that records call counts so specs can assert +// the write-through-vs-apply split (local writes hit the Store; applied deltas +// must not). +type fakeStore struct { + mu sync.Mutex + data map[string]*job + upsertCalls int + deleteCalls int + listCalls int +} + +func newFakeStore(seed ...*job) *fakeStore { + s := &fakeStore{data: map[string]*job{}} + for _, j := range seed { + s.data[j.ID] = j + } + return s +} + +func (s *fakeStore) List(_ context.Context) ([]*job, error) { + s.mu.Lock() + defer s.mu.Unlock() + s.listCalls++ + out := make([]*job, 0, len(s.data)) + for _, j := range s.data { + out = append(out, j) + } + return out, nil +} + +func (s *fakeStore) Upsert(_ context.Context, j *job) error { + s.mu.Lock() + defer s.mu.Unlock() + s.upsertCalls++ + s.data[j.ID] = j + return nil +} + +func (s *fakeStore) Delete(_ context.Context, k string) error { + s.mu.Lock() + defer s.mu.Unlock() + s.deleteCalls++ + delete(s.data, k) + return nil +} + +// add simulates a peer replica writing to the shared DB out-of-band (e.g. while +// this replica was partitioned), so a re-hydrate can be observed to pick it up. +func (s *fakeStore) add(j *job) { + s.mu.Lock() + defer s.mu.Unlock() + s.data[j.ID] = j +} + +func (s *fakeStore) counts() (upsert, del, list int) { + s.mu.Lock() + defer s.mu.Unlock() + return s.upsertCalls, s.deleteCalls, s.listCalls +} + +var _ = Describe("SyncedMap", func() { + ctx := context.Background() + + Describe("cross-replica delta propagation", func() { + var ( + bus *testutil.FakeBus + a, b *syncstate.SyncedMap[string, *job] + ) + + BeforeEach(func() { + bus = testutil.NewFakeBus() + a = syncstate.New(syncstate.Config[string, *job]{Name: stateName, Key: jobKey, Nats: bus}) + b = syncstate.New(syncstate.Config[string, *job]{Name: stateName, Key: jobKey, Nats: bus}) + Expect(a.Start(ctx)).To(Succeed()) + Expect(b.Start(ctx)).To(Succeed()) + }) + + AfterEach(func() { + Expect(a.Close()).To(Succeed()) + Expect(b.Close()).To(Succeed()) + }) + + It("propagates a Set on A to B", func() { + Expect(a.Set(ctx, &job{ID: "1", Status: "running"})).To(Succeed()) + + got, ok := b.Get("1") + Expect(ok).To(BeTrue(), "replica B should see the value A just set") + Expect(got.Status).To(Equal("running")) + }) + + It("prunes a Delete on A from B", func() { + Expect(a.Set(ctx, &job{ID: "1", Status: "running"})).To(Succeed()) + _, present := b.Get("1") + Expect(present).To(BeTrue(), "precondition: B must have the value before the delete") + + Expect(a.Delete(ctx, "1")).To(Succeed()) + + _, ok := b.Get("1") + Expect(ok).To(BeFalse(), "a delete on A must remove the key from B") + }) + }) + + Describe("hydration", func() { + It("hydrates on Start from a preloaded Store", func() { + store := newFakeStore(&job{ID: "x", Status: "done"}) + m := syncstate.New(syncstate.Config[string, *job]{Name: stateName, Key: jobKey, Store: store}) + Expect(m.Start(ctx)).To(Succeed()) + + got, ok := m.Get("x") + Expect(ok).To(BeTrue(), "Start must populate the map from the Store") + Expect(got.Status).To(Equal("done")) + }) + + It("uses the Loader when Store is nil", func() { + m := syncstate.New(syncstate.Config[string, *job]{ + Name: stateName, + Key: jobKey, + Loader: func(_ context.Context) ([]*job, error) { + return []*job{{ID: "l", Status: "loaded"}}, nil + }, + }) + Expect(m.Start(ctx)).To(Succeed()) + + got, ok := m.Get("l") + Expect(ok).To(BeTrue(), "Loader output must hydrate the map when there is no Store") + Expect(got.Status).To(Equal("loaded")) + }) + }) + + Describe("echo-loop guard", func() { + It("applies its own broadcast once and does not re-publish", func() { + bus := testutil.NewFakeBus() + a := syncstate.New(syncstate.Config[string, *job]{Name: stateName, Key: jobKey, Nats: bus}) + b := syncstate.New(syncstate.Config[string, *job]{Name: stateName, Key: jobKey, Nats: bus}) + Expect(a.Start(ctx)).To(Succeed()) + Expect(b.Start(ctx)).To(Succeed()) + defer func() { + Expect(a.Close()).To(Succeed()) + Expect(b.Close()).To(Succeed()) + }() + + Expect(a.Set(ctx, &job{ID: "e", Status: "running"})).To(Succeed()) + + // One local write must produce exactly one broadcast: A and B both + // receive it and apply to memory, but the apply path never re-publishes. + Expect(bus.PublishCount(deltaSubject())).To(Equal(1), + "the apply path must not re-broadcast, otherwise replicas storm") + Expect(a.List()).To(HaveLen(1), "A must not double-store its own echo") + _, ok := b.Get("e") + Expect(ok).To(BeTrue()) + }) + }) + + Describe("Store write-through vs apply", func() { + It("writes the Store on local Set/Delete but not on an applied delta", func() { + bus := testutil.NewFakeBus() + storeA := newFakeStore() + storeB := newFakeStore() + a := syncstate.New(syncstate.Config[string, *job]{Name: stateName, Key: jobKey, Nats: bus, Store: storeA}) + b := syncstate.New(syncstate.Config[string, *job]{Name: stateName, Key: jobKey, Nats: bus, Store: storeB}) + Expect(a.Start(ctx)).To(Succeed()) + Expect(b.Start(ctx)).To(Succeed()) + defer func() { + Expect(a.Close()).To(Succeed()) + Expect(b.Close()).To(Succeed()) + }() + + Expect(a.Set(ctx, &job{ID: "w", Status: "running"})).To(Succeed()) + + upA, _, _ := storeA.counts() + upB, _, _ := storeB.counts() + Expect(upA).To(Equal(1), "local Set must write through to its own Store") + Expect(upB).To(Equal(0), "the apply path must never write the peer's Store") + + Expect(a.Delete(ctx, "w")).To(Succeed()) + _, delA, _ := storeA.counts() + _, delB, _ := storeB.counts() + Expect(delA).To(Equal(1), "local Delete must delete from its own Store") + Expect(delB).To(Equal(0), "the apply path must never delete from the peer's Store") + }) + }) + + Describe("OnApply hook", func() { + It("fires with the correct op and key on an applied delta", func() { + bus := testutil.NewFakeBus() + var ( + mu sync.Mutex + ops []string + keys []string + ) + a := syncstate.New(syncstate.Config[string, *job]{Name: stateName, Key: jobKey, Nats: bus}) + b := syncstate.New(syncstate.Config[string, *job]{ + Name: stateName, Key: jobKey, Nats: bus, + OnApply: func(op string, k string, _ *job) { + mu.Lock() + ops = append(ops, op) + keys = append(keys, k) + mu.Unlock() + }, + }) + Expect(a.Start(ctx)).To(Succeed()) + Expect(b.Start(ctx)).To(Succeed()) + defer func() { + Expect(a.Close()).To(Succeed()) + Expect(b.Close()).To(Succeed()) + }() + + Expect(a.Set(ctx, &job{ID: "o", Status: "running"})).To(Succeed()) + Expect(a.Delete(ctx, "o")).To(Succeed()) + + mu.Lock() + defer mu.Unlock() + Expect(ops).To(Equal([]string{"set", "delete"})) + Expect(keys).To(Equal([]string{"o", "o"})) + }) + }) + + Describe("standalone (nil Nats)", func() { + It("works in-memory with no panic and nothing to broadcast", func() { + m := syncstate.New(syncstate.Config[string, *job]{Name: stateName, Key: jobKey}) + Expect(m.Start(ctx)).To(Succeed()) + defer func() { Expect(m.Close()).To(Succeed()) }() + + Expect(func() { + Expect(m.Set(ctx, &job{ID: "s", Status: "running"})).To(Succeed()) + }).ToNot(Panic()) + + got, ok := m.Get("s") + Expect(ok).To(BeTrue()) + Expect(got.Status).To(Equal("running")) + Expect(m.List()).To(HaveLen(1)) + Expect(m.Snapshot()).To(HaveKey("s")) + + Expect(m.Delete(ctx, "s")).To(Succeed()) + _, ok = m.Get("s") + Expect(ok).To(BeFalse()) + }) + }) + + Describe("reconnect re-hydrate", func() { + It("re-reads the source when the messaging client reconnects", func() { + bus := testutil.NewFakeBus() + store := newFakeStore(&job{ID: "init", Status: "running"}) + m := syncstate.New(syncstate.Config[string, *job]{Name: stateName, Key: jobKey, Nats: bus, Store: store}) + Expect(m.Start(ctx)).To(Succeed()) + defer func() { Expect(m.Close()).To(Succeed()) }() + + _, ok := m.Get("init") + Expect(ok).To(BeTrue()) + + // A peer writes to the shared DB while we are unaware (no delta seen). + store.add(&job{ID: "late", Status: "running"}) + _, ok = m.Get("late") + Expect(ok).To(BeFalse(), "the new row should not appear before a re-hydrate") + + bus.TriggerReconnect() + + _, ok = m.Get("late") + Expect(ok).To(BeTrue(), "reconnect must re-hydrate from the source and pick up drift") + _, _, list := store.counts() + Expect(list).To(Equal(2), "exactly one Start hydrate plus one reconnect re-hydrate") + }) + }) +}) diff --git a/core/services/testutil/fakebus.go b/core/services/testutil/fakebus.go new file mode 100644 index 000000000..7452d810f --- /dev/null +++ b/core/services/testutil/fakebus.go @@ -0,0 +1,160 @@ +package testutil + +import ( + "encoding/json" + "strings" + "sync" + "time" + + "github.com/mudler/LocalAI/core/services/messaging" +) + +// FakeBus is an in-memory messaging.MessagingClient that delivers each published +// message synchronously to every registered subscriber whose subject filter +// matches, including NATS-style wildcard subjects (`*` matches exactly one +// token). +// +// Synchronous delivery keeps specs deterministic: the moment Publish returns, +// every matching subscriber's handler has already run, so the spec body can read +// the resulting state without polling. It is the shared test double for every +// cross-replica-sync adopter (gallery, syncstate, ...) so they exercise the same +// delivery semantics. It deliberately depends only on the standard library and +// the messaging package — no test framework — so it is importable anywhere. +type FakeBus struct { + mu sync.Mutex + subs []fakeBusSub + // publishCounts records how many messages were published per subject, so a + // spec can assert the echo-loop guard (an applied delta must not re-publish). + publishCounts map[string]int + + // reconnectCbs back the optional OnReconnect/TriggerReconnect pair, letting a + // spec exercise the component's reconnect re-hydrate path without a real + // NATS server. + reconnectCbs []func() +} + +type fakeBusSub struct { + subject string + handler func([]byte) +} + +// NewFakeBus returns a ready-to-use in-memory bus. +func NewFakeBus() *FakeBus { + return &FakeBus{publishCounts: map[string]int{}} +} + +// subjectMatches reports whether a subscription filter matches a concrete +// subject, honoring the single-token `*` wildcard used by NATS. +func subjectMatches(filter, subject string) bool { + if filter == subject { + return true + } + fp := strings.Split(filter, ".") + sp := strings.Split(subject, ".") + if len(fp) != len(sp) { + return false + } + for i := range fp { + if fp[i] == "*" { + continue + } + if fp[i] != sp[i] { + return false + } + } + return true +} + +// Publish marshals data as JSON and delivers it synchronously to every matching +// subscriber. +func (b *FakeBus) Publish(subject string, data any) error { + payload, err := json.Marshal(data) + if err != nil { + return err + } + b.mu.Lock() + b.publishCounts[subject]++ + subs := append([]fakeBusSub(nil), b.subs...) + b.mu.Unlock() + for _, s := range subs { + if subjectMatches(s.subject, subject) { + s.handler(payload) + } + } + return nil +} + +// PublishCount returns how many messages were published on the exact subject. +func (b *FakeBus) PublishCount(subject string) int { + b.mu.Lock() + defer b.mu.Unlock() + return b.publishCounts[subject] +} + +type fakeBusSubscription struct { + bus *FakeBus + subRef fakeBusSub +} + +func (s *fakeBusSubscription) Unsubscribe() error { + s.bus.mu.Lock() + defer s.bus.mu.Unlock() + for i, candidate := range s.bus.subs { + if candidate.subject == s.subRef.subject { + s.bus.subs = append(s.bus.subs[:i], s.bus.subs[i+1:]...) + return nil + } + } + return nil +} + +func (b *FakeBus) Subscribe(subject string, handler func([]byte)) (messaging.Subscription, error) { + sub := fakeBusSub{subject: subject, handler: handler} + b.mu.Lock() + b.subs = append(b.subs, sub) + b.mu.Unlock() + return &fakeBusSubscription{bus: b, subRef: sub}, nil +} + +func (b *FakeBus) QueueSubscribe(subject, _ string, handler func([]byte)) (messaging.Subscription, error) { + return b.Subscribe(subject, handler) +} + +func (b *FakeBus) QueueSubscribeReply(string, string, func([]byte, func([]byte))) (messaging.Subscription, error) { + return &fakeBusSubscription{bus: b}, nil +} + +func (b *FakeBus) SubscribeReply(string, func([]byte, func([]byte))) (messaging.Subscription, error) { + return &fakeBusSubscription{bus: b}, nil +} + +func (b *FakeBus) Request(string, []byte, time.Duration) ([]byte, error) { + return nil, nil +} + +func (b *FakeBus) IsConnected() bool { return true } +func (b *FakeBus) Close() {} + +// OnReconnect mirrors *messaging.Client.OnReconnect so a spec can drive the +// component's reconnect re-hydrate path. The component detects this method via an +// optional interface assertion; implementing it here keeps the fake a faithful +// stand-in for the concrete client. +func (b *FakeBus) OnReconnect(cb func()) { + if cb == nil { + return + } + b.mu.Lock() + b.reconnectCbs = append(b.reconnectCbs, cb) + b.mu.Unlock() +} + +// TriggerReconnect runs every registered reconnect callback, simulating a NATS +// reconnect event. +func (b *FakeBus) TriggerReconnect() { + b.mu.Lock() + cbs := append([]func(){}, b.reconnectCbs...) + b.mu.Unlock() + for _, cb := range cbs { + cb() + } +} diff --git a/tests/e2e/distributed/syncstate_distributed_test.go b/tests/e2e/distributed/syncstate_distributed_test.go new file mode 100644 index 000000000..acd2797e6 --- /dev/null +++ b/tests/e2e/distributed/syncstate_distributed_test.go @@ -0,0 +1,161 @@ +package distributed_test + +import ( + "context" + + "github.com/mudler/LocalAI/core/services/distributed" + "github.com/mudler/LocalAI/core/services/messaging" + "github.com/mudler/LocalAI/core/services/syncstate" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + pgdriver "gorm.io/driver/postgres" + "gorm.io/gorm" + "gorm.io/gorm/logger" +) + +// ftSyncStore adapts the real FineTuneStore to syncstate.Store, exactly as the +// finetune service does in production. Defined here (rather than reusing the +// service's unexported adapter) so the e2e exercises the store + component over +// real infrastructure without pulling in backend execution. +type ftSyncStore struct{ s *distributed.FineTuneStore } + +func (a ftSyncStore) List(_ context.Context) ([]*distributed.FineTuneJobRecord, error) { + recs, err := a.s.ListAll() + if err != nil { + return nil, err + } + out := make([]*distributed.FineTuneJobRecord, len(recs)) + for i := range recs { + r := recs[i] + out[i] = &r + } + return out, nil +} + +func (a ftSyncStore) Upsert(_ context.Context, r *distributed.FineTuneJobRecord) error { + return a.s.Upsert(r) +} + +func (a ftSyncStore) Delete(_ context.Context, k string) error { return a.s.Delete(k) } + +// This suite is the real-infrastructure counterpart to the fake-bus unit tests: +// two SyncedMap instances stand in for two LocalAI frontend replicas, each with +// its OWN NATS connection to a shared NATS server and a SHARED PostgreSQL store - +// the exact distributed-mode invariant (single shared DB, per-replica process +// state). It proves the delta path works over the wire and that a late-joining +// replica recovers via store hydrate (the at-most-once gap a fake bus cannot +// exercise). +var _ = Describe("SyncedMap two-replica sync over real NATS", Label("Distributed"), func() { + var ( + infra *TestInfra + ftStore *distributed.FineTuneStore + ) + + BeforeEach(func() { + infra = SetupInfra("localai_syncstate_dist_test") + + db, err := gorm.Open(pgdriver.Open(infra.PGURL), &gorm.Config{ + Logger: logger.Default.LogMode(logger.Silent), + }) + Expect(err).ToNot(HaveOccurred()) + + ftStore, err = distributed.NewFineTuneStore(db) + Expect(err).ToNot(HaveOccurred()) + }) + + // newReplica builds an independent "replica": its own NATS client to the + // shared server plus a SyncedMap over the shared store, started (hydrate + + // subscribe) and cleaned up automatically. + newReplica := func() *syncstate.SyncedMap[string, *distributed.FineTuneJobRecord] { + GinkgoHelper() + nc, err := messaging.New(infra.NatsURL) + Expect(err).ToNot(HaveOccurred()) + + sm := syncstate.New(syncstate.Config[string, *distributed.FineTuneJobRecord]{ + Name: "finetune.jobs", + Key: func(r *distributed.FineTuneJobRecord) string { return r.ID }, + Nats: nc, + Store: ftSyncStore{s: ftStore}, + }) + Expect(sm.Start(infra.Ctx)).To(Succeed()) + FlushNATS(nc) // ensure the subscription is registered server-side before any publish + DeferCleanup(func() { + _ = sm.Close() + nc.Close() + }) + return sm + } + + rec := func(id, status string) *distributed.FineTuneJobRecord { + return &distributed.FineTuneJobRecord{ + ID: id, UserID: "u1", Model: "m", Backend: "b", + TrainingType: "lora", TrainingMethod: "sft", Status: status, + } + } + + It("propagates a create from replica A to replica B over the wire", func() { + a := newReplica() + b := newReplica() + + Expect(a.Set(infra.Ctx, rec("job-1", "queued"))).To(Succeed()) + + Eventually(func() bool { _, ok := b.Get("job-1"); return ok }, "10s", "50ms"). + Should(BeTrue(), "replica B must observe the job created on A via NATS") + + got, ok := b.Get("job-1") + Expect(ok).To(BeTrue()) + Expect(got.Status).To(Equal("queued")) + }) + + It("propagates an update and a delete across replicas", func() { + a := newReplica() + b := newReplica() + + Expect(a.Set(infra.Ctx, rec("job-2", "queued"))).To(Succeed()) + Eventually(func() bool { _, ok := b.Get("job-2"); return ok }, "10s", "50ms").Should(BeTrue()) + + // Update on A -> B reflects the new status. + Expect(a.Set(infra.Ctx, rec("job-2", "training"))).To(Succeed()) + Eventually(func() string { + if r, ok := b.Get("job-2"); ok { + return r.Status + } + return "" + }, "10s", "50ms").Should(Equal("training")) + + // Delete on A -> B prunes (a reload-from-path could not do this). + Expect(a.Delete(infra.Ctx, "job-2")).To(Succeed()) + Eventually(func() bool { _, ok := b.Get("job-2"); return ok }, "10s", "50ms"). + Should(BeFalse(), "replica B must drop the job deleted on A") + }) + + It("hydrates a late-joining replica from the shared store (missed-delta recovery)", func() { + a := newReplica() + + // Written (and broadcast) BEFORE replica C exists, so C can never receive + // the delta - it can only learn the job by hydrating from shared Postgres + // on Start. This is the at-most-once gap a fake bus cannot exercise. + Expect(a.Set(infra.Ctx, rec("job-3", "completed"))).To(Succeed()) + Eventually(func() (*distributed.FineTuneJobRecord, error) { return ftStore.Get("job-3") }, "10s", "50ms"). + ShouldNot(BeNil(), "write-through must reach the shared store first") + + c := newReplica() // joins late; Start() hydrates from the store synchronously + + got, ok := c.Get("job-3") + Expect(ok).To(BeTrue(), "late replica must recover the job via store hydrate, not a delta") + Expect(got.Status).To(Equal("completed")) + }) + + It("write-through persists a local Set to the shared PostgreSQL store", func() { + a := newReplica() + + Expect(a.Set(infra.Ctx, rec("job-4", "queued"))).To(Succeed()) + + persisted, err := ftStore.Get("job-4") + Expect(err).ToNot(HaveOccurred()) + Expect(persisted.ID).To(Equal("job-4")) + Expect(persisted.Status).To(Equal("queued")) + }) +}) From 8aba4fdba3f6335e8571b42bf2988f66af492b0a Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sat, 27 Jun 2026 23:24:21 +0200 Subject: [PATCH 05/17] chore(fish-speech): drop the darwin/metal build target (#10561) The fish-speech metal-darwin-arm64 backend build has been failing on every release (v4.5.3, v4.5.4, v4.5.5) and is a standing red on the darwin backend matrix. fish-speech pulls `tokenizers` transitively from its upstream source (`pip install -e fish-speech-src`), and on darwin/arm64 there is no prebuilt wheel for the pinned old `tokenizers` version, so pip builds it from source. Modern rustc rejects that old crate as a hard error: error: casting `&T` to `&mut T` is undefined behavior ... --> tokenizers-lib/src/models/bpe/trainer.rs:517:47 = note: `#[deny(invalid_reference_casting)]` on by default error: could not compile `tokenizers` (lib) due to 1 previous error This is deterministic, not a flake, and there is no clean fix that does not either pin a stale Rust toolchain or downgrade a soundness lint guarding real UB. Until upstream fish-speech moves to a tokenizers version that compiles on current toolchains, drop darwin support so the release backend build stays green. The Linux/CUDA/ROCm/Intel/L4T variants are unaffected. Removes: - the `-metal-darwin-arm64-fish-speech` entry from `includeDarwin` in backend-matrix.yml - the `metal:` capability mappings and the concrete `metal-fish-speech` / `metal-fish-speech-development` gallery entries in backend/index.yaml - the now-unused darwin-only requirements-mps.txt Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- .github/backend-matrix.yml | 3 --- backend/index.yaml | 12 ------------ backend/python/fish-speech/requirements-mps.txt | 2 -- 3 files changed, 17 deletions(-) delete mode 100644 backend/python/fish-speech/requirements-mps.txt diff --git a/.github/backend-matrix.yml b/.github/backend-matrix.yml index 6147f1250..1d6496231 100644 --- a/.github/backend-matrix.yml +++ b/.github/backend-matrix.yml @@ -4991,9 +4991,6 @@ includeDarwin: - backend: "qwen-tts" tag-suffix: "-metal-darwin-arm64-qwen-tts" build-type: "mps" - - backend: "fish-speech" - tag-suffix: "-metal-darwin-arm64-fish-speech" - build-type: "mps" - backend: "voxcpm" tag-suffix: "-metal-darwin-arm64-voxcpm" build-type: "mps" diff --git a/backend/index.yaml b/backend/index.yaml index 2841d0f79..8c0f1bc8d 100644 --- a/backend/index.yaml +++ b/backend/index.yaml @@ -1356,7 +1356,6 @@ intel: "intel-fish-speech" amd: "rocm-fish-speech" nvidia-l4t: "nvidia-l4t-fish-speech" - metal: "metal-fish-speech" default: "cpu-fish-speech" nvidia-cuda-13: "cuda13-fish-speech" nvidia-cuda-12: "cuda12-fish-speech" @@ -4870,7 +4869,6 @@ intel: "intel-fish-speech-development" amd: "rocm-fish-speech-development" nvidia-l4t: "nvidia-l4t-fish-speech-development" - metal: "metal-fish-speech-development" default: "cpu-fish-speech-development" nvidia-cuda-13: "cuda13-fish-speech-development" nvidia-cuda-12: "cuda12-fish-speech-development" @@ -4946,16 +4944,6 @@ uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-fish-speech" mirrors: - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-fish-speech -- !!merge <<: *fish-speech - name: "metal-fish-speech" - uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-fish-speech" - mirrors: - - localai/localai-backends:latest-metal-darwin-arm64-fish-speech -- !!merge <<: *fish-speech - name: "metal-fish-speech-development" - uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-fish-speech" - mirrors: - - localai/localai-backends:master-metal-darwin-arm64-fish-speech ## faster-qwen3-tts - !!merge <<: *faster-qwen3-tts name: "faster-qwen3-tts-development" diff --git a/backend/python/fish-speech/requirements-mps.txt b/backend/python/fish-speech/requirements-mps.txt deleted file mode 100644 index ff5c00f19..000000000 --- a/backend/python/fish-speech/requirements-mps.txt +++ /dev/null @@ -1,2 +0,0 @@ -torch -torchaudio From 1154be5eea3d624be6e64d3f491885fabe8b6845 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sat, 27 Jun 2026 23:34:52 +0200 Subject: [PATCH 06/17] fix(config): fall back to DefaultContextSize for unparseable GGUFs; pin NVFP4 gallery context_size (#10563) The GGUF metadata parser (gpustack/gguf-parser-go) cannot read NVFP4-quantized GGUFs at all: it errors with "read tensor info 0: This quantized type is currently unsupported" because NVFP4 is a ggml tensor type it does not know. When ParseGGUFFile errors, the llama-cpp defaults hook skips guessGGUFFromFile entirely and the deferred fallback sets the context window to the conservative GGUFFallbackContextSize (1024). The result: a model that trains to 262144 tokens runs with n_ctx=1024, and every prompt over ~1k tokens fails with "request (N tokens) exceeds the available context size (1024 tokens)". Two changes: - Drop GGUFFallbackContextSize (1024) and fall back to DefaultContextSize (4096) in both the GGUF run-estimate path (gguf.go) and the deferred hook fallback (hooks_llamacpp.go). 1024 is a sensible floor for a tiny CPU GGUF but a footgun for a large, long-context model whose header simply cannot be parsed. Strengthen the existing "GGUF unreadable" test to assert the value. - Set context_size explicitly on the four NVFP4 gallery entries (qwen3.6-35b-a3b-nvfp4-mtp, qwopus3.6-27b-v2-mtp-nvfp4, qwopus3.6-27b-coder-mtp-nvfp4, qwen3.6-27b-nvfp4-mtp) so the parser failure is irrelevant for them. 32768 matches sibling Qwen entries and is safe on memory; operators can raise it toward the 262144 train length. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- core/config/defaults.go | 10 ++++------ core/config/gguf.go | 2 +- core/config/hooks_llamacpp.go | 2 +- core/config/hooks_test.go | 4 ++++ gallery/index.yaml | 14 ++++++++++++++ 5 files changed, 24 insertions(+), 8 deletions(-) diff --git a/core/config/defaults.go b/core/config/defaults.go index 18625fab3..bb993d075 100644 --- a/core/config/defaults.go +++ b/core/config/defaults.go @@ -12,14 +12,12 @@ package config // these; config never imports backend. const ( // DefaultContextSize is the fallback context window when none is configured - // or estimable from the model. + // or estimable from the model. It is also the fallback for a GGUF whose + // metadata yields no usable estimate or that the parser cannot read at all + // (e.g. a quant type it does not know, such as NVFP4): a model-agnostic + // safe default beats a tiny, surprising window that truncates real prompts. DefaultContextSize = 4096 - // GGUFFallbackContextSize is the context window for a GGUF model whose - // metadata yields no usable estimate (see guessGGUFFromFile). Deliberately - // smaller than DefaultContextSize to stay conservative on memory there. - GGUFFallbackContextSize = 1024 - // DefaultNGPULayers means "offload all layers"; the backend (fit_params) // clamps to what actually fits in device memory. DefaultNGPULayers = 99999999 diff --git a/core/config/gguf.go b/core/config/gguf.go index 16e43c914..177e68749 100644 --- a/core/config/gguf.go +++ b/core/config/gguf.go @@ -33,7 +33,7 @@ func guessGGUFFromFile(cfg *ModelConfig, f *gguf.GGUFFile, defaultCtx int) { cSize := int(ctxSize) cfg.ContextSize = &cSize } else { - defaultCtx = GGUFFallbackContextSize + defaultCtx = DefaultContextSize cfg.ContextSize = &defaultCtx } } diff --git a/core/config/hooks_llamacpp.go b/core/config/hooks_llamacpp.go index 09bdbe868..07ccdda7b 100644 --- a/core/config/hooks_llamacpp.go +++ b/core/config/hooks_llamacpp.go @@ -34,7 +34,7 @@ func llamaCppDefaults(cfg *ModelConfig, modelPath string) { // Default context size if not set, regardless of whether GGUF parsing succeeds defer func() { if cfg.ContextSize == nil { - ctx := GGUFFallbackContextSize + ctx := DefaultContextSize cfg.ContextSize = &ctx } }() diff --git a/core/config/hooks_test.go b/core/config/hooks_test.go index 6e18ad7cc..a1b30b8d9 100644 --- a/core/config/hooks_test.go +++ b/core/config/hooks_test.go @@ -248,7 +248,11 @@ var _ = Describe("Backend hooks and parser defaults", func() { } cfg.SetDefaults(ModelPath(dir)) + // An unreadable/unparseable GGUF (e.g. a quant type the parser does + // not know, such as NVFP4) yields no estimate, so the hook must fall + // back to DefaultContextSize rather than a tiny, surprising value. Expect(cfg.ContextSize).NotTo(BeNil()) + Expect(*cfg.ContextSize).To(Equal(DefaultContextSize)) }) }) diff --git a/gallery/index.yaml b/gallery/index.yaml index cc975a83a..f39993333 100644 --- a/gallery/index.yaml +++ b/gallery/index.yaml @@ -579,6 +579,10 @@ icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.6/Figures/qwen3.6_35b_a3b_score.png overrides: backend: llama-cpp + # NVFP4 GGUFs use a quant type the GGUF metadata parser cannot read, so + # context size cannot be auto-derived; set it explicitly (the model trains + # to 262144, 32768 is a safe default operators can raise). + context_size: 32768 function: automatic_tool_parsing_fallback: true grammar: @@ -611,6 +615,9 @@ - gguf overrides: backend: llama-cpp + # NVFP4 GGUFs use a quant type the GGUF metadata parser cannot read, so + # context size cannot be auto-derived; set it explicitly. + context_size: 32768 function: automatic_tool_parsing_fallback: true grammar: @@ -638,6 +645,9 @@ icon: https://cdn-uploads.huggingface.co/production/uploads/66309bd090589b7c65950665/sGQKmrMc6L6guMoaB5_Y2.png overrides: backend: llama-cpp + # NVFP4 GGUFs use a quant type the GGUF metadata parser cannot read, so + # context size cannot be auto-derived; set it explicitly. + context_size: 32768 function: automatic_tool_parsing_fallback: true grammar: @@ -688,6 +698,10 @@ icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.6/Figures/qwen3.6_27b_score.png overrides: backend: llama-cpp + # NVFP4 GGUFs use a quant type the GGUF metadata parser cannot read, so + # context size cannot be auto-derived; set it explicitly (the model trains + # to 262144, 32768 is a safe default operators can raise). + context_size: 32768 function: automatic_tool_parsing_fallback: true grammar: From fdff11470178e301a57c0e3ba1d7de862d6b07bb Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 00:40:21 +0200 Subject: [PATCH 07/17] ci(vibevoice): skip the ASR transcription e2e on release tag builds (#10567) The `tests-vibevoice-cpp-grpc-transcription` job downloads the vibevoice ASR model (`vibevoice-asr-q4_k.gguf`, ~10 GB) and decodes it through the e2e-backends harness. On release tag pushes the detect step forces the full matrix (run-all=true), so this job runs and consistently times out: the inner `go test -timeout 30m` cannot pull a 10 GB file from HuggingFace's throttled Xet CDN within budget (curl --max-time 600 x5 retries overruns the deadline), leaving an orphaned curl and a 30m panic. It has been red on every release (v4.5.3/4/5). Guard the job's `if` with `!startsWith(github.ref, 'refs/tags/')` so it no longer runs on tag/release builds. It still runs on PRs and branch pushes that touch vibevoice-cpp, so real regressions are caught off the release path. A proper fix (a small ASR test GGUF) can re-enable it on tags later. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- .github/workflows/test-extra.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test-extra.yml b/.github/workflows/test-extra.yml index 650f464a2..b8d212219 100644 --- a/.github/workflows/test-extra.yml +++ b/.github/workflows/test-extra.yml @@ -1008,7 +1008,11 @@ jobs: # image + working dir. tests-vibevoice-cpp-grpc-transcription: needs: detect-changes - if: needs.detect-changes.outputs.vibevoice-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true' + # Skip on release tag pushes: the ASR Q4_K model is ~10 GB and cannot be + # pulled from HF within the inner `go test -timeout 30m` budget on a CI + # runner, so every tag build hung and timed out. Still runs on PRs/branch + # pushes that touch vibevoice-cpp so regressions are caught off the release path. + if: (needs.detect-changes.outputs.vibevoice-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true') && !startsWith(github.ref, 'refs/tags/') runs-on: bigger-runner timeout-minutes: 150 steps: From f1fcafb888ed93d0f0c5a80970021e1a20b16083 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 01:21:33 +0200 Subject: [PATCH 08/17] fix(gallery): match mmproj/model quant as a whole token so F16 no longer selects BF16 (#10559) (#10564) pickPreferredGroup matched a quant preference against the shard base filename with strings.Contains. Because `f16` is a substring of `bf16`, asking for the `F16` mmproj quant would wrongly satisfy a `BF16` file and select it when its group came first. Match the preference as a whole token instead: it must be delimited by a non-alphanumeric character (or the string start/end) on both outer edges. Separators inside the preference itself (e.g. `ud-q4_k_xl`) are left untouched, and all occurrences are scanned before rejecting. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- core/gallery/importers/llama-cpp.go | 39 +++++++++- core/gallery/importers/llama-cpp_test.go | 98 ++++++++++++++++++++++++ 2 files changed, 134 insertions(+), 3 deletions(-) diff --git a/core/gallery/importers/llama-cpp.go b/core/gallery/importers/llama-cpp.go index 39a732560..5797e6352 100644 --- a/core/gallery/importers/llama-cpp.go +++ b/core/gallery/importers/llama-cpp.go @@ -25,8 +25,8 @@ var ( type LlamaCPPImporter struct{} -func (i *LlamaCPPImporter) Name() string { return "llama-cpp" } -func (i *LlamaCPPImporter) Modality() string { return "text" } +func (i *LlamaCPPImporter) Name() string { return "llama-cpp" } +func (i *LlamaCPPImporter) Modality() string { return "text" } func (i *LlamaCPPImporter) AutoDetects() bool { return true } // AdditionalBackends advertises drop-in replacements that share the @@ -293,7 +293,7 @@ func pickPreferredGroup(groups []hfapi.ShardGroup, prefs []string) *hfapi.ShardG for _, pref := range prefs { lower := strings.ToLower(pref) for i := range groups { - if strings.Contains(strings.ToLower(groups[i].Base), lower) { + if quantTokenMatches(strings.ToLower(groups[i].Base), lower) { return &groups[i] } } @@ -301,6 +301,39 @@ func pickPreferredGroup(groups []hfapi.ShardGroup, prefs []string) *hfapi.ShardG return &groups[len(groups)-1] } +// quantTokenMatches reports whether pref appears in base as a whole token +// rather than as a substring of a larger alphanumeric run. Both arguments +// must already be lowercased. +// +// A plain strings.Contains is wrong here: `f16` is a substring of `bf16`, so +// asking for the `F16` quant used to wrongly select a `BF16` file (#10559). +// Only the OUTER edges of the matched preference must hit a boundary — a +// non-alphanumeric char (or the start/end of base). Separators inside the +// preference itself (e.g. `ud-q4_k_xl`) are intentionally left untouched. +func quantTokenMatches(base, pref string) bool { + if pref == "" { + return false + } + for start := strings.Index(base, pref); start != -1; { + end := start + len(pref) + leftOK := start == 0 || !isAlphaNum(base[start-1]) + rightOK := end == len(base) || !isAlphaNum(base[end]) + if leftOK && rightOK { + return true + } + next := strings.Index(base[start+1:], pref) + if next == -1 { + break + } + start += next + 1 + } + return false +} + +func isAlphaNum(b byte) bool { + return (b >= 'a' && b <= 'z') || (b >= '0' && b <= '9') +} + // maybeApplyMTPDefaults parses the picked GGUF header (range-fetched over // HTTP for HF/URL imports) and, if the file declares a Multi-Token Prediction // head, appends the auto-MTP option keys to modelConfig.Options. Failures diff --git a/core/gallery/importers/llama-cpp_test.go b/core/gallery/importers/llama-cpp_test.go index f141fc29f..e3f730945 100644 --- a/core/gallery/importers/llama-cpp_test.go +++ b/core/gallery/importers/llama-cpp_test.go @@ -374,6 +374,104 @@ var _ = Describe("LlamaCPPImporter", func() { }) }) + Context("quant token boundary matching", func() { + // Regression for #10559: the quant preference must match as a whole + // token, not as a substring. Asking for `F16` used to select a + // `BF16` mmproj because strings.Contains("...bf16.gguf", "f16") is + // true — the leading `b` was ignored. + + const repoBase = "https://huggingface.co/acme/example-GGUF/resolve/main/" + + hfFile := func(path, sha string) hfapi.ModelFile { + return hfapi.ModelFile{ + Path: path, + SHA256: sha, + URL: repoBase + path, + } + } + + withHF := func(preferences string, files ...hfapi.ModelFile) Details { + d := Details{ + URI: "https://huggingface.co/acme/example-GGUF", + HuggingFace: &hfapi.ModelDetails{ + ModelID: "acme/example-GGUF", + Files: files, + }, + } + if preferences != "" { + d.Preferences = json.RawMessage(preferences) + } + return d + } + + It("selects the F16 mmproj over BF16 (BF16 listed first)", func() { + details := withHF(`{"name":"VL","mmproj_quantizations":"F16"}`, + hfFile("model-Q4_K_M.gguf", "model"), + hfFile("mmproj-x-BF16.gguf", "bf16"), + hfFile("mmproj-x-F16.gguf", "f16"), + ) + + modelConfig, err := importer.Import(details) + + Expect(err).ToNot(HaveOccurred()) + Expect(modelConfig.ConfigFile).To(ContainSubstring("mmproj: llama-cpp/mmproj/VL/mmproj-x-F16.gguf"), fmt.Sprintf("%+v", modelConfig)) + Expect(modelConfig.ConfigFile).ToNot(ContainSubstring("BF16"), fmt.Sprintf("%+v", modelConfig)) + }) + + It("selects the F16 mmproj over BF16 (F16 listed first)", func() { + details := withHF(`{"name":"VL","mmproj_quantizations":"F16"}`, + hfFile("model-Q4_K_M.gguf", "model"), + hfFile("mmproj-x-F16.gguf", "f16"), + hfFile("mmproj-x-BF16.gguf", "bf16"), + ) + + modelConfig, err := importer.Import(details) + + Expect(err).ToNot(HaveOccurred()) + Expect(modelConfig.ConfigFile).To(ContainSubstring("mmproj: llama-cpp/mmproj/VL/mmproj-x-F16.gguf"), fmt.Sprintf("%+v", modelConfig)) + Expect(modelConfig.ConfigFile).ToNot(ContainSubstring("BF16"), fmt.Sprintf("%+v", modelConfig)) + }) + + It("selects BF16 when BF16 is the requested mmproj quant", func() { + details := withHF(`{"name":"VL","mmproj_quantizations":"BF16"}`, + hfFile("model-Q4_K_M.gguf", "model"), + hfFile("mmproj-x-F16.gguf", "f16"), + hfFile("mmproj-x-BF16.gguf", "bf16"), + ) + + modelConfig, err := importer.Import(details) + + Expect(err).ToNot(HaveOccurred()) + Expect(modelConfig.ConfigFile).To(ContainSubstring("mmproj: llama-cpp/mmproj/VL/mmproj-x-BF16.gguf"), fmt.Sprintf("%+v", modelConfig)) + }) + + It("still matches a normal model quant with internal separators", func() { + // ud-q4_k_xl contains `-`/`_` internally; only the outer edges + // must hit a token boundary. + details := withHF(`{"name":"M","quantizations":"ud-q4_k_xl"}`, + hfFile("model-UD-Q4_K_XL.gguf", "xl"), + hfFile("model-Q3_K_M.gguf", "q3"), + ) + + modelConfig, err := importer.Import(details) + + Expect(err).ToNot(HaveOccurred()) + Expect(modelConfig.ConfigFile).To(ContainSubstring("model: llama-cpp/models/M/model-UD-Q4_K_XL.gguf"), fmt.Sprintf("%+v", modelConfig)) + }) + + It("falls back to the last group when no preference matches", func() { + details := withHF(`{"name":"M","quantizations":"Q2_K"}`, + hfFile("model-Q8_0.gguf", "q8"), + hfFile("model-Q3_K_M.gguf", "q3"), + ) + + modelConfig, err := importer.Import(details) + + Expect(err).ToNot(HaveOccurred()) + Expect(modelConfig.ConfigFile).To(ContainSubstring("model: llama-cpp/models/M/model-Q3_K_M.gguf"), fmt.Sprintf("%+v", modelConfig)) + }) + }) + Context("AdditionalBackends", func() { It("advertises ik-llama-cpp and turboquant as drop-in replacements", func() { entries := importer.AdditionalBackends() From 91885c2c7e36bf79b31bcc96713491b643d53c47 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 01:22:48 +0200 Subject: [PATCH 09/17] fix(distributed): return empty backend list for agent nodes instead of failing backend.list (#10545) (#10565) Opening an AGENT-type worker node's detail page errored with "failed to list backends on node" / NATS "nodes..backend.list: no responders available". Agent workers only subscribe to agent.*, jobs.*, mcp.* and .backend.stop; they never subscribe to backend.list, so the per-node ListBackendsOnNodeEndpoint request had no responder and timed out. The aggregate cluster-wide list already guards this in managers_distributed.go (skip nodes whose NodeType is set and not "backend"). The single-node endpoint lacked the same guard. Thread the NodeRegistry into ListBackendsOnNodeEndpoint and short-circuit to an empty (non-nil) list for non-backend node types before issuing the doomed NATS request, mirroring the aggregate-list gate so both views stay consistent. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- core/http/endpoints/localai/nodes.go | 16 ++- .../localai/nodes_backends_list_test.go | 103 ++++++++++++++++++ core/http/routes/nodes.go | 2 +- 3 files changed, 118 insertions(+), 3 deletions(-) create mode 100644 core/http/endpoints/localai/nodes_backends_list_test.go diff --git a/core/http/endpoints/localai/nodes.go b/core/http/endpoints/localai/nodes.go index e91eda6f4..71b4cbb11 100644 --- a/core/http/endpoints/localai/nodes.go +++ b/core/http/endpoints/localai/nodes.go @@ -25,6 +25,7 @@ import ( "github.com/mudler/LocalAI/core/http/auth" "github.com/mudler/LocalAI/core/schema" "github.com/mudler/LocalAI/core/services/galleryop" + "github.com/mudler/LocalAI/core/services/messaging" "github.com/mudler/LocalAI/core/services/nodes" "github.com/mudler/LocalAI/core/services/nodes/prefixcache" "github.com/mudler/LocalAI/pkg/httpclient" @@ -550,12 +551,23 @@ func DeleteBackendOnNodeEndpoint(unloader nodes.NodeCommandSender) echo.HandlerF } // ListBackendsOnNodeEndpoint lists installed backends on a worker node via NATS. -func ListBackendsOnNodeEndpoint(unloader nodes.NodeCommandSender) echo.HandlerFunc { +func ListBackendsOnNodeEndpoint(unloader nodes.NodeCommandSender, registry *nodes.NodeRegistry) echo.HandlerFunc { return func(c echo.Context) error { + nodeID := c.Param("id") + // Agent-type workers don't run backends and never subscribe to the + // nodes..backend.list NATS subject, so the request would hang + // until timeout with "no responders". Their backend list is simply + // empty. Mirror the aggregate-list guard in managers_distributed.go + // (skip nodes whose NodeType is set and not "backend") so the + // single-node and cluster-wide views stay consistent. + if node, err := registry.Get(c.Request().Context(), nodeID); err == nil { + if node.NodeType != "" && node.NodeType != nodes.NodeTypeBackend { + return c.JSON(http.StatusOK, []messaging.NodeBackendInfo{}) + } + } if unloader == nil { return c.JSON(http.StatusServiceUnavailable, nodeError(http.StatusServiceUnavailable, "NATS not configured")) } - nodeID := c.Param("id") reply, err := unloader.ListBackends(nodeID) if err != nil { xlog.Error("Failed to list backends on node", "node", nodeID, "error", err) diff --git a/core/http/endpoints/localai/nodes_backends_list_test.go b/core/http/endpoints/localai/nodes_backends_list_test.go new file mode 100644 index 000000000..c625e8e95 --- /dev/null +++ b/core/http/endpoints/localai/nodes_backends_list_test.go @@ -0,0 +1,103 @@ +package localai + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + + "github.com/labstack/echo/v4" + "github.com/mudler/LocalAI/core/services/messaging" + "github.com/mudler/LocalAI/core/services/nodes" + "github.com/mudler/LocalAI/core/services/testutil" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// stubNodeCommandSender records whether ListBackends was invoked so the test can +// assert the endpoint short-circuits (no NATS request) for agent-type nodes. +type stubNodeCommandSender struct { + listBackendsCalled bool +} + +func (s *stubNodeCommandSender) InstallBackend(_, _, _, _, _, _, _ string, _ int, _ string, _ func(messaging.BackendInstallProgressEvent)) (*messaging.BackendInstallReply, error) { + return &messaging.BackendInstallReply{}, nil +} + +func (s *stubNodeCommandSender) UpgradeBackend(_, _, _, _, _, _ string, _ int, _ string, _ func(messaging.BackendInstallProgressEvent)) (*messaging.BackendUpgradeReply, error) { + return &messaging.BackendUpgradeReply{}, nil +} + +func (s *stubNodeCommandSender) DeleteBackend(_, _ string) (*messaging.BackendDeleteReply, error) { + return &messaging.BackendDeleteReply{Success: true}, nil +} + +func (s *stubNodeCommandSender) ListBackends(_ string) (*messaging.BackendListReply, error) { + s.listBackendsCalled = true + return &messaging.BackendListReply{Backends: []messaging.NodeBackendInfo{{Name: "llama-cpp"}}}, nil +} + +func (s *stubNodeCommandSender) StopBackend(_, _ string) error { return nil } + +func (s *stubNodeCommandSender) UnloadModelOnNode(_, _ string) error { return nil } + +var _ = Describe("ListBackendsOnNodeEndpoint", func() { + var registry *nodes.NodeRegistry + + BeforeEach(func() { + db := testutil.SetupTestDB() + var err error + registry, err = nodes.NewNodeRegistry(db) + Expect(err).ToNot(HaveOccurred()) + }) + + callEndpoint := func(unloader nodes.NodeCommandSender, nodeID string) *httptest.ResponseRecorder { + e := echo.New() + req := httptest.NewRequest(http.MethodGet, "/", nil) + rec := httptest.NewRecorder() + c := e.NewContext(req, rec) + c.SetParamNames("id") + c.SetParamValues(nodeID) + handler := ListBackendsOnNodeEndpoint(unloader, registry) + Expect(handler(c)).To(Succeed()) + return rec + } + + It("returns an empty list for an agent node without issuing a NATS request", func() { + ctx := context.Background() + node := &nodes.BackendNode{Name: "agent-1", NodeType: nodes.NodeTypeAgent} + Expect(registry.Register(ctx, node, true)).To(Succeed()) + + stub := &stubNodeCommandSender{} + rec := callEndpoint(stub, node.ID) + + Expect(rec.Code).To(Equal(http.StatusOK)) + Expect(stub.listBackendsCalled).To(BeFalse(), + "agent workers don't subscribe to backend.list; the endpoint must not issue the doomed NATS request") + + var list []messaging.NodeBackendInfo + Expect(json.Unmarshal(rec.Body.Bytes(), &list)).To(Succeed()) + Expect(list).To(BeEmpty()) + // Must be `[]`, not `null`, so the UI can render it. + Expect(rec.Body.String()).To(ContainSubstring("[]")) + }) + + It("consults the unloader (NATS) for a backend node", func() { + ctx := context.Background() + node := &nodes.BackendNode{Name: "backend-1", NodeType: nodes.NodeTypeBackend, Address: "10.0.0.1:50051"} + Expect(registry.Register(ctx, node, true)).To(Succeed()) + + stub := &stubNodeCommandSender{} + rec := callEndpoint(stub, node.ID) + + Expect(rec.Code).To(Equal(http.StatusOK)) + Expect(stub.listBackendsCalled).To(BeTrue(), + "backend nodes must still be queried over NATS") + + var list []messaging.NodeBackendInfo + Expect(json.Unmarshal(rec.Body.Bytes(), &list)).To(Succeed()) + Expect(list).To(HaveLen(1)) + Expect(list[0].Name).To(Equal("llama-cpp")) + }) +}) diff --git a/core/http/routes/nodes.go b/core/http/routes/nodes.go index f6a2124b8..e35bea240 100644 --- a/core/http/routes/nodes.go +++ b/core/http/routes/nodes.go @@ -88,7 +88,7 @@ func RegisterNodeAdminRoutes(e *echo.Echo, registry *nodes.NodeRegistry, unloade admin.POST("/:id/approve", localai.ApproveNodeEndpoint(registry, authDB, hmacSecret, natsCfg)) // Backend management on workers - admin.GET("/:id/backends", localai.ListBackendsOnNodeEndpoint(unloader)) + admin.GET("/:id/backends", localai.ListBackendsOnNodeEndpoint(unloader, registry)) admin.POST("/:id/backends/install", localai.InstallBackendOnNodeEndpoint(unloader, galleryService, opcache, appConfig)) admin.POST("/:id/backends/delete", localai.DeleteBackendOnNodeEndpoint(unloader)) From f3d829e2ef7d8b73241c90506d6f1c5d37981575 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 01:23:07 +0200 Subject: [PATCH 10/17] feat(distributed): add LOCALAI_DISTRIBUTED_SHARED_MODELS to skip staging on shared volumes (#10556) (#10566) In distributed mode, even when the frontend and workers share the same models directory via a shared volume mount, starting a model on a worker re-staged (re-downloaded) it: stageModelFiles always uploads model files into a tracking-key-namespaced subdir on the worker, and the staging probe only checks that staged location, so a file already present on the shared volume at the canonical path was never reused. Add a config switch LOCALAI_DISTRIBUTED_SHARED_MODELS (default false). When enabled, the operator asserts that all nodes mount the SAME models directory at the SAME path, so staging is unnecessary: the frontend's absolute model paths are already valid on the worker. In that mode stageModelFiles returns the cloned opts unchanged without uploading, leaving the path fields pointing at their canonical absolute paths so the worker loads them directly from the shared volume. The value is plumbed from DistributedConfig through SmartRouterOptions into the SmartRouter. Docs and docker-compose.distributed.yaml updated. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- core/application/distributed.go | 1 + core/cli/run.go | 4 + core/config/distributed_config.go | 15 ++++ core/services/nodes/router.go | 22 +++++ .../nodes/router_sharedmodels_test.go | 85 +++++++++++++++++++ docker-compose.distributed.yaml | 12 ++- docs/content/features/distributed-mode.md | 9 ++ 7 files changed, 146 insertions(+), 2 deletions(-) create mode 100644 core/services/nodes/router_sharedmodels_test.go diff --git a/core/application/distributed.go b/core/application/distributed.go index 3235e4304..a1efe0304 100644 --- a/core/application/distributed.go +++ b/core/application/distributed.go @@ -355,6 +355,7 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade PrefixProvider: prefixProvider, PrefixConfig: prefixCfg, Pressure: pressure, + SharedModels: cfg.Distributed.SharedModels, }) // Wire staging-progress broadcasting so file-staging shows up on every diff --git a/core/cli/run.go b/core/cli/run.go index 0302b5706..c3f7e51c8 100644 --- a/core/cli/run.go +++ b/core/cli/run.go @@ -160,6 +160,7 @@ type RunCMD struct { RegistrationRequireAuth bool `env:"LOCALAI_REGISTRATION_REQUIRE_AUTH" default:"false" help:"Fail startup when distributed mode is enabled but LOCALAI_REGISTRATION_TOKEN is empty (node endpoints and worker file-transfer server would otherwise be unauthenticated)" group:"distributed"` DistributedRequireAuth bool `env:"LOCALAI_DISTRIBUTED_REQUIRE_AUTH" default:"false" help:"Umbrella switch: require BOTH NATS JWT credentials and a registration token when distributed mode is enabled (implies --nats-require-auth and --registration-require-auth)" group:"distributed"` AutoApproveNodes bool `env:"LOCALAI_AUTO_APPROVE_NODES" default:"false" help:"Auto-approve new worker nodes (skip admin approval)" group:"distributed"` + DistributedSharedModels bool `env:"LOCALAI_DISTRIBUTED_SHARED_MODELS" default:"false" help:"Assert that every node mounts the SAME models directory at the SAME path (shared volume). When true, the router skips staging model files to workers and loads them directly from the shared path, avoiding re-downloads." group:"distributed"` DistributedPrefixCache bool `env:"LOCALAI_DISTRIBUTED_PREFIX_CACHE" default:"true" help:"Enable prefix-cache-aware routing in distributed mode (default true). When false, routing falls back to round-robin." group:"distributed"` DistributedPrefixCacheTTL string `env:"LOCALAI_DISTRIBUTED_PREFIX_CACHE_TTL" help:"Idle-timeout for prefix-cache index entries; also drives the background eviction cadence (every TTL/2). Default 5m." group:"distributed"` BackendInstallTimeout string `env:"LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT" help:"NATS round-trip timeout for backend.install requests sent to worker nodes (default 15m). Increase for slow links pulling multi-GB images." group:"distributed"` @@ -310,6 +311,9 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error { if r.DistributedRequireAuth { opts = append(opts, config.EnableDistributedRequireAuth) } + if r.DistributedSharedModels { + opts = append(opts, config.EnableDistributedSharedModels) + } if r.NatsAccountSeed != "" { opts = append(opts, config.WithNatsAccountSeed(r.NatsAccountSeed)) } diff --git a/core/config/distributed_config.go b/core/config/distributed_config.go index 3403487a9..7cb0a7c57 100644 --- a/core/config/distributed_config.go +++ b/core/config/distributed_config.go @@ -31,6 +31,14 @@ type DistributedConfig struct { // available to enforce just one layer. RequireAuth bool // LOCALAI_DISTRIBUTED_REQUIRE_AUTH AutoApproveNodes bool // --auto-approve-nodes / LOCALAI_AUTO_APPROVE_NODES (skip admin approval for new workers) + // SharedModels asserts that every node (frontend and workers) mounts the + // SAME models directory at the SAME path (e.g. a shared volume, as in + // docker-compose.distributed.yaml). When true, the router skips staging + // model files to workers entirely: the frontend's absolute model paths are + // already valid on the worker, so re-uploading them into a per-model + // subdirectory only re-downloads what is already present (#10556). Default + // false preserves the historical per-node staging behavior. + SharedModels bool // --distributed-shared-models / LOCALAI_DISTRIBUTED_SHARED_MODELS // NATS JWT auth (optional; see pkg/natsauth and docs/features/distributed-mode.md) NatsAccountSeed string // LOCALAI_NATS_ACCOUNT_SEED — account signing seed to mint per-node worker JWTs @@ -282,6 +290,13 @@ var EnableAutoApproveNodes = func(o *ApplicationConfig) { o.Distributed.AutoApproveNodes = true } +// EnableDistributedSharedModels marks the cluster as sharing one models +// directory across all nodes, so the router skips staging model files to +// workers (see DistributedConfig.SharedModels). +var EnableDistributedSharedModels = func(o *ApplicationConfig) { + o.Distributed.SharedModels = true +} + // DisablePrefixCache turns off prefix-cache-aware routing (falls back to // round-robin). Prefix-cache routing is enabled by default in distributed mode. var DisablePrefixCache = func(o *ApplicationConfig) { diff --git a/core/services/nodes/router.go b/core/services/nodes/router.go index ce3de3290..664672e39 100644 --- a/core/services/nodes/router.go +++ b/core/services/nodes/router.go @@ -63,6 +63,11 @@ type SmartRouterOptions struct { // The reconciler reads the same instance to autoscale a saturated cache-warm // replica. nil disables recording (the disabled path stays a no-op). Pressure *prefixcache.Pressure + // SharedModels asserts that every node mounts the same models directory at + // the same path. When true, stageModelFiles skips all uploading and leaves + // the absolute model paths untouched so the worker loads them directly from + // the shared volume (#10556). See config.DistributedConfig.SharedModels. + SharedModels bool } // SmartRouter routes inference requests to the best available backend node. @@ -93,6 +98,9 @@ type SmartRouter struct { // per-request routing doesn't stall behind a busy backend's serialized // HealthCheck/Predict. See probe_cache.go for the rationale. probeCache *probeCache + // sharedModels skips file staging when all nodes mount the same models + // directory at the same path (see SmartRouterOptions.SharedModels). + sharedModels bool } // probeCacheTTL is how long a successful gRPC HealthCheck on a backend is @@ -122,6 +130,7 @@ func NewSmartRouter(registry ModelRouter, opts SmartRouterOptions) *SmartRouter prefixProvider: opts.PrefixProvider, prefixConfig: opts.PrefixConfig, pressure: opts.Pressure, + sharedModels: opts.SharedModels, } } @@ -947,6 +956,19 @@ func (r *SmartRouter) buildClientForAddr(node *BackendNode, addr string, paralle // simply remove the {ModelsPath}/{trackingKey}/ directory. func (r *SmartRouter) stageModelFiles(ctx context.Context, node *BackendNode, opts *pb.ModelOptions, trackingKey string) (*pb.ModelOptions, error) { opts = proto.Clone(opts).(*pb.ModelOptions) + + // Shared-models mode: every node mounts the same models directory at the + // same path, so the frontend's absolute model paths are already valid on the + // worker. Staging would only re-upload files that already exist on the shared + // volume (under a tracking-key subdir the probe never reuses), re-downloading + // the model on every load (#10556). Return the clone untouched: no upload, no + // path rewrite, no staging tracker. + if r.sharedModels { + xlog.Info("Skipping model file staging: shared-models mode is on (LOCALAI_DISTRIBUTED_SHARED_MODELS); worker loads directly from the shared volume", + "node", node.Name, "modelFile", opts.ModelFile, "trackingKey", trackingKey) + return opts, nil + } + xlog.Info("Staging model files for remote node", "node", node.Name, "modelFile", opts.ModelFile, "trackingKey", trackingKey) // Derive the frontend models directory from ModelFile and Model. diff --git a/core/services/nodes/router_sharedmodels_test.go b/core/services/nodes/router_sharedmodels_test.go new file mode 100644 index 000000000..8cdf061cd --- /dev/null +++ b/core/services/nodes/router_sharedmodels_test.go @@ -0,0 +1,85 @@ +package nodes + +import ( + "context" + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + pb "github.com/mudler/LocalAI/pkg/grpc/proto" +) + +// These tests cover shared-models mode (LOCALAI_DISTRIBUTED_SHARED_MODELS): when +// every node mounts the same models directory at the same path, the router must +// NOT stage model files to workers. The canonical absolute path is already valid +// on the worker, so staging would only re-download what is already present +// (#10556). +var _ = Describe("stageModelFiles shared-models mode", func() { + var ( + stager *fakeFileStager + node *BackendNode + tmp string + gguf string + modelID = "ornith-1.0-35b" + ) + + BeforeEach(func() { + stager = &fakeFileStager{} + node = &BackendNode{ID: "node-1", Name: "node-1", Address: "10.0.0.1:50051"} + tmp = GinkgoT().TempDir() + + modelDir := filepath.Join(tmp, "models", "llama-cpp", "models") + Expect(os.MkdirAll(modelDir, 0o755)).To(Succeed()) + gguf = filepath.Join(modelDir, "ornith.gguf") + Expect(os.WriteFile(gguf, []byte("weights"), 0o644)).To(Succeed()) + }) + + It("does not stage and keeps the canonical absolute ModelFile when shared-models is enabled", func() { + router := &SmartRouter{ + fileStager: stager, + stagingTracker: NewStagingTracker(), + sharedModels: true, + } + + opts := &pb.ModelOptions{ + Model: "llama-cpp/models/ornith.gguf", + ModelFile: gguf, + } + + staged, err := router.stageModelFiles(context.Background(), node, opts, modelID) + Expect(err).ToNot(HaveOccurred()) + + // The file stager must never be touched: no upload, no re-download. + Expect(stager.ensureCalls).To(BeEmpty()) + // The worker loads directly from the shared volume, so the path is unchanged. + Expect(staged.ModelFile).To(Equal(gguf)) + }) + + It("stages files (existing behavior) when shared-models is disabled", func() { + router := &SmartRouter{ + fileStager: stager, + stagingTracker: NewStagingTracker(), + sharedModels: false, + } + + opts := &pb.ModelOptions{ + Model: "llama-cpp/models/ornith.gguf", + ModelFile: gguf, + } + + staged, err := router.stageModelFiles(context.Background(), node, opts, modelID) + Expect(err).ToNot(HaveOccurred()) + + // Default mode uploads the model file to the worker. + Expect(stager.ensureCalls).ToNot(BeEmpty()) + stagedLocals := make([]string, 0, len(stager.ensureCalls)) + for _, c := range stager.ensureCalls { + stagedLocals = append(stagedLocals, c.localPath) + } + Expect(stagedLocals).To(ContainElement(gguf)) + // ModelFile is rewritten to the remote (tracking-key namespaced) path. + Expect(staged.ModelFile).ToNot(Equal(gguf)) + }) +}) diff --git a/docker-compose.distributed.yaml b/docker-compose.distributed.yaml index 778293c84..81397e4ac 100644 --- a/docker-compose.distributed.yaml +++ b/docker-compose.distributed.yaml @@ -57,6 +57,11 @@ services: LOCALAI_AGENT_POOL_VECTOR_ENGINE: "postgres" LOCALAI_AGENT_POOL_DATABASE_URL: "postgresql://localai:localai@postgres:5432/localai?sslmode=disable" LOCALAI_REGISTRATION_TOKEN: "changeme" # Change this in production! + # Shared-models mode (optional): set when every node mounts the SAME + # models directory at the SAME path (see "Shared Volume Mode" below). + # The router then skips gRPC file staging and workers load models + # directly from the shared volume instead of re-downloading them. + # LOCALAI_DISTRIBUTED_SHARED_MODELS: "true" # Auth (required for distributed mode — must use PostgreSQL) LOCALAI_AUTH: "true" LOCALAI_AUTH_DATABASE_URL: "postgresql://localai:localai@postgres:5432/localai?sslmode=disable" @@ -157,8 +162,11 @@ services: # Then add to the volumes section: # shared_models: # - # With shared volumes, model files are already available on the backend — - # gRPC file staging becomes a no-op (paths match). + # With shared volumes the model files are already present on every worker at + # the same path. Set LOCALAI_DISTRIBUTED_SHARED_MODELS=true on the frontend + # (see its environment above) so the router skips gRPC file staging and the + # worker loads the model directly from the shared path instead of + # re-downloading it into a per-model subdirectory. # --- Adding More Workers --- # Copy the worker-1 service above and change: diff --git a/docs/content/features/distributed-mode.md b/docs/content/features/distributed-mode.md index e5a0b790a..a64f6636f 100644 --- a/docs/content/features/distributed-mode.md +++ b/docs/content/features/distributed-mode.md @@ -67,6 +67,7 @@ The frontend is a standard LocalAI instance with distributed mode enabled. These | `--registration-require-auth` | `LOCALAI_REGISTRATION_REQUIRE_AUTH` | `false` | Fail startup when distributed mode is enabled but the registration token is empty (node endpoints and worker file-transfer would otherwise be unauthenticated) | | `--distributed-require-auth` | `LOCALAI_DISTRIBUTED_REQUIRE_AUTH` | `false` | **Umbrella switch.** Implies both `--nats-require-auth` and `--registration-require-auth` — one knob to lock down the NATS bus *and* the registration/file-transfer layer. Set this in production instead of the two granular flags. | | `--auto-approve-nodes` | `LOCALAI_AUTO_APPROVE_NODES` | `false` | Auto-approve new worker nodes (skip admin approval) | +| `--distributed-shared-models` | `LOCALAI_DISTRIBUTED_SHARED_MODELS` | `false` | Assert that every node mounts the **same** models directory at the **same** path (a shared volume). When `true`, the router skips file staging entirely and workers load models directly from the shared path instead of re-downloading them. See [Shared models directory](#shared-models-directory). | | `--auth` | `LOCALAI_AUTH` | `false` | **Must be `true`** for distributed mode | | `--auth-database-url` | `LOCALAI_AUTH_DATABASE_URL` | *(required)* | PostgreSQL connection URL | | `--backend-install-timeout` | `LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT` | `15m` | How long the frontend waits for a worker to acknowledge a backend install before considering the request stalled. Raise it when workers pull large backend images over slow links. If a worker takes longer than this, the operation shows as "still installing in background" in the admin UI and clears once the worker finishes. | @@ -133,6 +134,14 @@ When S3 is not configured, model files are transferred directly from the fronten For high-throughput or very large model files, S3 can be more efficient since it avoids streaming through the frontend. +### Shared models directory + +If every node (frontend and workers) mounts the **same** models directory at the **same** path - for example a shared volume or network filesystem, as shown in the "Shared Volume Mode" section of `docker-compose.distributed.yaml` - the model files are already present on each worker at their canonical path. In that case staging is wasted work: it copies files that already exist into a per-model subdirectory the worker then loads from, which shows up as a re-download of a model you already have. + +Set `LOCALAI_DISTRIBUTED_SHARED_MODELS=true` (or `--distributed-shared-models`) on the frontend to skip staging entirely. The router then leaves the model's absolute paths untouched and the worker loads them directly from the shared volume. + +This flag is a contract you assert: all nodes must mount identical paths. Leave it off (the default) when workers have independent models directories - the frontend stages files to them over HTTP (or S3) as described above. + {{% notice warning %}} The worker HTTP file transfer server is authenticated by `LOCALAI_REGISTRATION_TOKEN`. If the token is **empty**, the server **fails open** — anyone who can reach the port gets read/write access to the worker's models/staging/data directories (a remote model-poisoning / exfiltration vector). The worker logs a loud warning at startup in this case. Always set `LOCALAI_REGISTRATION_TOKEN` in distributed mode, and set `LOCALAI_DISTRIBUTED_REQUIRE_AUTH=true` (frontend **and** workers) to make a missing token *or* missing NATS credentials a hard startup error rather than a silent fail-open. Firewall the file-transfer port (gRPC base − 1) so only the frontend can reach it. {{% /notice %}} From 471e38e4e76feacfebf3d111433aa0189fd12248 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 01:55:44 +0200 Subject: [PATCH 11/17] chore: :arrow_up: Update leejet/stable-diffusion.cpp to `9956436c925a367daeab097598b1ea1f32d3503f` (#10533) :arrow_up: Update leejet/stable-diffusion.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --- backend/go/stablediffusion-ggml/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/go/stablediffusion-ggml/Makefile b/backend/go/stablediffusion-ggml/Makefile index 7a9917ea8..97940c18d 100644 --- a/backend/go/stablediffusion-ggml/Makefile +++ b/backend/go/stablediffusion-ggml/Makefile @@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1) # stablediffusion.cpp (ggml) STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp -STABLEDIFFUSION_GGML_VERSION?=8caa3f908ae6d4a4bef531e73b9a969f266a3d1f +STABLEDIFFUSION_GGML_VERSION?=9956436c925a367daeab097598b1ea1f32d3503f CMAKE_ARGS+=-DGGML_MAX_NAME=128 From ade9cc9e37ff8e2c237f30332f52134f5c8fd016 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 02:02:15 +0200 Subject: [PATCH 12/17] fix(openresponses): bound resume-stream buffer and enforce response ownership (#10569) The background=true resumable-stream path had two latent issues. 1. Unbounded resume buffer. AppendEvent grew StreamEvents without limit, so a long-running or abandoned background generation could consume process memory without bound. The store now caps the buffer (event count and total bytes, mirroring llama.cpp's byte-capped slot ring), evicting oldest events from the front and advancing a droppedThrough watermark. GetEventsAfter returns ErrOffsetLost when the requested starting_after is below the watermark, and handleStreamResume surfaces that as HTTP 409 before committing to the SSE response, so a resuming client gets a clear error instead of a silently truncated stream. 2. Missing ownership check (IDOR). GET /responses/:id, its stream resume, and /cancel looked up responses purely by ID, letting any caller who knows or guesses an ID read or cancel another caller's response. Responses now carry the creating caller's identity (auth.GetUser), stamped at creation and compared on read/cancel/resume; a mismatch returns 404 (not 403) so existence is not leaked. Backward compatible: responses with no owner (single-key / no-auth deployments) remain accessible. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- .../http/endpoints/openresponses/responses.go | 55 ++++++- core/http/endpoints/openresponses/store.go | 137 +++++++++++++++--- .../endpoints/openresponses/store_test.go | 80 ++++++++++ 3 files changed, 248 insertions(+), 24 deletions(-) diff --git a/core/http/endpoints/openresponses/responses.go b/core/http/endpoints/openresponses/responses.go index 916380d01..f8d741508 100644 --- a/core/http/endpoints/openresponses/responses.go +++ b/core/http/endpoints/openresponses/responses.go @@ -3,6 +3,7 @@ package openresponses import ( "context" "encoding/json" + "errors" "fmt" "time" @@ -10,6 +11,7 @@ import ( "github.com/labstack/echo/v4" "github.com/mudler/LocalAI/core/backend" "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/http/auth" mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp" openaiEndpoint "github.com/mudler/LocalAI/core/http/endpoints/openai" "github.com/mudler/LocalAI/core/http/middleware" @@ -246,8 +248,11 @@ func ResponsesEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, eval // Create cancellable context for background execution bgCtx, bgCancel := context.WithCancel(context.Background()) - // Store the background response + // Store the background response and stamp its owner before the ID + // is returned to the client, so later GET/cancel/resume can verify + // the caller owns it. store.StoreBackground(responseID, input, queuedResponse, bgCancel, input.Stream) + store.SetOwner(responseID, ownerFromContext(c)) // Start background processing goroutine go func() { @@ -1587,6 +1592,7 @@ func handleOpenResponsesNonStream(c echo.Context, responseID string, createdAt i if shouldStore { store := GetGlobalStore() store.Store(responseID, input, response) + store.SetOwner(responseID, ownerFromContext(c)) } return c.JSON(200, response) @@ -2322,6 +2328,7 @@ func handleOpenResponsesStream(c echo.Context, responseID string, createdAt int6 if shouldStore { store := GetGlobalStore() store.Store(responseID, input, responseCompleted) + store.SetOwner(responseID, ownerFromContext(c)) } // Send [DONE] @@ -2966,6 +2973,18 @@ func convertORToolsToOpenAIFormat(orTools []schema.ORFunctionTool) []functions.T return result } +// ownerFromContext returns the identity (user ID) of the authenticated +// caller, or empty string when no authentication was performed (single-key / +// no-auth deployments). It is the value stamped on a response at creation and +// compared on read/cancel/resume to prevent one caller from accessing +// another's response by guessing its ID. +func ownerFromContext(c echo.Context) string { + if u := auth.GetUser(c); u != nil { + return u.ID + } + return "" +} + // GetResponseEndpoint returns a handler for GET /responses/:id // This endpoint is used for polling background responses or resuming streaming // @Summary Get a response by ID @@ -2991,6 +3010,12 @@ func GetResponseEndpoint() func(c echo.Context) error { return sendOpenResponsesError(c, 404, "not_found", fmt.Sprintf("response not found: %s", responseID), "id") } + // Enforce response ownership. Return 404 (not 403) on mismatch so the + // existence of another caller's response is not leaked. + if !accessAllowed(stored, ownerFromContext(c)) { + return sendOpenResponsesError(c, 404, "not_found", fmt.Sprintf("response not found: %s", responseID), "id") + } + // Check if streaming resume is requested streamParam := c.QueryParam("stream") if streamParam == "true" { @@ -3022,16 +3047,21 @@ func GetResponseEndpoint() func(c echo.Context) error { // handleStreamResume handles resuming a streaming response from a specific sequence number func handleStreamResume(c echo.Context, store *ResponseStore, responseID string, stored *StoredResponse, startingAfter int) error { + // Fetch buffered events before committing to an SSE response so an + // offset-lost gap can be reported as a clean HTTP status rather than a + // silently truncated event stream. + events, err := store.GetEventsAfter(responseID, startingAfter) + if err != nil { + if errors.Is(err, ErrOffsetLost) { + return sendOpenResponsesError(c, 409, "invalid_request_error", fmt.Sprintf("starting_after=%d is older than the oldest retained event; the resume buffer evicted those events and the stream cannot be resumed from that point", startingAfter), "starting_after") + } + return sendOpenResponsesError(c, 500, "server_error", fmt.Sprintf("failed to get events: %v", err), "") + } + c.Response().Header().Set("Content-Type", "text/event-stream") c.Response().Header().Set("Cache-Control", "no-cache") c.Response().Header().Set("Connection", "keep-alive") - // Get buffered events after the starting point - events, err := store.GetEventsAfter(responseID, startingAfter) - if err != nil { - return sendOpenResponsesError(c, 500, "server_error", fmt.Sprintf("failed to get events: %v", err), "") - } - // Send all buffered events for _, event := range events { fmt.Fprintf(c.Response().Writer, "event: %s\ndata: %s\n\n", event.EventType, string(event.Data)) @@ -3126,6 +3156,17 @@ func CancelResponseEndpoint() func(c echo.Context) error { } store := GetGlobalStore() + + // Look up first so ownership can be checked before any mutation. + stored, err := store.Get(responseID) + if err != nil { + return sendOpenResponsesError(c, 404, "not_found", fmt.Sprintf("response not found: %s", responseID), "id") + } + // Return 404 (not 403) on owner mismatch so existence is not leaked. + if !accessAllowed(stored, ownerFromContext(c)) { + return sendOpenResponsesError(c, 404, "not_found", fmt.Sprintf("response not found: %s", responseID), "id") + } + response, err := store.Cancel(responseID) if err != nil { return sendOpenResponsesError(c, 404, "not_found", fmt.Sprintf("response not found: %s", responseID), "id") diff --git a/core/http/endpoints/openresponses/store.go b/core/http/endpoints/openresponses/store.go index bea5b7413..ab52261e5 100644 --- a/core/http/endpoints/openresponses/store.go +++ b/core/http/endpoints/openresponses/store.go @@ -3,6 +3,7 @@ package openresponses import ( "context" "encoding/json" + "errors" "fmt" "sync" "time" @@ -11,6 +12,30 @@ import ( "github.com/mudler/xlog" ) +const ( + // defaultMaxStreamEvents bounds how many resume-buffer events a single + // background response retains. Without a cap, a long-running or abandoned + // background generation grows StreamEvents without limit and can exhaust + // process memory. When the cap is exceeded the oldest events are evicted + // from the front (see AppendEvent). Mirrors llama.cpp's byte-capped slot + // ring used for resumable /slots state. + defaultMaxStreamEvents = 8192 + + // defaultMaxStreamBytes caps the total serialized size of retained + // resume-buffer events, evicting oldest-first when exceeded. This guards + // against a handful of very large events defeating the count cap. 0 + // disables the byte cap (count cap still applies). + defaultMaxStreamBytes = 64 << 20 // 64 MiB +) + +// ErrOffsetLost is returned by GetEventsAfter when the requested +// starting_after sequence number is older than the oldest event still +// retained in the resume buffer (i.e. the events between the requested +// offset and the current watermark were evicted by the cap). Callers should +// surface this to clients as a distinct error instead of silently returning +// a truncated stream that omits the dropped events. +var ErrOffsetLost = errors.New("resume offset lost: requested events were evicted from the buffer") + // ResponseStore provides thread-safe storage for Open Responses API responses type ResponseStore struct { mu sync.RWMutex @@ -18,6 +43,12 @@ type ResponseStore struct { ttl time.Duration // Time-to-live for stored responses (0 = no expiration) cleanupCtx context.Context cleanupCancel context.CancelFunc + + // maxStreamEvents / maxStreamBytes bound the per-response resume buffer. + // Set once at construction from the default constants; tests may lower + // them. A value <= 0 disables that particular cap. + maxStreamEvents int + maxStreamBytes int } // StreamedEvent represents a buffered SSE event for streaming resume @@ -35,6 +66,12 @@ type StoredResponse struct { StoredAt time.Time ExpiresAt *time.Time // nil if no expiration + // Owner is the identity (user ID) that created this response. It is set + // once at creation and never mutated, so it can be read without holding + // mu. Empty means "no owner" (single-key / no-auth deployments), in which + // case ownership checks are skipped for backward compatibility. + Owner string + // Background execution support CancelFunc context.CancelFunc // For cancellation of background tasks StreamEvents []StreamedEvent // Buffered events for streaming resume @@ -42,6 +79,14 @@ type StoredResponse struct { IsBackground bool // Was created with background=true EventsChan chan struct{} // Signals new events for live subscribers mu sync.RWMutex // Protect concurrent access to this response + + // streamBytes tracks the total serialized size of the events currently + // retained in StreamEvents, used to enforce the byte cap. droppedThrough + // is the highest sequence number evicted from the front of the buffer + // (-1 = nothing evicted); it is the watermark GetEventsAfter compares + // against to detect a lost resume offset. Both are guarded by mu. + streamBytes int + droppedThrough int } var getGlobalStore = sync.OnceValue(func() *ResponseStore { @@ -81,8 +126,10 @@ func (s *ResponseStore) SetTTL(ttl time.Duration) { // If ttl is 0, responses are stored indefinitely func NewResponseStore(ttl time.Duration) *ResponseStore { store := &ResponseStore{ - responses: make(map[string]*StoredResponse), - ttl: ttl, + responses: make(map[string]*StoredResponse), + ttl: ttl, + maxStreamEvents: defaultMaxStreamEvents, + maxStreamBytes: defaultMaxStreamBytes, } // Start cleanup goroutine if TTL is set @@ -109,11 +156,12 @@ func (s *ResponseStore) Store(responseID string, request *schema.OpenResponsesRe } stored := &StoredResponse{ - Request: request, - Response: response, - Items: items, - StoredAt: time.Now(), - ExpiresAt: nil, + Request: request, + Response: response, + Items: items, + StoredAt: time.Now(), + ExpiresAt: nil, + droppedThrough: -1, } // Set expiration if TTL is configured @@ -256,16 +304,17 @@ func (s *ResponseStore) StoreBackground(responseID string, request *schema.OpenR } stored := &StoredResponse{ - Request: request, - Response: response, - Items: items, - StoredAt: time.Now(), - ExpiresAt: nil, - CancelFunc: cancelFunc, - StreamEvents: []StreamedEvent{}, - StreamEnabled: streamEnabled, - IsBackground: true, - EventsChan: make(chan struct{}, 100), // Buffered channel for event notifications + Request: request, + Response: response, + Items: items, + StoredAt: time.Now(), + ExpiresAt: nil, + CancelFunc: cancelFunc, + StreamEvents: []StreamedEvent{}, + StreamEnabled: streamEnabled, + IsBackground: true, + EventsChan: make(chan struct{}, 100), // Buffered channel for event notifications + droppedThrough: -1, } // Set expiration if TTL is configured @@ -349,6 +398,25 @@ func (s *ResponseStore) AppendEvent(responseID string, event *schema.ORStreamEve EventType: event.Type, Data: data, }) + stored.streamBytes += len(data) + + // Evict oldest events from the front once either cap is exceeded. The + // byte cap never evicts the only remaining event (a single oversized + // event is still served once). Each eviction advances droppedThrough so + // a later resume below the watermark is reported as ErrOffsetLost rather + // than silently skipping the dropped events. + for (s.maxStreamEvents > 0 && len(stored.StreamEvents) > s.maxStreamEvents) || + (s.maxStreamBytes > 0 && stored.streamBytes > s.maxStreamBytes && len(stored.StreamEvents) > 1) { + evicted := stored.StreamEvents[0] + stored.streamBytes -= len(evicted.Data) + if evicted.SequenceNumber > stored.droppedThrough { + stored.droppedThrough = evicted.SequenceNumber + } + // Release the evicted payload so it can be GC'd even though the + // backing array element is still owned by the slice until reuse. + stored.StreamEvents[0].Data = nil + stored.StreamEvents = stored.StreamEvents[1:] + } stored.mu.Unlock() // Notify any subscribers of new event @@ -374,6 +442,14 @@ func (s *ResponseStore) GetEventsAfter(responseID string, startingAfter int) ([] stored.mu.RLock() defer stored.mu.RUnlock() + // If the requested offset is older than the watermark, the events the + // client expects next (those in (startingAfter, droppedThrough]) were + // evicted by the cap. Signal the gap rather than returning a stream that + // silently skips them. + if startingAfter < stored.droppedThrough { + return nil, ErrOffsetLost + } + var result []StreamedEvent for _, event := range stored.StreamEvents { if event.SequenceNumber > startingAfter { @@ -447,3 +523,30 @@ func (s *ResponseStore) IsStreamEnabled(responseID string) (bool, error) { return stored.StreamEnabled, nil } + +// SetOwner records the identity that owns a stored response. It is called +// once, right after the response is stored and before its ID is handed back +// to any client, so no lock on the stored response is required. A no-op for +// an empty owner or unknown response ID. +func (s *ResponseStore) SetOwner(responseID, owner string) { + if owner == "" { + return + } + + s.mu.RLock() + stored, exists := s.responses[responseID] + s.mu.RUnlock() + if !exists { + return + } + + stored.Owner = owner +} + +// accessAllowed reports whether a caller identified by callerID may read or +// mutate the given stored response. An empty owner (single-key / no-auth +// deployments) is accessible by anyone, preserving backward compatibility; +// otherwise the caller identity must match the recorded owner. +func accessAllowed(stored *StoredResponse, callerID string) bool { + return stored.Owner == "" || stored.Owner == callerID +} diff --git a/core/http/endpoints/openresponses/store_test.go b/core/http/endpoints/openresponses/store_test.go index 360e32df4..d59db2d38 100644 --- a/core/http/endpoints/openresponses/store_test.go +++ b/core/http/endpoints/openresponses/store_test.go @@ -585,6 +585,86 @@ var _ = Describe("ResponseStore", func() { Expect(enabled2).To(BeFalse()) }) + It("should bound the resume buffer and evict oldest events past the cap", func() { + // Lower the caps so the test stays fast; production defaults are + // large. Same-package access to the unexported fields is fine. + store.maxStreamEvents = 5 + store.maxStreamBytes = 0 // count cap only for this test + + responseID := "resp_buffer_cap" + request := &schema.OpenResponsesRequest{Model: "test"} + response := &schema.ORResponseResource{ + ID: responseID, + Object: "response", + Status: schema.ORStatusInProgress, + } + + _, cancel := context.WithCancel(context.Background()) + defer cancel() + + store.StoreBackground(responseID, request, response, cancel, true) + + // Append well past the cap. + const total = 20 + for i := range total { + err := store.AppendEvent(responseID, &schema.ORStreamEvent{ + Type: "response.output_text.delta", + SequenceNumber: i, + }) + Expect(err).ToNot(HaveOccurred()) + } + + stored, err := store.Get(responseID) + Expect(err).ToNot(HaveOccurred()) + + // (a) Buffer length stays bounded by the cap. + Expect(len(stored.StreamEvents)).To(Equal(5)) + + // (b) Oldest events were evicted: only the last 5 sequence numbers + // remain (15..19). + Expect(stored.StreamEvents[0].SequenceNumber).To(Equal(15)) + Expect(stored.StreamEvents[len(stored.StreamEvents)-1].SequenceNumber).To(Equal(19)) + + // Asking for events after the last retained seq still works. + retained, err := store.GetEventsAfter(responseID, 14) + Expect(err).ToNot(HaveOccurred()) + Expect(retained).To(HaveLen(5)) + + // (c) Asking below the dropped watermark returns ErrOffsetLost. + _, err = store.GetEventsAfter(responseID, 0) + Expect(err).To(MatchError(ErrOffsetLost)) + + _, err = store.GetEventsAfter(responseID, -1) + Expect(err).To(MatchError(ErrOffsetLost)) + }) + + It("should record and enforce response ownership", func() { + responseID := "resp_owner_test" + request := &schema.OpenResponsesRequest{Model: "test"} + response := &schema.ORResponseResource{ID: responseID, Object: "response", Status: schema.ORStatusCompleted} + + store.Store(responseID, request, response) + store.SetOwner(responseID, "userA") + + stored, err := store.Get(responseID) + Expect(err).ToNot(HaveOccurred()) + Expect(stored.Owner).To(Equal("userA")) + + // Owner matches -> allowed; different identity -> denied. + Expect(accessAllowed(stored, "userA")).To(BeTrue()) + Expect(accessAllowed(stored, "userB")).To(BeFalse()) + + // Backward compatibility: a response with no owner is accessible + // by any caller (single-key / no-auth deployments). + noOwnerID := "resp_no_owner" + store.Store(noOwnerID, request, &schema.ORResponseResource{ID: noOwnerID, Object: "response"}) + noOwner, err := store.Get(noOwnerID) + Expect(err).ToNot(HaveOccurred()) + Expect(noOwner.Owner).To(BeEmpty()) + Expect(accessAllowed(noOwner, "anyone")).To(BeTrue()) + Expect(accessAllowed(noOwner, "")).To(BeTrue()) + }) + It("should notify subscribers of new events", func() { responseID := "resp_events_chan" request := &schema.OpenResponsesRequest{Model: "test"} From 6740e988d24633393d260e6c8e330cacd7e788b9 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 08:56:06 +0200 Subject: [PATCH 13/17] chore: :arrow_up: Update ggml-org/whisper.cpp to `0ae02cdb2c7317b50991367c165736ce42ed96ac` (#10532) :arrow_up: Update ggml-org/whisper.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --- backend/go/whisper/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/go/whisper/Makefile b/backend/go/whisper/Makefile index 6dd13dd2c..99d959c0c 100644 --- a/backend/go/whisper/Makefile +++ b/backend/go/whisper/Makefile @@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1) # whisper.cpp version WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp -WHISPER_CPP_VERSION?=43d78af5be58f41d6ffbc227d608f104577741ea +WHISPER_CPP_VERSION?=0ae02cdb2c7317b50991367c165736ce42ed96ac SO_TARGET?=libgowhisper.so CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF From e68ca109c587468271ca9fc42f3cfe1f2374cfcd Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 08:56:24 +0200 Subject: [PATCH 14/17] chore: :arrow_up: Update CrispStrobe/CrispASR to `6514c9da00b03a2f0f1b49a43fae4f3a01a41844` (#10535) :arrow_up: Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --- backend/go/crispasr/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/go/crispasr/Makefile b/backend/go/crispasr/Makefile index d6921b15a..e8d36851d 100644 --- a/backend/go/crispasr/Makefile +++ b/backend/go/crispasr/Makefile @@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1) # CrispASR version (release tag) CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR -CRISPASR_VERSION?=8f1218141b792b8868861c1af17ba1e361b05dc0 +CRISPASR_VERSION?=6514c9da00b03a2f0f1b49a43fae4f3a01a41844 SO_TARGET?=libgocrispasr.so CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF From 13b1ae53bcd14288c3f234a5a0ed902de337bead Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 08:56:40 +0200 Subject: [PATCH 15/17] chore: :arrow_up: Update ggml-org/llama.cpp to `0ed235ea2c17a19fc8238668653946721ed136fd` (#10536) * :arrow_up: Update ggml-org/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(llama-cpp): link server-stream.cpp TU into grpc-server for upstream 0ed235ea (#10536) Upstream llama.cpp 0ed235ea added an SSE stream-resumption layer in a new translation unit tools/server/server-stream.cpp, which defines stream_session, stream_pipe_producer and the g_stream_sessions manager. server-context.cpp (already #included into grpc-server.cpp) now calls into it via spipe->cleanup(), stream_aware_should_stop() and stream_session_attach_pipe(), so without the new TU the grpc-server link fails on every arch with: undefined reference to `stream_pipe_producer::cleanup()' prepare.sh already copies every tools/server/* file into tools/grpc-server/, so the source is present; the only missing piece was including its definitions. Add an __has_include-guarded #include "server-stream.cpp" before server-context.cpp, mirroring the existing server-chat.cpp and server-schema.cpp guards, keeping the source compatible with older pins/forks that predate the split. The file is self-contained (its only external symbols come from server-common, already in the TU) so it adds no new undefined references; the http route-handler factories it also defines are unused in the grpc path but harmless. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(llama-cpp): build renamed ggml-rpc-server target for upstream 0ed235ea (#10536) Upstream renamed the RPC server CMake target and binary from `rpc-server` to `ggml-rpc-server` (tools/rpc/CMakeLists.txt: `set(TARGET ggml-rpc-server)`), so the RPC-enabled grpc build failed with "No rule to make target 'rpc-server'". The grpc-server itself links fine after the server-stream.cpp fix; this only updates the RPC target name and the binary path copied to llama-cpp-rpc-server. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Ettore Di Giacinto Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: Ettore Di Giacinto --- backend/cpp/llama-cpp/Makefile | 6 +++--- backend/cpp/llama-cpp/grpc-server.cpp | 13 +++++++++++++ 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/backend/cpp/llama-cpp/Makefile b/backend/cpp/llama-cpp/Makefile index b02bdac07..f542ac3b4 100644 --- a/backend/cpp/llama-cpp/Makefile +++ b/backend/cpp/llama-cpp/Makefile @@ -1,5 +1,5 @@ -LLAMA_VERSION?=9d5d882d8cd0f0a9283d87ed5e6fe3ee0d925fb1 +LLAMA_VERSION?=0ed235ea2c17a19fc8238668653946721ed136fd LLAMA_REPO?=https://github.com/ggerganov/llama.cpp CMAKE_ARGS?= @@ -156,11 +156,11 @@ llama-cpp-grpc: llama.cpp cp -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build purge $(info ${GREEN}I llama-cpp build info:grpc${RESET}) - CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" TARGET="--target grpc-server --target rpc-server" $(MAKE) VARIANT="llama-cpp-grpc-build" build-llama-cpp-grpc-server + CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" TARGET="--target grpc-server --target ggml-rpc-server" $(MAKE) VARIANT="llama-cpp-grpc-build" build-llama-cpp-grpc-server cp -rfv $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build/grpc-server llama-cpp-grpc llama-cpp-rpc-server: llama-cpp-grpc - cp -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build/llama.cpp/build/bin/rpc-server llama-cpp-rpc-server + cp -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build/llama.cpp/build/bin/ggml-rpc-server llama-cpp-rpc-server llama.cpp: mkdir -p llama.cpp diff --git a/backend/cpp/llama-cpp/grpc-server.cpp b/backend/cpp/llama-cpp/grpc-server.cpp index 9d17e23b1..a02d461f4 100644 --- a/backend/cpp/llama-cpp/grpc-server.cpp +++ b/backend/cpp/llama-cpp/grpc-server.cpp @@ -30,6 +30,19 @@ #define LOCALAI_HAS_SERVER_SCHEMA 1 #include "server-schema.cpp" #endif +// server-stream.cpp exists only in llama.cpp after the upstream refactor that +// added the SSE stream-resumption layer (stream_session/stream_pipe_producer). +// server-context.cpp calls into it (spipe->cleanup(), stream_aware_should_stop, +// stream_session_attach_pipe), so its definitions must be part of this +// translation unit or the link fails with "undefined reference to +// stream_pipe_producer::cleanup()". The file is self-contained (its only +// external symbols come from server-common, already pulled in above) and the +// http route-handler factories it also defines are unused here but harmless. +// __has_include keeps the source compatible with older pins/forks that predate +// the split. +#if __has_include("server-stream.cpp") +#include "server-stream.cpp" +#endif #include "server-context.cpp" // LocalAI From d3a26f961d19d81efd215b960b52ef170cd2e633 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 08:57:11 +0200 Subject: [PATCH 16/17] fix(ik-llama): port multimodal path to mtmd API and bump to f96eaddb (#10534) (#10568) * fix(ik-llama): port multimodal path to mtmd API and bump to f96eaddb (#10534) The IK_LLAMA_VERSION bump to f96eaddba8bed6a9a5e628bbf6a566775c70b49c pulls in upstream commit "Prune examples/llava", which deletes examples/llava (clip.* / llava.*). The ik-llama backend's grpc-server.cpp built a local `myclip` library from those files and called the removed clip/llava C API, so the bump no longer builds. ik_llama keeps its multimodal stack in the surviving `mtmd` library (examples/mtmd/, public headers mtmd.h + mtmd-helper.h). This ports the backend's multimodal path onto the high-level mtmd_* / mtmd_helper_* API in place, leaving the text path (which still uses ik_llama's retained old common API) untouched: - Makefile: bump IK_LLAMA_VERSION to f96eaddb. - prepare.sh: drop the clip/llava source copy + sed block; mtmd is a library target, no source copy needed. - CMakeLists.txt: remove the `myclip` target; link `mtmd` and add its include dir; build grpc-server as C++17 (mtmd headers require it). - patches: drop 0002 (targeted the deleted examples/llava/clip.cpp; the mtmd clip.cpp never calls ggml_quantize_chunk, so the fix is unneeded). Keep 0001 (verified still applies). - grpc-server.cpp / utils.hpp: replace clip_model_load + clip_image_load_from_bytes + llava_image_embed_make_with_clip_img + the manual [img-N] prefix splitting and per-image llava_embd_batch decode loop with mtmd_init_from_file (moved after the model load, which it requires), mtmd_helper_bitmap_init_from_buf, mtmd_tokenize and mtmd_helper_eval_chunks. Legacy [img-N] tags are translated, in order, into mtmd media markers (mtmd_default_marker()); the post-image suffix text stays on the normal token path so the sampling loop is unchanged. Supersedes #10534. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(ik-llama): align json alias to ordered_json to resolve mtmd.h conflict (#10534) mtmd.h declares `using json = nlohmann::ordered_json` at global scope (and its mtmd.cpp depends on it), while ik_llama's whole server/common stack also uses ordered_json. Our grpc-server.cpp/utils.hpp kept a plain `nlohmann::json` alias, which now collides with mtmd.h once it is included for the multimodal port: "conflicting declaration 'using json = ...'". Switch our two aliases to ordered_json to match; it is API-compatible (utils.hpp already used ordered_json for its log helper) and our json never crosses into an unordered-json API. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- backend/cpp/ik-llama-cpp/CMakeLists.txt | 23 +- backend/cpp/ik-llama-cpp/Makefile | 2 +- backend/cpp/ik-llama-cpp/grpc-server.cpp | 316 +++++++++--------- ...2-clip-ggml-quantize-chunk-user-data.patch | 11 - backend/cpp/ik-llama-cpp/prepare.sh | 25 +- backend/cpp/ik-llama-cpp/utils.hpp | 20 +- 6 files changed, 182 insertions(+), 215 deletions(-) delete mode 100644 backend/cpp/ik-llama-cpp/patches/0002-clip-ggml-quantize-chunk-user-data.patch diff --git a/backend/cpp/ik-llama-cpp/CMakeLists.txt b/backend/cpp/ik-llama-cpp/CMakeLists.txt index 545dc59db..c0157a0c6 100644 --- a/backend/cpp/ik-llama-cpp/CMakeLists.txt +++ b/backend/cpp/ik-llama-cpp/CMakeLists.txt @@ -1,15 +1,6 @@ -## Clip/LLaVA library for multimodal support — built locally from copied sources -set(TARGET myclip) -add_library(${TARGET} clip.cpp clip.h llava.cpp llava.h) -install(TARGETS ${TARGET} LIBRARY) -target_include_directories(myclip PUBLIC .) -target_include_directories(myclip PUBLIC ../..) -target_include_directories(myclip PUBLIC ../../common) -target_link_libraries(${TARGET} PRIVATE common ggml llama ${CMAKE_THREAD_LIBS_INIT}) -target_compile_features(${TARGET} PRIVATE cxx_std_11) -if (NOT MSVC) - target_compile_options(${TARGET} PRIVATE -Wno-cast-qual) -endif() +## Multimodal support is provided by the in-tree `mtmd` library target +## (examples/mtmd/), which the grpc-server links and includes below. clip/llava +## were pruned upstream; the high-level mtmd_* / mtmd_helper_* API is used instead. set(TARGET grpc-server) set(CMAKE_CXX_STANDARD 17) @@ -67,12 +58,16 @@ add_library(hw_grpc_proto ${hw_proto_hdrs} ) add_executable(${TARGET} grpc-server.cpp json.hpp) -target_link_libraries(${TARGET} PRIVATE common llama myclip ${CMAKE_THREAD_LIBS_INIT} absl::flags hw_grpc_proto +# mtmd public headers (mtmd.h / mtmd-helper.h) live in examples/mtmd/. +# Linking the mtmd target also propagates this include dir, but we add it +# explicitly for clarity. +target_include_directories(${TARGET} PRIVATE ../mtmd) +target_link_libraries(${TARGET} PRIVATE common llama mtmd ${CMAKE_THREAD_LIBS_INIT} absl::flags hw_grpc_proto absl::flags_parse gRPC::${_REFLECTION} gRPC::${_GRPC_GRPCPP} protobuf::${_PROTOBUF_LIBPROTOBUF}) -target_compile_features(${TARGET} PRIVATE cxx_std_11) +target_compile_features(${TARGET} PRIVATE cxx_std_17) if(TARGET BUILD_INFO) add_dependencies(${TARGET} BUILD_INFO) endif() diff --git a/backend/cpp/ik-llama-cpp/Makefile b/backend/cpp/ik-llama-cpp/Makefile index d76a07854..ef261a0a6 100644 --- a/backend/cpp/ik-llama-cpp/Makefile +++ b/backend/cpp/ik-llama-cpp/Makefile @@ -1,5 +1,5 @@ -IK_LLAMA_VERSION?=b84902d2ad27c34f989f23947200c4b91b1568fd +IK_LLAMA_VERSION?=f96eaddba8bed6a9a5e628bbf6a566775c70b49c LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp CMAKE_ARGS?= diff --git a/backend/cpp/ik-llama-cpp/grpc-server.cpp b/backend/cpp/ik-llama-cpp/grpc-server.cpp index ff1408630..96eea3b58 100644 --- a/backend/cpp/ik-llama-cpp/grpc-server.cpp +++ b/backend/cpp/ik-llama-cpp/grpc-server.cpp @@ -11,8 +11,8 @@ #include #include #include -#include "clip.h" -#include "llava.h" +#include "mtmd.h" +#include "mtmd-helper.h" #include "log.h" #include "common.h" #include "json.hpp" @@ -45,7 +45,9 @@ using backend::HealthMessage; ///// LLAMA.CPP server code below -using json = nlohmann::json; +// Match mtmd.h and ik_llama's server/common headers, which all use +// nlohmann::ordered_json; a plain nlohmann::json alias collides at global scope. +using json = nlohmann::ordered_json; struct server_params { @@ -219,6 +221,11 @@ struct llama_client_slot // multimodal std::vector images; + // Full prompt with mtmd media markers (mtmd_default_marker()) substituted in + // place of the legacy [img-N] tags, covering the text up to and including the + // last image. The text after the last image is kept in params.input_suffix and + // decoded through the normal token path so the sampling loop is unchanged. + std::string mtmd_prompt; // stats size_t sent_count = 0; @@ -252,14 +259,14 @@ struct llama_client_slot for (slot_image & img : images) { - free(img.image_embedding); - if (img.img_data) { - clip_image_u8_free(img.img_data); + if (img.bitmap) { + mtmd_bitmap_free(img.bitmap); + img.bitmap = nullptr; } - img.prefix_prompt = ""; } images.clear(); + mtmd_prompt = ""; } bool has_budget(gpt_params &global_params) { @@ -396,46 +403,13 @@ struct llama_metrics { } }; -struct llava_embd_batch { - std::vector pos; - std::vector n_seq_id; - std::vector seq_id_0; - std::vector seq_ids; - std::vector logits; - llama_batch batch; - llava_embd_batch(float * embd, int32_t n_tokens, llama_pos pos_0, llama_seq_id seq_id) { - pos .resize(n_tokens); - n_seq_id.resize(n_tokens); - seq_ids .resize(n_tokens + 1); - logits .resize(n_tokens); - seq_id_0.resize(1); - seq_id_0[0] = seq_id; - seq_ids [n_tokens] = nullptr; - batch = { - /*n_tokens =*/ n_tokens, - /*tokens =*/ nullptr, - /*embd =*/ embd, - /*pos =*/ pos.data(), - /*n_seq_id =*/ n_seq_id.data(), - /*seq_id =*/ seq_ids.data(), - /*logits =*/ logits.data(), - }; - for (int i = 0; i < n_tokens; i++) { - batch.pos [i] = pos_0 + i; - batch.n_seq_id[i] = 1; - batch.seq_id [i] = seq_id_0.data(); - batch.logits [i] = false; - } - } -}; - struct llama_server_context { llama_model *model = nullptr; llama_context *ctx = nullptr; const llama_vocab * vocab = nullptr; - clip_ctx *clp_ctx = nullptr; + mtmd_context *mctx = nullptr; gpt_params params; @@ -491,11 +465,6 @@ struct llama_server_context if (!params.mmproj.path.empty()) { multimodal = true; LOG_INFO("Multi Modal Mode Enabled", {}); - clp_ctx = clip_model_load(params.mmproj.path.c_str(), /*verbosity=*/ 1); - if(clp_ctx == nullptr) { - LOG_ERR("unable to load clip model: %s", params.mmproj.path.c_str()); - return false; - } if (params.n_ctx < 2048) { // request larger context for the image embedding params.n_ctx = 2048; @@ -512,10 +481,24 @@ struct llama_server_context } if (multimodal) { - const int n_embd_clip = clip_n_mmproj_embd(clp_ctx); - const int n_embd_llm = llama_model_n_embd(model); - if (n_embd_clip != n_embd_llm) { - LOG("%s: embedding dim of the multimodal projector (%d) is not equal to that of LLaMA (%d). Make sure that you use the correct mmproj file.\n", __func__, n_embd_clip, n_embd_llm); + // mtmd_init_from_file requires the already-loaded text model, so it must + // run AFTER llama_init_from_gpt_params. It validates the projector + // against the model internally and returns nullptr on dim mismatch, so + // the explicit clip_n_mmproj_embd check is no longer needed. + mtmd_context_params mparams = mtmd_context_params_default(); + mparams.use_gpu = params.mmproj_use_gpu; + mparams.print_timings = false; + mparams.n_threads = params.n_threads_mtmd != -1 ? params.n_threads_mtmd + : params.n_threads_batch != -1 ? params.n_threads_batch + : params.n_threads; + mparams.verbosity = GGML_LOG_LEVEL_INFO; + mparams.flash_attn_type = params.flash_attn ? LLAMA_FLASH_ATTN_TYPE_ENABLED + : LLAMA_FLASH_ATTN_TYPE_DISABLED; + mparams.image_min_tokens = params.image_min_tokens; + mparams.image_max_tokens = params.image_max_tokens; + mctx = mtmd_init_from_file(params.mmproj.path.c_str(), model, mparams); + if (mctx == nullptr) { + LOG_ERR("unable to load multimodal projector: %s", params.mmproj.path.c_str()); llama_free(ctx); llama_free_model(model); return false; @@ -865,8 +848,8 @@ struct llama_server_context slot_image img_sl; img_sl.id = img.count("id") != 0 ? img["id"].get() : slot->images.size(); - img_sl.img_data = clip_image_u8_init(); - if (!clip_image_load_from_bytes(image_buffer.data(), image_buffer.size(), img_sl.img_data)) + img_sl.bitmap = mtmd_helper_bitmap_init_from_buf(mctx, image_buffer.data(), image_buffer.size()); + if (img_sl.bitmap == nullptr) { LOG_ERR("%s: failed to load image, slot_id: %d, img_sl_id: %d", __func__, @@ -879,50 +862,74 @@ struct llama_server_context {"slot_id", slot->id}, {"img_sl_id", img_sl.id} }); - img_sl.request_encode_image = true; slot->images.push_back(img_sl); } - // process prompt - // example: system prompt [img-102] user [img-103] describe [img-134] -> [{id: 102, prefix: 'system prompt '}, {id: 103, prefix: ' user '}, {id: 134, prefix: ' describe '}]} + // Translate the legacy [img-N] tags into mtmd media markers, in + // order, and collect the matching bitmaps in marker order so they + // line up with the markers passed to mtmd_tokenize(). The text after + // the last image stays in input_suffix and is decoded through the + // normal token path, so the sampling loop is unchanged. + // example: system prompt [img-102] user [img-103] describe [img-134] if (slot->images.size() > 0 && !slot->prompt.is_array()) { + const std::string marker = mtmd_default_marker(); std::string prompt = slot->prompt.get(); - size_t pos = 0, begin_prefix = 0; + std::string built_prompt; + std::vector ordered; + size_t pos = 0, copy_from = 0; std::string pattern = "[img-"; - while ((pos = prompt.find(pattern, pos)) != std::string::npos) { - size_t end_prefix = pos; - pos += pattern.length(); - size_t end_pos = prompt.find(']', pos); - if (end_pos != std::string::npos) - { - std::string image_id = prompt.substr(pos, end_pos - pos); - try - { - int img_id = std::stoi(image_id); - bool found = false; - for (slot_image &img : slot->images) - { - if (img.id == img_id) { - found = true; - img.prefix_prompt = prompt.substr(begin_prefix, end_prefix - begin_prefix); - begin_prefix = end_pos + 1; - break; - } - } - if (!found) { - LOG("ERROR: Image with id: %i, not found.\n", img_id); - slot->images.clear(); - return false; - } - } catch (const std::invalid_argument& e) { - LOG("Invalid image number id in prompt\n"); - slot->images.clear(); - return false; + + auto free_images = [&]() { + for (slot_image &img : slot->images) { + if (img.bitmap) { + mtmd_bitmap_free(img.bitmap); + img.bitmap = nullptr; } } + slot->images.clear(); + }; + + while ((pos = prompt.find(pattern, pos)) != std::string::npos) { + size_t tag_begin = pos; + pos += pattern.length(); + size_t end_pos = prompt.find(']', pos); + if (end_pos == std::string::npos) { + break; + } + std::string image_id = prompt.substr(pos, end_pos - pos); + try + { + int img_id = std::stoi(image_id); + bool found = false; + for (slot_image &img : slot->images) + { + if (img.id == img_id) { + found = true; + // text before this tag, then the media marker + built_prompt += prompt.substr(copy_from, tag_begin - copy_from); + built_prompt += marker; + copy_from = end_pos + 1; + ordered.push_back(img); + break; + } + } + if (!found) { + LOG("ERROR: Image with id: %i, not found.\n", img_id); + free_images(); + return false; + } + } catch (const std::invalid_argument& e) { + LOG("Invalid image number id in prompt\n"); + free_images(); + return false; + } + pos = end_pos + 1; } + // bitmaps are consumed in marker order by mtmd_tokenize() + slot->images = ordered; + slot->mtmd_prompt = built_prompt; slot->prompt = ""; - slot->params.input_suffix = prompt.substr(begin_prefix); + slot->params.input_suffix = prompt.substr(copy_from); slot->params.cache_prompt = false; // multimodal doesn't support cache prompt } } @@ -1176,21 +1183,10 @@ struct llama_server_context bool process_images(llama_client_slot &slot) const { - for (slot_image &img : slot.images) - { - if (!img.request_encode_image) - { - continue; - } - - if (!llava_image_embed_make_with_clip_img(clp_ctx, params.n_threads, img.img_data, &img.image_embedding, &img.image_tokens)) { - LOG("Error processing the given image"); - return false; - } - - img.request_encode_image = false; - } - + // With the mtmd pipeline, image encoding is no longer eager: the bitmaps + // are tokenized and encoded together with the surrounding text inside + // ingest_images() via mtmd_tokenize() + mtmd_helper_eval_chunks(). This + // just reports whether the slot carries any images to process. return slot.images.size() > 0; } @@ -1435,69 +1431,70 @@ struct llama_server_context } } - // for multiple images processing + // Tokenize the multimodal prompt (text interleaved with media markers) together + // with the slot's bitmaps, then decode the resulting chunks into the llama + // context via the high-level mtmd helper. The helper runs llama_decode() on the + // text chunks and mtmd_encode() + llama_decode() on the image chunks, handling + // batching and any pre/post decode setup (e.g. non-causal attention for gemma3). + // Advances slot.n_past by the number of positions consumed, then leaves the + // post-image suffix tokens in `batch` so the normal decode + sampling loop + // produces the first generated token. bool ingest_images(llama_client_slot &slot, int n_batch) { - int image_idx = 0; - - while (image_idx < (int) slot.images.size()) + if (mctx == nullptr) { - slot_image &img = slot.images[image_idx]; + LOG("%s : multimodal context is not initialized\n", __func__); + return false; + } - // process prefix prompt - for (int32_t i = 0; i < (int32_t) batch.n_tokens; i += n_batch) - { - const int32_t n_tokens = std::min(n_batch, (int32_t) (batch.n_tokens - i)); - llama_batch batch_view = { - n_tokens, - batch.token + i, - nullptr, - batch.pos + i, - batch.n_seq_id + i, - batch.seq_id + i, - batch.logits + i, - }; - if (llama_decode(ctx, batch_view)) - { - LOG("%s : failed to eval\n", __func__); - return false; - } - } + // bitmaps stay owned by slot.images (freed on reset()); pass non-owning ptrs + std::vector bitmaps; + bitmaps.reserve(slot.images.size()); + for (const slot_image &img : slot.images) + { + bitmaps.push_back(img.bitmap); + } - // process image with llm - for (int i = 0; i < img.image_tokens; i += n_batch) - { - int n_eval = img.image_tokens - i; - if (n_eval > n_batch) - { - n_eval = n_batch; - } + mtmd_input_text inp_txt; + inp_txt.text = slot.mtmd_prompt.c_str(); + inp_txt.add_special = add_bos_token; + inp_txt.parse_special = true; - const int n_embd = llama_model_n_embd(model); - float * embd = img.image_embedding + i * n_embd; - llava_embd_batch llava_batch = llava_embd_batch(embd, n_eval, slot.n_past, 0); - if (llama_decode(ctx, llava_batch.batch)) - { - LOG("%s : failed to eval image\n", __func__); - return false; - } - slot.n_past += n_eval; - } - image_idx++; + mtmd::input_chunks chunks(mtmd_input_chunks_init()); + int32_t res = mtmd_tokenize(mctx, + chunks.ptr.get(), + &inp_txt, + bitmaps.data(), + bitmaps.size()); + if (res != 0) + { + LOG("%s : failed to tokenize multimodal prompt, res = %d\n", __func__, res); + return false; + } - common_batch_clear(batch); + const llama_pos start_pos = (llama_pos) system_tokens.size() + slot.n_past; + llama_pos new_n_past = start_pos; + if (mtmd_helper_eval_chunks(mctx, + ctx, + chunks.ptr.get(), + start_pos, + slot.id, + n_batch, + /*logits_last=*/ false, + &new_n_past) != 0) + { + LOG("%s : failed to eval multimodal chunks\n", __func__); + return false; + } + slot.n_past += (int32_t) (new_n_past - start_pos); - // append prefix of next image - const auto json_prompt = (image_idx >= (int) slot.images.size()) ? - slot.params.input_suffix : // no more images, then process suffix prompt - (json)(slot.images[image_idx].prefix_prompt); - - std::vector append_tokens = tokenize(json_prompt, false); // has next image - for (int i = 0; i < (int) append_tokens.size(); ++i) - { - common_batch_add(batch, append_tokens[i], system_tokens.size() + slot.n_past, { slot.id }, true); - slot.n_past += 1; - } + // queue the post-image suffix text for the normal decode + sampling path + common_batch_clear(batch); + std::vector suffix_tokens = tokenize(slot.params.input_suffix, false); + for (llama_token tok : suffix_tokens) + { + common_batch_add(batch, tok, system_tokens.size() + slot.n_past, { slot.id }, false); + slot.n_past += 1; } return true; @@ -1884,8 +1881,11 @@ struct llama_server_context const bool has_images = process_images(slot); - // process the prefix of first image - std::vector prefix_tokens = has_images ? tokenize(slot.images[0].prefix_prompt, add_bos_token) : prompt_tokens; + // For the multimodal path the whole pre-image / inter-image text is + // tokenized and decoded inside ingest_images() via mtmd, so no prefix + // tokens are queued here; the post-image suffix is appended by + // ingest_images() for the normal decode + sampling loop. + std::vector prefix_tokens = has_images ? std::vector() : prompt_tokens; int32_t slot_npast = slot.n_past_se > 0 ? slot.n_past_se : slot.n_past; diff --git a/backend/cpp/ik-llama-cpp/patches/0002-clip-ggml-quantize-chunk-user-data.patch b/backend/cpp/ik-llama-cpp/patches/0002-clip-ggml-quantize-chunk-user-data.patch deleted file mode 100644 index 5724f4d06..000000000 --- a/backend/cpp/ik-llama-cpp/patches/0002-clip-ggml-quantize-chunk-user-data.patch +++ /dev/null @@ -1,11 +0,0 @@ ---- a/examples/llava/clip.cpp -+++ b/examples/llava/clip.cpp -@@ -2494,7 +2494,7 @@ - } - new_data = work.data(); - -- new_size = ggml_quantize_chunk(new_type, f32_data, new_data, 0, n_elms/cur->ne[0], cur->ne[0], nullptr); -+ new_size = ggml_quantize_chunk(new_type, f32_data, new_data, 0, n_elms/cur->ne[0], cur->ne[0], nullptr, nullptr); - } else { - new_type = cur->type; - new_data = cur->data; diff --git a/backend/cpp/ik-llama-cpp/prepare.sh b/backend/cpp/ik-llama-cpp/prepare.sh index fb0ba7624..b6c03c0f9 100644 --- a/backend/cpp/ik-llama-cpp/prepare.sh +++ b/backend/cpp/ik-llama-cpp/prepare.sh @@ -17,28 +17,9 @@ cp -r grpc-server.cpp llama.cpp/examples/grpc-server/ cp -r utils.hpp llama.cpp/examples/grpc-server/ cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/examples/grpc-server/ -## Copy clip/llava files for multimodal support (built as myclip library) -cp -rfv llama.cpp/examples/llava/clip.h llama.cpp/examples/grpc-server/clip.h -cp -rfv llama.cpp/examples/llava/clip.cpp llama.cpp/examples/grpc-server/clip.cpp -cp -rfv llama.cpp/examples/llava/llava.cpp llama.cpp/examples/grpc-server/llava.cpp -# Prepend llama.h include to llava.h -echo '#include "llama.h"' > llama.cpp/examples/grpc-server/llava.h -cat llama.cpp/examples/llava/llava.h >> llama.cpp/examples/grpc-server/llava.h -# Copy clip-impl.h if it exists -if [ -f llama.cpp/examples/llava/clip-impl.h ]; then - cp -rfv llama.cpp/examples/llava/clip-impl.h llama.cpp/examples/grpc-server/clip-impl.h -fi -# Copy stb_image.h -if [ -f llama.cpp/vendor/stb/stb_image.h ]; then - cp -rfv llama.cpp/vendor/stb/stb_image.h llama.cpp/examples/grpc-server/stb_image.h -elif [ -f llama.cpp/common/stb_image.h ]; then - cp -rfv llama.cpp/common/stb_image.h llama.cpp/examples/grpc-server/stb_image.h -fi - -## Fix API compatibility in llava.cpp (llama_n_embd -> llama_model_n_embd) -if [ -f llama.cpp/examples/grpc-server/llava.cpp ]; then - sed -i 's/llama_n_embd(/llama_model_n_embd(/g' llama.cpp/examples/grpc-server/llava.cpp -fi +## Multimodal support is provided by the `mtmd` library target (examples/mtmd/), +## which the grpc-server links and includes directly. No source copy is needed: +## clip/llava were pruned upstream and the high-level mtmd_* API is used instead. set +e if grep -q "grpc-server" llama.cpp/examples/CMakeLists.txt; then diff --git a/backend/cpp/ik-llama-cpp/utils.hpp b/backend/cpp/ik-llama-cpp/utils.hpp index e5cf2a009..f0e599212 100644 --- a/backend/cpp/ik-llama-cpp/utils.hpp +++ b/backend/cpp/ik-llama-cpp/utils.hpp @@ -11,9 +11,12 @@ #include "json.hpp" -#include "clip.h" +#include "mtmd.h" -using json = nlohmann::json; +// mtmd.h and ik_llama's entire server/common stack (chat.h, server-common.h, +// server-task.h, ...) declare `using json = nlohmann::ordered_json`, so match it +// here: a plain `nlohmann::json` alias collides with mtmd.h's at global scope. +using json = nlohmann::ordered_json; extern bool server_verbose; @@ -111,13 +114,12 @@ struct slot_image { int32_t id; - bool request_encode_image = false; - float * image_embedding = nullptr; - int32_t image_tokens = 0; - - clip_image_u8 * img_data; - - std::string prefix_prompt; // before of this image + // mtmd bitmap (image/audio) decoded from the request buffer. Owned by the + // slot; freed via mtmd_bitmap_free() on reset. The high-level mtmd pipeline + // (mtmd_tokenize + mtmd_helper_eval_chunks) consumes these directly, so the + // legacy eager-encode fields (embedding/tokens) and per-image prefix prompt + // are no longer needed. + mtmd_bitmap * bitmap = nullptr; }; // completion token output with probabilities From de2ec2f1368e7da584a7ff4e8d0211405b6e9ee3 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Sun, 28 Jun 2026 09:29:08 +0200 Subject: [PATCH 17/17] feat(backends): add voice-detect + face-detect ggml backends (replace Python insightface/speaker-recognition) (#10441) * feat(voice-detect): add Go purego backend for voice-detect.cpp Add backend/go/voice-detect implementing the Backend gRPC voice subset (VoiceEmbed/VoiceVerify/VoiceAnalyze) over libvoicedetect.so via purego, mirroring the parakeet-cpp / omnivoice-cpp backends. The flat voicedetect_capi C ABI is dlopen'd cgo-less; malloc'd string and float-vector returns are owned by Go and released through the matching capi free functions, with the per-ctx last error surfaced into Go errors. Calls are serialized via base.SingleThread since the C context is not reentrant. Proto field mapping: - VoiceEmbed: VoiceEmbedRequest.audio (path) -> embed_path -> Embedding+Model. - VoiceVerify: audio1/audio2 + threshold (<=0 falls back to the verify_threshold option, default 0.25) -> verify_paths -> verified/distance/ threshold/confidence/model/processing_time_ms. - VoiceAnalyze: audio (path) -> analyze_path_json; the JSON age/gender/emotion document maps to a single VoiceAnalysis segment (start/end 0; gender "label" -> dominant_gender with the remaining float scores as the gender map; emotion label/scores -> dominant_emotion/emotion). The Makefile pins voice-detect.cpp to 47546430, clones+builds libvoicedetect.so with ggml static-linked (PIC, GGML_NATIVE off) so dlopen needs no external libggml/libvoicedetect; ldd on the artifact shows only system libs. Ginkgo tests cover option parsing and analyze-JSON mapping; embed/verify smoke specs gate on VOICEDETECT_BACKEND_TEST_MODEL + VOICEDETECT_BACKEND_TEST_WAV. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(voice-detect): wire backend into index, gallery and build Register the voice-detect.cpp speaker-recognition + voice-analysis backend (added in Voice-INT-A) into LocalAI's distribution surfaces, mirroring the ced backend (the closest mudler C++/ggml audio analogue): - backend/index.yaml: add the &voicedetect meta-backend (capabilities platform map, no top-level uri) plus the full set of concrete per-arch image entries (cpu/cuda12/cuda13/metal/rocm/sycl/vulkan/l4t and the -development variants). Referential integrity audited - every alias target resolves. - gallery/index.yaml: add 5 model entries on backend voice-detect - ECAPA-TDNN, WeSpeaker ResNet34, 3D-Speaker ERes2Net, CAM++ and the wav2vec2 age/gender/emotion analyze model. The engine architecture is read from GGUF metadata (voicedetect.arch) at load. GGUF artifacts are not yet published: each files: entry points at the intended mudler/voice-detect-gguf location with a TODO to fill sha256 after upload (no fabricated hashes). - .github/backend-matrix.yml: add the linux build matrix block + the darwin metal entry mirroring ced. - .github/workflows/bump_deps.yaml: track mudler/voice-detect.cpp via VOICEDETECT_VERSION (pin 47546430, = 4754643). - core/config/backend_capabilities.go: register voice-detect in the backend capability map (VoiceVerify/VoiceEmbed/VoiceAnalyze -> speaker_recognition), mirroring speaker-recognition. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(face-detect): add purego Go backend for face-detect.cpp Add the LocalAI Go backend that dlopens libfacedetect.so (the flat facedetect_capi_* C-ABI) via purego, mirroring the sibling voice-detect backend. Implements the Face subset of the Backend gRPC service: - Embeddings(PredictOptions): Images[0] base64 -> temp file -> embed_path -> L2-normalized ArcFace embedding. - Detect(DetectOptions): src -> detect_path_json -> Detection boxes (class_name "face", [x1,y1,x2,y2] -> x/y/w/h). - FaceVerify(FaceVerifyRequest): two images + threshold + anti_spoof -> verify_paths; best-effort img areas via detect. - FaceAnalyze(FaceAnalyzeRequest): img -> analyze_path_json -> per-face age + gender ("M"/"F" normalized to "Man"/"Woman"). The Makefile pins face-detect.cpp to 636a1963 and builds the shared lib with ggml + vendored libjpeg-turbo static (PIC), so the .so is ldd-clean (no libggml) and exports only facedetect_capi_* (no jpeg_ symbols). Gated Ginkgo e2e mirrors voice-detect. Note for the gallery-wiring task: backend registration (index.yaml, gallery, core/config/backend_capabilities.go) is intentionally not touched here. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(voice-detect): replace em dashes in net-new descriptions Project style forbids em/en dashes. Replace the three U+2014 chars introduced by the voice-detect gallery/index wiring with `-`/`:`. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(face-detect): wire backend into index, gallery and build Register the face-detect.cpp face detection / embedding / verification / analysis backend (added in Face-INT-A) into LocalAI's distribution surfaces, mirroring the voice-detect wiring (the closest mudler C++/ggml recognition analogue): - backend/index.yaml: add the &facedetect meta-backend (capabilities platform map, no top-level uri to avoid the meta-backend gotcha) plus the full set of concrete per-arch image entries (cpu/cuda12/cuda13/ metal/rocm/sycl-f16/sycl-f32/vulkan/l4t and the -development variants), 22 entries. Referential integrity audited: every alias target resolves. - gallery/index.yaml: add 4 model entries on backend face-detect - face-detect-buffalo-l/m/s (insightface SCRFD + ArcFace/MBF, NON-COMMERCIAL) and face-detect-yunet-sface (OpenCV-Zoo YuNet + SFace, APACHE-2.0, the commercial-friendly alternative). The detector/embedder architecture is read from GGUF metadata (facedetect.arch) at load; only the real verify_threshold option is set (0.35 buffalo, 0.363 sface). GGUF artifacts are not yet published: each files: entry points at the intended mudler/face-detect-gguf location with a TODO to fill sha256 after upload (no fabricated hashes). - core/config/backend_capabilities.go: register face-detect in the backend capability map (Embedding/Detect/FaceVerify/FaceAnalyze -> face_recognition), mirroring insightface. - .github/backend-matrix.yml: add the linux build matrix block + the darwin metal entry mirroring voice-detect. - .github/workflows/bump_deps.yaml: track mudler/face-detect.cpp via FACEDETECT_VERSION (pin 636a1963). Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(recon): voice-detect metal build branch + face-detect gallery usecases Add the missing metal BUILD_TYPE branch to the voice-detect Makefile forwarding -DVOICEDETECT_GGML_METAL=ON, mirroring face-detect, so the darwin metal CI artifact is built with the Metal backend instead of CPU-only. Expand the 4 face-detect gallery models' known_usecases to [face_recognition, detection, embeddings] to match the backend capabilities map and the mirrored insightface-buffalo entries, so auto-selection for /v1/detect and /embeddings works. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * docs(recon): document voice-detect and face-detect ggml backends Document the new standalone C++/ggml biometric backends as the recommended/default option for face and voice recognition, keeping the existing Python insightface / speaker-recognition backends framed as the legacy path. - features/face-recognition.md: add a face-detect (ggml) backend section with the gallery entries (buffalo-l/m/s non-commercial, yunet-sface Apache-2.0), licensing, and verify/detect/analyze quickstart. - features/voice-recognition.md: add a voice-detect (ggml) backend section with the gallery entries (ecapa-tdnn, wespeaker-resnet34, eres2net, campplus speaker recognizers; emotion-wav2vec2 non-commercial analyze head) and quickstart. - reference/compatibility-table.md: add face-detect.cpp and voice-detect.cpp rows to the Vision, Detection & Recognition table. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(gallery): publish recon backend GGUF uris + sha256 Fill in the published HuggingFace GGUF uris and verified sha256 for the 9 recon gallery entries (voice-detect-* and face-detect-*), and remove the TODO publish markers. Correct the eres2net, campplus, and emotion-wav2vec2 uris to the actual published filenames. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(gallery): re-embed buffalo anti-spoof + add audeering age/gender voice model Update the 3 buffalo face-detect GGUF sha256 (anti-spoof ensemble now embedded and re-uploaded under the same filenames/uris) and note the FaceVerify anti_spoof request flag in each description. Add a new voice-detect-age-gender-wav2vec2 gallery entry mirroring the emotion model. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(gallery): add face-detect-buffalo-sc and antelopev2 packs Add gallery entries for two newly-published insightface face packs on the face-detect backend: buffalo_sc (smallest pack, SCRFD-500M + small ArcFace) and antelopev2 (higher-accuracy, SCRFD-10G + ArcFace glint360k R100, 512-d). Both are non-commercial research-only. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(recon): honor LocalAI per-model threads in voice/face-detect backends LocalAI spawns one backend process per model and serves requests concurrently, so the engines' own min(hardware_concurrency, 8) default can oversubscribe cores. Forward the per-model Threads value from the gRPC LoadModel options into the engine via VOICEDETECT_THREADS / FACEDETECT_THREADS (read at backend construction) before the capi load. A non-positive Threads is treated as unset, leaving the engine default. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump backend pins to CPU-optimized engine commits voice-detect.cpp -> 0d9c1b3 (radix-2 FFT FBank, threads, flash attn + cached pos-conv); face-detect.cpp -> 523aee1 (thread-gated direct conv, threads). Brings the CPU optimizations into the LocalAI backend builds. GGUF format and parity unchanged, so the published HF GGUFs remain valid. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump backend pins to round-2 CPU-optimized engines voice-detect.cpp -> fe7e6a3 (ERes2Net 1x1->mul_mat, CAM++ layout+context, wav2vec2 conv-LN, ECAPA capture-drop, AVX512 dispatch opt-in); face-detect.cpp -> 9c8adb7 (AVX2 Winograd F(2x2,3x3) for SCRFD/ArcFace 3x3 convs, ArcFace BN-fold). Parity unchanged (cosine=1.0); GGUF format unchanged, HF GGUFs valid. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump backend pins to round-3 Winograd engines voice-detect.cpp -> 45122ec (Winograd F(2x2,3x3) for WeSpeaker/ERes2Net 3x3 convs, -22%/-20% @8t); face-detect.cpp -> cd5c962 (Winograd F(4x4,3x3) for SCRFD large maps, -22% @1t on top of F(2x2), more load-stable). Parity held (cosine=1.0); GGUF format unchanged, HF GGUFs valid. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump backend pins to round-4 Winograd engines (CPU opt complete) voice-detect.cpp -> d2839ca (CAM++ FCM 2D convs through Winograd, -15.5%/-10.3%); face-detect.cpp -> c1db23d (AVX2-vectorized Winograd tile transforms, SCRFD detect -14%/-9.6%). Final CPU optimization round; the conv-kernel lever class is now exhausted (parity held cosine=1.0; GGUF/parity unchanged, HF GGUFs valid). Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump face-detect pin to deep-kernel engine (7ae5c4d) face-detect.cpp -> 7ae5c4d: register-blocked winograd-domain GEMM microkernel (2.8x isolated GFLOP/s), AVX-512 zmm evolution behind runtime CPUID dispatch (ship-safe, AVX2 fallback bit-identical), bias/relu fused into the winograd output transform, and SFace Conv+BN fold + bias/PReLU fusion. SCRFD detect ~1.4x faster end-to-end vs the round-4 baseline; parity bit-exact; portable single binary (function-multiversioned, no global -mavx512f). Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump voice-detect pin to ECAPA operand-order win (e9c56ae) voice-detect.cpp -> e9c56ae: weight-as-src0 mul_mat order in ECAPA's F32 conv1d_same (routes through tinyBLAS sgemm); ECAPA embed 1.67x @1t / ~1.3x @8t, parity cosine=1.0. Isolated to encoder.cpp (ECAPA-only); ERes2Net/CAM++/WeSpeaker do not call conv1d_same so are provably unaffected. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to FMA-throughput engines (voice f7b9f89, face 2d2d5f0) face -> 2d2d5f0: route ArcFace 3x3 body convs through the AVX-512 winograd microkernel (kWinoMinSize 80->14); ArcFace 1.62x @1t, SCRFD detect to 0.966 of MLAS @1t, no regression. voice -> f7b9f89: runtime-CPUID-dispatched AVX-512 winograd-GEMM microkernel (ship-safe, AVX2 fallback bit-identical); WeSpeaker 1.90x @1t. Parity cosine=1.0 throughout; portable single binaries. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to MLAS-class direct-conv engines (voice 7ecfd07, face be22d67) Hand-tuned nChw16c AVX-512 register-tiled direct-conv microkernel (~263 GFLOP/s, within 6-7% of MLAS per-op efficiency), runtime-CPUID-dispatched + AVX2 fallback, fused bias/relu. voice 7ecfd07: default 3x3-s1 kernel for WeSpeaker (+37%/+32%) + ERes2Net, CAM++ pinned to Winograd. face be22d67: shape-gated to the ArcFace recognizer body (+25-27% @8t); SCRFD detector stays on Winograd (no regression). Parity cosine=1.0 / detect <=1px on AVX-512 + AVX2 paths. Portable single binaries. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump voice pin to Phase-A blocked backbone (f4e7eef) WeSpeaker ResNet34 runs as one nChw16c blocked island (2 reorders/forward vs ~60) on AVX-512, default; per-conv directconv fallback on AVX2. +2.9% @1t / +17-19% @8t vs per-conv directconv, parity cosine=1.0. The conv microkernel is already FMA-bound near peak (~0.86-0.98x MLAS-implied); residual to MLAS is sub-peak edge + non-conv tail, documented in docs/cpu-optimization.md. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to breadth blocked-backbone (voice 7f66871, face d80092b) voice 7f66871: AVX2-vectorized (ymm) blocked island - AVX2-only hosts now run the blocked backbone for WeSpeaker (2.3x over per-conv-AVX2, cosine=1.0); ERes2Net stays per-conv (blocked regresses, opt-in only); CAM++ Winograd-pinned. face d80092b: ArcFace recognizer blocked island, AVX-512 default (-13% @8t, ~0.90x MLAS, the closest conv result), auto per-conv on AVX2; SCRFD untouched on Winograd (0 island invocations during detect). Parity cosine=1.0 / detect <=1px throughout. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to small-spatial + stem conv kernels (voice 99b1804, face 47fdab6) Measured-gap-driven conv kernels: small-spatial (fill the register tile when output width <= tile width) + small-IC stem + strided-1x1/downsample recovery. ArcFace recognizer 0.57 -> 0.70x MLAS @1t (the closest conv model), WeSpeaker 0.65 -> 0.79x @1t. Parity cosine=1.0 / detect <=1px. The OC-block-sharing lever was a measured dead-end (deep stride-1 is L3-weight-bandwidth bound, not read-port bound) and was NOT shipped. Kernel ceiling reached; further gap needs an algorithm-class change (cache-blocked weight-stationary GEMM, or q8 weights). Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to GPU persistent-graph + multi-model-safe cache (voice 45d2e6b, face 0a4799a) GPU wins (CUDA/ggml backend, no CPU-path change): persistent per-shape graph+context cache in Backend::compute() eliminates the per-call cudaGraph re-instantiation churn -> wav2vec2 emotion+age-gender now AT GPU parity with torch-cuDNN on GB10 (0.97-0.98x), CAM++ -5.7ms; bit-identical parity. Cache hardened multi-model-safe (invalidate-on-free keyed by the ModelLoader weights buffer) so LocalAI multi-model hosting cannot stale-hit. Conv models still trail cuDNN (im2col-materialization-bound) - cuDNN implicit-GEMM lever next. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump pins to cuDNN-conv-capable engines (voice b6e4356, face 6107a24) Adds the opt-in cuDNN implicit-GEMM conv path (VOICEDETECT_GGML_CUDNN / FACEDETECT_GGML_CUDNN, DEFAULT OFF -> zero build/runtime dep until enabled). On GPU it kills the im2col-materialization bottleneck and reaches torch-cuDNN parity on the spill-bound convs: SCRFD detect 14.8->6.4ms (2.3x, ~parity), WeSpeaker ~parity, ERes2Net beats torch (1.10x); ArcFace/CAM++ neutral (no spill). Parity exact (SCRFD <=1px, cosine=1.0). To USE it in LocalAI, the CUDA backend build must enable the flag AND bundle libcudnn - deferred until a cuDNN-bundled GPU image; flag stays OFF here. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(recon): enable cuDNN conv path on arm64+CUDA13 recon backends The voice-detect.cpp / face-detect.cpp engines have an opt-in cuDNN implicit-GEMM conv path behind VOICEDETECT_GGML_CUDNN / FACEDETECT_GGML_CUDNN (default OFF) that kills im2col on the GPU and reaches torch-cuDNN parity (SCRFD 2.3x, WeSpeaker/ERes2Net parity), measured on the GB10 (arm64, CUDA 13, sm_121a). Enable it for the CUDA build, but only where cuDNN actually ships: the arm64 + CUDA 13 image (GB10/Jetson/L4T). x86 CUDA images carry no cuDNN, so flipping it on globally for BUILD_TYPE=cublas would be a link failure. The Makefiles gate on CUDA_MAJOR_VERSION=13 + arch (TARGETARCH from the matrix/Docker build, uname -m fallback for local builds). backend/Dockerfile.golang already installs the runtime libcudnn9-cuda-13 in the arm64+CUDA13 apt block; add the matching libcudnn9-dev-cuda-13 so the build-time link resolves. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): bump voice-detect pin to ERes2Net blocked-default (30beecd) Defaults VD_ERES2NET_BLOCKED ON: routes the ERes2Net Res2Net body through the blocked nChw16c AVX-512 directconv island instead of the 1x1 mul_mat fast path (CONT-transpose + skinny low-K GEMM). On the shipped GGML_NATIVE=OFF build (ggml mul_mat is AVX2-only) this wins ~2x at every thread count (2.07x@1t, 2.2x@4t, 2.05x@8t); pure-AVX2 fallback still 1.3-1.62x. Parity exact (cosine=1.000000 vs golden), so registered voices + verify/identify thresholds are unaffected. The prior default-OFF rested on a stale comment whose 23pct regression only held on the non-shipping GGML_NATIVE=ON build. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * docs(readme): announce native voice-detect + face-detect backends in Latest News Add a Latest News entry for the new from-scratch C++/ggml biometric backends (voice-detect.cpp + face-detect.cpp) that replace the Python insightface and speaker-recognition backends: no Python/onnxruntime at inference, self-contained GGUF, bit-exact parity, GPU cuDNN parity. Mirrors the parakeet.cpp / locate-anything.cpp native-backend news entries. Refs PR #10441. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(recon): re-pin to the squashed engine release commits The voice-detect.cpp and face-detect.cpp histories were squashed to a single release commit, which orphaned the previous pins (voice 30beecd, face 6107a24). Re-pin to the new single-commit SHAs (voice 3d51077, face 06914b0); the tree is identical, so the backend build is unchanged. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- .github/backend-matrix.yml | 304 ++++++++++++ .github/workflows/bump_deps.yaml | 8 + README.md | 1 + backend/Dockerfile.golang | 2 +- backend/go/face-detect/.gitignore | 18 + backend/go/face-detect/Makefile | 110 +++++ backend/go/face-detect/gofacedetect.go | 431 +++++++++++++++++ backend/go/face-detect/gofacedetect_test.go | 230 +++++++++ backend/go/face-detect/main.go | 65 +++ backend/go/face-detect/options.go | 47 ++ backend/go/face-detect/package.sh | 68 +++ backend/go/face-detect/run.sh | 16 + backend/go/face-detect/test.sh | 15 + backend/go/voice-detect/.gitignore | 18 + backend/go/voice-detect/Makefile | 107 +++++ backend/go/voice-detect/govoicedetect.go | 273 +++++++++++ backend/go/voice-detect/govoicedetect_test.go | 144 ++++++ backend/go/voice-detect/main.go | 64 +++ backend/go/voice-detect/options.go | 46 ++ backend/go/voice-detect/package.sh | 68 +++ backend/go/voice-detect/run.sh | 16 + backend/go/voice-detect/test.sh | 14 + backend/index.yaml | 302 ++++++++++++ core/config/backend_capabilities.go | 13 + docs/content/features/face-recognition.md | 93 +++- docs/content/features/voice-recognition.md | 90 +++- docs/content/reference/compatibility-table.md | 2 + gallery/index.yaml | 453 ++++++++++++++++++ 28 files changed, 3002 insertions(+), 16 deletions(-) create mode 100644 backend/go/face-detect/.gitignore create mode 100644 backend/go/face-detect/Makefile create mode 100644 backend/go/face-detect/gofacedetect.go create mode 100644 backend/go/face-detect/gofacedetect_test.go create mode 100644 backend/go/face-detect/main.go create mode 100644 backend/go/face-detect/options.go create mode 100644 backend/go/face-detect/package.sh create mode 100644 backend/go/face-detect/run.sh create mode 100644 backend/go/face-detect/test.sh create mode 100644 backend/go/voice-detect/.gitignore create mode 100644 backend/go/voice-detect/Makefile create mode 100644 backend/go/voice-detect/govoicedetect.go create mode 100644 backend/go/voice-detect/govoicedetect_test.go create mode 100644 backend/go/voice-detect/main.go create mode 100644 backend/go/voice-detect/options.go create mode 100755 backend/go/voice-detect/package.sh create mode 100755 backend/go/voice-detect/run.sh create mode 100755 backend/go/voice-detect/test.sh diff --git a/.github/backend-matrix.yml b/.github/backend-matrix.yml index 1d6496231..a497a72c1 100644 --- a/.github/backend-matrix.yml +++ b/.github/backend-matrix.yml @@ -3745,6 +3745,302 @@ include: dockerfile: "./backend/Dockerfile.golang" context: "./" ubuntu-version: '2404' + # voice-detect + - build-type: 'cublas' + cuda-major-version: "12" + cuda-minor-version: "8" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-nvidia-cuda-12-voice-detect' + runs-on: 'ubuntu-latest' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'cublas' + cuda-major-version: "13" + cuda-minor-version: "0" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-nvidia-cuda-13-voice-detect' + runs-on: 'ubuntu-latest' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'cublas' + cuda-major-version: "13" + cuda-minor-version: "0" + platforms: 'linux/arm64' + skip-drivers: 'false' + tag-latest: 'auto' + tag-suffix: '-nvidia-l4t-cuda-13-arm64-voice-detect' + base-image: "ubuntu:24.04" + ubuntu-version: '2404' + runs-on: 'ubuntu-24.04-arm' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + - build-type: '' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + platform-tag: 'amd64' + tag-latest: 'auto' + tag-suffix: '-cpu-voice-detect' + runs-on: 'ubuntu-latest' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: '' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/arm64' + platform-tag: 'arm64' + tag-latest: 'auto' + tag-suffix: '-cpu-voice-detect' + runs-on: 'ubuntu-24.04-arm' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'sycl_f32' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-intel-sycl-f32-voice-detect' + runs-on: 'ubuntu-latest' + base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04" + skip-drivers: 'false' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'sycl_f16' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-intel-sycl-f16-voice-detect' + runs-on: 'ubuntu-latest' + base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04" + skip-drivers: 'false' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'vulkan' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + platform-tag: 'amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-vulkan-voice-detect' + runs-on: 'ubuntu-latest' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'vulkan' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/arm64' + platform-tag: 'arm64' + tag-latest: 'auto' + tag-suffix: '-gpu-vulkan-voice-detect' + runs-on: 'ubuntu-24.04-arm' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'cublas' + cuda-major-version: "12" + cuda-minor-version: "0" + platforms: 'linux/arm64' + skip-drivers: 'false' + tag-latest: 'auto' + tag-suffix: '-nvidia-l4t-arm64-voice-detect' + base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0" + runs-on: 'ubuntu-24.04-arm' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2204' + - build-type: 'hipblas' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-rocm-hipblas-voice-detect' + base-image: "rocm/dev-ubuntu-24.04:7.2.1" + runs-on: 'ubuntu-latest' + skip-drivers: 'false' + backend: "voice-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + # face-detect + - build-type: 'cublas' + cuda-major-version: "12" + cuda-minor-version: "8" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-nvidia-cuda-12-face-detect' + runs-on: 'ubuntu-latest' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'cublas' + cuda-major-version: "13" + cuda-minor-version: "0" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-nvidia-cuda-13-face-detect' + runs-on: 'ubuntu-latest' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'cublas' + cuda-major-version: "13" + cuda-minor-version: "0" + platforms: 'linux/arm64' + skip-drivers: 'false' + tag-latest: 'auto' + tag-suffix: '-nvidia-l4t-cuda-13-arm64-face-detect' + base-image: "ubuntu:24.04" + ubuntu-version: '2404' + runs-on: 'ubuntu-24.04-arm' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + - build-type: '' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + platform-tag: 'amd64' + tag-latest: 'auto' + tag-suffix: '-cpu-face-detect' + runs-on: 'ubuntu-latest' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: '' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/arm64' + platform-tag: 'arm64' + tag-latest: 'auto' + tag-suffix: '-cpu-face-detect' + runs-on: 'ubuntu-24.04-arm' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'sycl_f32' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-intel-sycl-f32-face-detect' + runs-on: 'ubuntu-latest' + base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04" + skip-drivers: 'false' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'sycl_f16' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-intel-sycl-f16-face-detect' + runs-on: 'ubuntu-latest' + base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04" + skip-drivers: 'false' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'vulkan' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + platform-tag: 'amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-vulkan-face-detect' + runs-on: 'ubuntu-latest' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'vulkan' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/arm64' + platform-tag: 'arm64' + tag-latest: 'auto' + tag-suffix: '-gpu-vulkan-face-detect' + runs-on: 'ubuntu-24.04-arm' + base-image: "ubuntu:24.04" + skip-drivers: 'false' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' + - build-type: 'cublas' + cuda-major-version: "12" + cuda-minor-version: "0" + platforms: 'linux/arm64' + skip-drivers: 'false' + tag-latest: 'auto' + tag-suffix: '-nvidia-l4t-arm64-face-detect' + base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0" + runs-on: 'ubuntu-24.04-arm' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2204' + - build-type: 'hipblas' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-rocm-hipblas-face-detect' + base-image: "rocm/dev-ubuntu-24.04:7.2.1" + runs-on: 'ubuntu-latest' + skip-drivers: 'false' + backend: "face-detect" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' # acestep-cpp - build-type: '' cuda-major-version: "" @@ -4928,6 +5224,14 @@ includeDarwin: tag-suffix: "-metal-darwin-arm64-ced" build-type: "metal" lang: "go" + - backend: "voice-detect" + tag-suffix: "-metal-darwin-arm64-voice-detect" + build-type: "metal" + lang: "go" + - backend: "face-detect" + tag-suffix: "-metal-darwin-arm64-face-detect" + build-type: "metal" + lang: "go" - backend: "acestep-cpp" tag-suffix: "-metal-darwin-arm64-acestep-cpp" build-type: "metal" diff --git a/.github/workflows/bump_deps.yaml b/.github/workflows/bump_deps.yaml index a2c37881f..afbe55b0b 100644 --- a/.github/workflows/bump_deps.yaml +++ b/.github/workflows/bump_deps.yaml @@ -46,6 +46,14 @@ jobs: variable: "CED_VERSION" branch: "master" file: "backend/go/ced/Makefile" + - repository: "mudler/voice-detect.cpp" + variable: "VOICEDETECT_VERSION" + branch: "master" + file: "backend/go/voice-detect/Makefile" + - repository: "mudler/face-detect.cpp" + variable: "FACEDETECT_VERSION" + branch: "master" + file: "backend/go/face-detect/Makefile" - repository: "mudler/depth-anything.cpp" variable: "DEPTHANYTHING_VERSION" branch: "master" diff --git a/README.md b/README.md index f7843950d..a4135d28e 100644 --- a/README.md +++ b/README.md @@ -177,6 +177,7 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett ## Latest News +- **June 2026**: New native biometric backends from the LocalAI team: [voice-detect.cpp](https://github.com/mudler/voice-detect.cpp) for speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion) and [face-detect.cpp](https://github.com/mudler/face-detect.cpp) for face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace). Both are from-scratch C++/ggml engines with no Python or onnxruntime at inference, self-contained GGUF weights, bit-exact parity with the reference, and GPU cuDNN parity, replacing the heavier Python `insightface` and `speaker-recognition` backends ([PR #10441](https://github.com/mudler/LocalAI/pull/10441)). - **June 2026**: New [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) (a tiny Go client for the Realtime API with a full talk-back voice loop and tool calling), plus [streaming of the realtime LLM / TTS / transcription pipeline stages](https://github.com/mudler/LocalAI/pull/10176) and [configurable WebRTC ICE candidates](https://github.com/mudler/LocalAI/pull/10231). - **June 2026**: Big speech push: the [parakeet.cpp](https://github.com/mudler/parakeet.cpp) ASR engine gains [NeMo-faithful segment timestamps](https://github.com/mudler/LocalAI/pull/10207), a [multilingual streaming Nemotron-3.5 model](https://github.com/mudler/LocalAI/pull/10199), [dynamic batching for concurrent transcription](https://github.com/mudler/LocalAI/pull/10112) and [CUDA graphs](https://github.com/mudler/LocalAI/pull/10273); the new [CrispASR backend](https://github.com/mudler/LocalAI/pull/10099) adds multi-architecture ASR + TTS, and [60 Piper TTS voices across 42 languages](https://github.com/mudler/LocalAI/pull/10296) land in the gallery (plus [per-request TTS instructions and params](https://github.com/mudler/LocalAI/pull/10172)). - **June 2026**: New backends and models: [locate-anything.cpp](https://github.com/mudler/LocalAI/pull/10264) for open-vocabulary object detection via ggml, [Ideogram4 image generation](https://github.com/mudler/LocalAI/pull/10201) in stablediffusion-ggml, [llama.cpp video input](https://github.com/mudler/LocalAI/pull/10216), and the [Gemma 4 QAT family with MTP speculative-decoding pairs](https://github.com/mudler/LocalAI/pull/10215). Plus an [interactive CLI chat mode](https://github.com/mudler/LocalAI/pull/10226) and [RAG source citations in agent responses](https://github.com/mudler/LocalAI/pull/10228). diff --git a/backend/Dockerfile.golang b/backend/Dockerfile.golang index d188cdf70..13032fa22 100644 --- a/backend/Dockerfile.golang +++ b/backend/Dockerfile.golang @@ -137,7 +137,7 @@ RUN </dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4) + +BUILD_TYPE?= +NATIVE?=false + +# Resolve the target arch. The backend matrix / Docker build pass TARGETARCH +# (amd64|arm64); fall back to uname -m (aarch64|x86_64) for a local build. +RECON_ARCH?=$(or $(TARGETARCH),$(shell uname -m)) + +# Build ggml + the vendored libjpeg-turbo statically into libfacedetect.so (PIC) +# so the shared lib is self-contained: dlopen needs no libggml*.so alongside it, +# only system libs (libstdc++/libgomp/libc) the runtime image already provides. +# The vendored jpeg symbols are hidden via -Wl,--exclude-libs,ALL on the C++ +# side, so only the facedetect_capi_* surface is exported. +CMAKE_ARGS?=-DCMAKE_BUILD_TYPE=Release -DFACEDETECT_SHARED=ON -DFACEDETECT_BUILD_CLI=OFF -DFACEDETECT_BUILD_TESTS=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON + +ifeq ($(NATIVE),false) + CMAKE_ARGS+=-DGGML_NATIVE=OFF +endif + +# face-detect.cpp gates its GGML backends behind FACEDETECT_GGML_* options and +# does set(GGML_CUDA ${FACEDETECT_GGML_CUDA} CACHE BOOL "" FORCE), so a bare +# -DGGML_CUDA=ON is overwritten back to OFF. Forward the FACEDETECT_GGML_* +# options instead. (openblas is not gated, so -DGGML_BLAS passes through.) +ifeq ($(BUILD_TYPE),cublas) + CMAKE_ARGS+=-DFACEDETECT_GGML_CUDA=ON + # Opt-in cuDNN implicit-GEMM conv path (kills im2col on GPU, SCRFD 2.3x + # vs torch-cuDNN parity). Only the arm64 + CUDA 13 image (GB10/Jetson/L4T) + # ships libcudnn9 + the -dev headers, so gate cuDNN to that variant. + # x86 CUDA images carry no cuDNN -> enabling it there is a link failure. + ifeq ($(CUDA_MAJOR_VERSION),13) + ifneq (,$(filter arm64 aarch64,$(RECON_ARCH))) + CMAKE_ARGS+=-DFACEDETECT_GGML_CUDNN=ON + endif + endif +else ifeq ($(BUILD_TYPE),openblas) + CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS +else ifeq ($(BUILD_TYPE),hipblas) + CMAKE_ARGS+=-DFACEDETECT_GGML_HIP=ON +else ifeq ($(BUILD_TYPE),vulkan) + CMAKE_ARGS+=-DFACEDETECT_GGML_VULKAN=ON +else ifeq ($(BUILD_TYPE),metal) + CMAKE_ARGS+=-DFACEDETECT_GGML_METAL=ON +endif + +.PHONY: face-detect-grpc package build clean purge test all + +all: face-detect-grpc + +# Clone the upstream face-detect.cpp source at the pinned commit. Directory acts +# as the target so make only re-clones when missing. After a FACEDETECT_VERSION +# bump, run 'make purge && make' to refetch. +sources/face-detect.cpp: + mkdir -p sources/face-detect.cpp + cd sources/face-detect.cpp && \ + git init -q && \ + git remote add origin $(FACEDETECT_REPO) && \ + git fetch --depth 1 origin $(FACEDETECT_VERSION) && \ + git checkout FETCH_HEAD && \ + git submodule update --init --recursive --depth 1 --single-branch + +# Build the shared lib + header out-of-tree, then stage them next to the Go +# sources so purego.Dlopen("libfacedetect.so") and the cgo-less build both pick +# them up. +libfacedetect.so: sources/face-detect.cpp + cmake -B sources/face-detect.cpp/build-shared -S sources/face-detect.cpp $(CMAKE_ARGS) + cmake --build sources/face-detect.cpp/build-shared --config Release -j$(JOBS) --target facedetect + cp -fv sources/face-detect.cpp/build-shared/libfacedetect.so* ./ 2>/dev/null || true + cp -fv sources/face-detect.cpp/include/facedetect_capi.h ./ + +face-detect-grpc: libfacedetect.so main.go gofacedetect.go options.go + CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o face-detect-grpc . + +package: face-detect-grpc + bash package.sh + +build: package + +# Test target. The embed/detect/verify/analyze smoke specs are gated on +# FACEDETECT_BACKEND_TEST_MODEL + FACEDETECT_BACKEND_TEST_IMAGE; without them the +# heavy specs auto-skip and only the pure-Go parsing specs run. +test: + LD_LIBRARY_PATH=$(CURDIR):$$LD_LIBRARY_PATH $(GOCMD) test ./... -count=1 + +clean: purge + rm -rf libfacedetect.so* facedetect_capi.h package face-detect-grpc + +purge: + rm -rf sources/face-detect.cpp diff --git a/backend/go/face-detect/gofacedetect.go b/backend/go/face-detect/gofacedetect.go new file mode 100644 index 000000000..4ad6c067c --- /dev/null +++ b/backend/go/face-detect/gofacedetect.go @@ -0,0 +1,431 @@ +package main + +import ( + "encoding/base64" + "encoding/json" + "errors" + "fmt" + "math" + "os" + "path/filepath" + "strconv" + "strings" + "time" + "unsafe" + + "github.com/mudler/LocalAI/pkg/grpc/base" + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + "github.com/mudler/xlog" +) + +// purego-bound entry points from libfacedetect.so. Names match +// facedetect_capi.h exactly so a `nm libfacedetect.so | grep facedetect_capi` +// is enough to spot drift. +// +// The opaque ctx and the malloc'd char*/float* return values are declared as +// uintptr so we get the raw pointer back and can release it via the matching +// capi free function. purego's native string/[]float32 returns would copy and +// forget the original pointer, leaking the C-owned buffer on every call. +var ( + CppAbiVersion func() int32 + CppLoad func(ggufPath string) uintptr + CppFree func(ctx uintptr) + CppLastError func(ctx uintptr) string + CppFreeString func(s uintptr) + CppFreeVec func(v uintptr) + CppEmbedPath func(ctx uintptr, imagePath string, outVec, outDim unsafe.Pointer) int32 + CppEmbedRGB func(ctx uintptr, rgb []byte, width, height int32, outVec, outDim unsafe.Pointer) int32 + CppDetectJSON func(ctx uintptr, imagePath string) uintptr + CppVerifyPaths func(ctx uintptr, a, b string, threshold float32, antiSpoof int32, outDistance, outVerified unsafe.Pointer) int32 + CppAnalyzeJSON func(ctx uintptr, imagePath string) uintptr +) + +// FaceDetect implements the face-recognition (biometric) subset of the Backend +// gRPC service over libfacedetect.so. The C side keeps a single loaded model +// pack plus a per-ctx last-error buffer and is not reentrant, so +// base.SingleThread serializes every call. +type FaceDetect struct { + base.SingleThread + opts loadOptions + ctxPtr uintptr +} + +func (f *FaceDetect) Load(opts *pb.ModelOptions) error { + model := opts.ModelFile + if model == "" { + model = opts.ModelPath + } + if !filepath.IsAbs(model) && opts.ModelPath != "" { + model = filepath.Join(opts.ModelPath, model) + } + if model == "" { + return errors.New("face-detect: ModelFile is required") + } + + f.opts = parseOptions(opts.Options) + if f.opts.modelName == "" { + f.opts.modelName = filepath.Base(model) + } + + // Propagate LocalAI's per-model thread budget to the engine. LocalAI spawns + // one backend process per model and serves requests concurrently, so the + // engine's own min(hardware_concurrency, 8) default can oversubscribe cores. + // FACEDETECT_THREADS is read by the engine at backend construction, so it + // must be set before the capi load. A non-positive Threads means "unset": + // leave the env alone so the engine keeps its sane default. + threads := opts.Threads + if threads > 0 { + if err := os.Setenv("FACEDETECT_THREADS", strconv.Itoa(int(threads))); err != nil { + return fmt.Errorf("face-detect: set FACEDETECT_THREADS: %w", err) + } + xlog.Info("face-detect: applying LocalAI thread budget", "threads", threads) + } + + xlog.Info("face-detect: loading model", "model", model, + "verify_threshold", f.opts.verifyThreshold, "abi", CppAbiVersion()) + + ctx := CppLoad(model) + if ctx == 0 { + // The last-error buffer lives on the ctx that was never returned, so + // surface the path the operator tried to load instead. + return fmt.Errorf("face-detect: facedetect_capi_load failed for %q", model) + } + f.ctxPtr = ctx + return nil +} + +// Embeddings returns the L2-normalized ArcFace embedding of the primary face in +// the supplied image. Mirroring the Python face backend, the image is read from +// Images[0] as a base64 payload; materializeImage decodes it to a temp file so +// the path-based C-API can run its own decode (cv2.imread parity). The gRPC +// server wraps the returned slice in an EmbeddingResult. +func (f *FaceDetect) Embeddings(req *pb.PredictOptions) ([]float32, error) { + if f.ctxPtr == 0 { + return nil, errors.New("face-detect: model not loaded") + } + if len(req.Images) == 0 || req.Images[0] == "" { + return nil, errors.New("face-detect: Embedding requires Images[0] to be a base64 image") + } + + path, cleanup, err := materializeImage(req.Images[0]) + if err != nil { + return nil, err + } + defer cleanup() + + return f.embedPath(path) +} + +func (f *FaceDetect) embedPath(path string) ([]float32, error) { + var vec uintptr + var dim int32 + rc := CppEmbedPath(f.ctxPtr, path, unsafe.Pointer(&vec), unsafe.Pointer(&dim)) + if rc != 0 || vec == 0 || dim <= 0 { + return nil, f.lastErr("embed", path) + } + defer CppFreeVec(vec) + // Copy out of the C-owned malloc'd buffer before freeing it. The + // uintptr->Pointer conversion trips vet's unsafeptr check, which can't tell + // a C heap pointer from Go-managed memory; safe here, the GC neither tracks + // nor moves this buffer and we copy immediately. + src := unsafe.Slice((*float32)(unsafe.Pointer(vec)), int(dim)) //nolint:govet // C-owned malloc'd vector, copied out before free + out := make([]float32, int(dim)) + copy(out, src) + return out, nil +} + +// Detect runs SCRFD over the image and returns one Detection per face. The +// C-API emits a box as [x1,y1,x2,y2] in pixels; the proto carries x/y plus +// width/height, so the corners are converted. The 5 facial landmarks the engine +// also returns are dropped: the Detection message has no field for them. +func (f *FaceDetect) Detect(req *pb.DetectOptions) (pb.DetectResponse, error) { + if f.ctxPtr == 0 { + return pb.DetectResponse{}, errors.New("face-detect: model not loaded") + } + if req.Src == "" { + return pb.DetectResponse{}, errors.New("face-detect: src image is required") + } + + path, cleanup, err := materializeImage(req.Src) + if err != nil { + return pb.DetectResponse{}, err + } + defer cleanup() + + faces, err := f.detectFaces(path) + if err != nil { + return pb.DetectResponse{}, err + } + + dets := make([]*pb.Detection, 0, len(faces)) + for _, fc := range faces { + if req.Threshold > 0 && fc.Score < req.Threshold { + continue + } + x, y, w, h := fc.xywh() + dets = append(dets, &pb.Detection{ + X: x, + Y: y, + Width: w, + Height: h, + Confidence: fc.Score, + ClassName: "face", + }) + } + return pb.DetectResponse{Detections: dets}, nil +} + +// FaceVerify embeds the primary face in each image and reports whether they are +// the same identity by cosine distance against a threshold. A request threshold +// <= 0 falls back to the model-configured default (verify_threshold option, +// 0.35 if unset). When anti_spoofing is set, the C-API applies a MiniFASNet +// veto internally (verified forced false on a spoof); the per-image liveness +// scores are not exposed by the verify entry point, so img*_is_real / +// img*_antispoof_score stay at their zero values. +func (f *FaceDetect) FaceVerify(req *pb.FaceVerifyRequest) (pb.FaceVerifyResponse, error) { + if f.ctxPtr == 0 { + return pb.FaceVerifyResponse{}, errors.New("face-detect: model not loaded") + } + if req.Img1 == "" || req.Img2 == "" { + return pb.FaceVerifyResponse{}, errors.New("face-detect: img1 and img2 are required") + } + + path1, cleanup1, err := materializeImage(req.Img1) + if err != nil { + return pb.FaceVerifyResponse{}, err + } + defer cleanup1() + path2, cleanup2, err := materializeImage(req.Img2) + if err != nil { + return pb.FaceVerifyResponse{}, err + } + defer cleanup2() + + threshold := req.Threshold + if threshold <= 0 { + threshold = f.opts.verifyThreshold + } + + antiSpoof := int32(0) + if req.AntiSpoofing { + antiSpoof = 1 + } + + started := time.Now() + var distance float32 + var verified int32 + rc := CppVerifyPaths(f.ctxPtr, path1, path2, threshold, antiSpoof, + unsafe.Pointer(&distance), unsafe.Pointer(&verified)) + if rc != 0 { + return pb.FaceVerifyResponse{}, f.lastErr("verify", req.Img1[:min(8, len(req.Img1))]+"...") + } + elapsedMs := float32(time.Since(started).Seconds() * 1000.0) + + // Confidence decays linearly from 100 at distance 0 to 0 at the threshold, + // matching the Python face backend's reporting. + confidence := float32(0) + if threshold > 0 { + confidence = float32(math.Max(0, math.Min(100, (1.0-float64(distance)/float64(threshold))*100.0))) + } + + return pb.FaceVerifyResponse{ + Verified: verified != 0, + Distance: distance, + Threshold: threshold, + Confidence: confidence, + Model: f.opts.modelName, + Img1Area: f.bestArea(path1), + Img2Area: f.bestArea(path2), + ProcessingTimeMs: elapsedMs, + }, nil +} + +// FaceAnalyze runs the genderage head on every detected face. The C-API returns +// "M"/"F" gender labels and a rounded age; the labels are normalized to the +// "Man"/"Woman" values the proto documents. +func (f *FaceDetect) FaceAnalyze(req *pb.FaceAnalyzeRequest) (pb.FaceAnalyzeResponse, error) { + if f.ctxPtr == 0 { + return pb.FaceAnalyzeResponse{}, errors.New("face-detect: model not loaded") + } + if req.Img == "" { + return pb.FaceAnalyzeResponse{}, errors.New("face-detect: img is required") + } + + path, cleanup, err := materializeImage(req.Img) + if err != nil { + return pb.FaceAnalyzeResponse{}, err + } + defer cleanup() + + ptr := CppAnalyzeJSON(f.ctxPtr, path) + if ptr == 0 { + return pb.FaceAnalyzeResponse{}, f.lastErr("analyze", path) + } + defer CppFreeString(ptr) + + faces, err := parseAnalyzeJSON(goStringFromCPtr(ptr)) + if err != nil { + return pb.FaceAnalyzeResponse{}, fmt.Errorf("face-detect: analyze JSON: %w", err) + } + return pb.FaceAnalyzeResponse{Faces: faces}, nil +} + +// faceBox is one entry of the detect/analyze JSON documents the engine emits. +type faceBox struct { + Score float32 `json:"score"` + Box []float32 `json:"box"` + Age float32 `json:"age"` + Gender string `json:"gender"` +} + +// xywh converts the engine's [x1,y1,x2,y2] box into the x/y/width/height the +// proto carries. A short or missing box yields zeros. +func (b faceBox) xywh() (x, y, w, h float32) { + if len(b.Box) < 4 { + return 0, 0, 0, 0 + } + return b.Box[0], b.Box[1], b.Box[2] - b.Box[0], b.Box[3] - b.Box[1] +} + +type facesJSON struct { + Faces []faceBox `json:"faces"` +} + +func (f *FaceDetect) detectFaces(path string) ([]faceBox, error) { + ptr := CppDetectJSON(f.ctxPtr, path) + if ptr == 0 { + return nil, f.lastErr("detect", path) + } + defer CppFreeString(ptr) + + var doc facesJSON + if err := json.Unmarshal([]byte(goStringFromCPtr(ptr)), &doc); err != nil { + return nil, fmt.Errorf("face-detect: detect JSON: %w", err) + } + return doc.Faces, nil +} + +// bestArea returns the FacialArea of the highest-scoring face in an image, or an +// empty area when detection fails or finds nothing. Best-effort: verify already +// succeeded, so a missing region must not turn a valid match into an error. +func (f *FaceDetect) bestArea(path string) *pb.FacialArea { + faces, err := f.detectFaces(path) + if err != nil || len(faces) == 0 { + return &pb.FacialArea{} + } + best := faces[0] + for _, fc := range faces[1:] { + if fc.Score > best.Score { + best = fc + } + } + x, y, w, h := best.xywh() + return &pb.FacialArea{X: x, Y: y, W: w, H: h} +} + +// parseAnalyzeJSON maps the engine's analyze document onto FaceAnalysis entries. +// The engine reports gender as "M"/"F"; both the dominant label and the score +// map are filled with the "Man"/"Woman" form the proto documents. +func parseAnalyzeJSON(doc string) ([]*pb.FaceAnalysis, error) { + var parsed facesJSON + if err := json.Unmarshal([]byte(doc), &parsed); err != nil { + return nil, err + } + + out := make([]*pb.FaceAnalysis, 0, len(parsed.Faces)) + for _, fc := range parsed.Faces { + x, y, w, h := fc.xywh() + fa := &pb.FaceAnalysis{ + Region: &pb.FacialArea{X: x, Y: y, W: w, H: h}, + FaceConfidence: fc.Score, + Age: fc.Age, + } + if label := normalizeGender(fc.Gender); label != "" { + fa.DominantGender = label + fa.Gender = map[string]float32{label: 1.0} + } + out = append(out, fa) + } + return out, nil +} + +// normalizeGender maps the engine's "M"/"F" code to the "Man"/"Woman" labels the +// proto documents. Unknown codes pass through unchanged. +func normalizeGender(g string) string { + switch strings.ToUpper(strings.TrimSpace(g)) { + case "M": + return "Man" + case "F": + return "Woman" + case "": + return "" + default: + return g + } +} + +// materializeImage decodes a base64 image payload into a temp file and returns +// its path plus a cleanup func. As a convenience for callers that already pass a +// filesystem path (e.g. a test fixture), an existing path is used as-is with a +// no-op cleanup. data: URI prefixes are stripped before decoding. +func materializeImage(src string) (path string, cleanup func(), err error) { + noop := func() {} + if src == "" { + return "", noop, errors.New("face-detect: empty image input") + } + if _, statErr := os.Stat(src); statErr == nil { + return src, noop, nil + } + + payload := src + if i := strings.Index(payload, ","); strings.HasPrefix(payload, "data:") && i >= 0 { + payload = payload[i+1:] + } + data, decErr := base64.StdEncoding.DecodeString(strings.TrimSpace(payload)) + if decErr != nil || len(data) == 0 { + return "", noop, errors.New("face-detect: image is neither an existing path nor valid base64") + } + + tmp, createErr := os.CreateTemp("", "face-detect-*.img") + if createErr != nil { + return "", noop, fmt.Errorf("face-detect: create temp image: %w", createErr) + } + cleanup = func() { _ = os.Remove(tmp.Name()) } + if _, wErr := tmp.Write(data); wErr != nil { + _ = tmp.Close() + cleanup() + return "", noop, fmt.Errorf("face-detect: write temp image: %w", wErr) + } + if cErr := tmp.Close(); cErr != nil { + cleanup() + return "", noop, fmt.Errorf("face-detect: close temp image: %w", cErr) + } + return tmp.Name(), cleanup, nil +} + +// lastErr wraps the C-API's per-ctx last-error buffer into a Go error. +func (f *FaceDetect) lastErr(op, subject string) error { + msg := strings.TrimSpace(CppLastError(f.ctxPtr)) + if msg == "" { + msg = "no error detail" + } + return fmt.Errorf("face-detect: %s failed for %q: %s", op, subject, msg) +} + +// goStringFromCPtr copies a NUL-terminated C string into Go memory. cptr is a +// malloc'd buffer the caller owns; release it via CppFreeString after the copy. +// +// The uintptr->Pointer conversion trips vet's unsafeptr check, which can't tell +// a C heap pointer from Go-managed memory. Safe here: the GC neither tracks nor +// moves the buffer and we dereference it immediately to copy the bytes out. +func goStringFromCPtr(cptr uintptr) string { + if cptr == 0 { + return "" + } + p := unsafe.Pointer(cptr) //nolint:govet // C-owned malloc'd buffer, not Go-GC memory (see doc above) + n := 0 + for *(*byte)(unsafe.Add(p, n)) != 0 { + n++ + } + return string(unsafe.Slice((*byte)(p), n)) +} diff --git a/backend/go/face-detect/gofacedetect_test.go b/backend/go/face-detect/gofacedetect_test.go new file mode 100644 index 000000000..54a942fba --- /dev/null +++ b/backend/go/face-detect/gofacedetect_test.go @@ -0,0 +1,230 @@ +package main + +import ( + "encoding/base64" + "os" + "sync" + "testing" + + "github.com/ebitengine/purego" + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestFaceDetect(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "face-detect Backend Suite") +} + +var ( + libLoadOnce sync.Once + libLoadErr error +) + +// ensureLibLoaded mirrors main.go's bootstrap so a Go test can drive the C-API +// bridge without spinning up the gRPC server. Records the error (the smoke +// specs skip themselves) when libfacedetect.so is not loadable from cwd +// (LD_LIBRARY_PATH or a symlink in ./). +func ensureLibLoaded() error { + libLoadOnce.Do(func() { + libName := os.Getenv("FACEDETECT_LIBRARY") + if libName == "" { + libName = "libfacedetect.so" + } + lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL) + if err != nil { + libLoadErr = err + return + } + purego.RegisterLibFunc(&CppAbiVersion, lib, "facedetect_capi_abi_version") + purego.RegisterLibFunc(&CppLoad, lib, "facedetect_capi_load") + purego.RegisterLibFunc(&CppFree, lib, "facedetect_capi_free") + purego.RegisterLibFunc(&CppLastError, lib, "facedetect_capi_last_error") + purego.RegisterLibFunc(&CppFreeString, lib, "facedetect_capi_free_string") + purego.RegisterLibFunc(&CppFreeVec, lib, "facedetect_capi_free_vec") + purego.RegisterLibFunc(&CppEmbedPath, lib, "facedetect_capi_embed_path") + purego.RegisterLibFunc(&CppEmbedRGB, lib, "facedetect_capi_embed_rgb") + purego.RegisterLibFunc(&CppDetectJSON, lib, "facedetect_capi_detect_path_json") + purego.RegisterLibFunc(&CppVerifyPaths, lib, "facedetect_capi_verify_paths") + purego.RegisterLibFunc(&CppAnalyzeJSON, lib, "facedetect_capi_analyze_path_json") + }) + return libLoadErr +} + +var _ = Describe("parseOptions", func() { + It("defaults verify_threshold to 0.35", func() { + o := parseOptions(nil) + Expect(o.verifyThreshold).To(Equal(float32(0.35))) + Expect(o.modelName).To(Equal("")) + }) + + It("parses verify_threshold, threshold alias and model_name", func() { + o := parseOptions([]string{"verify_threshold:0.4", "model_name:buffalo_l", "unknown:x"}) + Expect(o.verifyThreshold).To(Equal(float32(0.4))) + Expect(o.modelName).To(Equal("buffalo_l")) + + o2 := parseOptions([]string{"threshold:0.3"}) + Expect(o2.verifyThreshold).To(Equal(float32(0.3))) + }) + + It("ignores non-positive thresholds and keeps the default", func() { + o := parseOptions([]string{"verify_threshold:0", "threshold:-1"}) + Expect(o.verifyThreshold).To(Equal(float32(0.35))) + }) +}) + +var _ = Describe("normalizeGender", func() { + It("maps M/F codes to Man/Woman", func() { + Expect(normalizeGender("M")).To(Equal("Man")) + Expect(normalizeGender("f")).To(Equal("Woman")) + Expect(normalizeGender(" m ")).To(Equal("Man")) + }) + + It("passes empty and unknown codes through", func() { + Expect(normalizeGender("")).To(Equal("")) + Expect(normalizeGender("nonbinary")).To(Equal("nonbinary")) + }) +}) + +var _ = Describe("faceBox.xywh", func() { + It("converts an [x1,y1,x2,y2] box to x/y/width/height", func() { + b := faceBox{Box: []float32{10, 20, 50, 80}} + x, y, w, h := b.xywh() + Expect(x).To(Equal(float32(10))) + Expect(y).To(Equal(float32(20))) + Expect(w).To(Equal(float32(40))) + Expect(h).To(Equal(float32(60))) + }) + + It("returns zeros for a short box", func() { + x, y, w, h := faceBox{Box: []float32{1, 2}}.xywh() + Expect([]float32{x, y, w, h}).To(Equal([]float32{0, 0, 0, 0})) + }) +}) + +var _ = Describe("parseAnalyzeJSON", func() { + It("maps region, age and gender for each face", func() { + doc := `{"faces":[ + {"score":0.997,"box":[10,20,50,80],"age":31,"gender":"M"}, + {"score":0.81,"box":[0,0,40,40],"age":24,"gender":"F"}]}` + faces, err := parseAnalyzeJSON(doc) + Expect(err).ToNot(HaveOccurred()) + Expect(faces).To(HaveLen(2)) + + Expect(faces[0].FaceConfidence).To(BeNumerically("~", 0.997, 1e-4)) + Expect(faces[0].Age).To(BeNumerically("~", 31, 1e-4)) + Expect(faces[0].DominantGender).To(Equal("Man")) + Expect(faces[0].Gender).To(HaveKeyWithValue("Man", float32(1.0))) + Expect(faces[0].Region.W).To(Equal(float32(40))) + Expect(faces[0].Region.H).To(Equal(float32(60))) + + Expect(faces[1].DominantGender).To(Equal("Woman")) + }) + + It("tolerates a missing gender field", func() { + faces, err := parseAnalyzeJSON(`{"faces":[{"score":0.5,"box":[0,0,10,10],"age":40}]}`) + Expect(err).ToNot(HaveOccurred()) + Expect(faces).To(HaveLen(1)) + Expect(faces[0].DominantGender).To(Equal("")) + Expect(faces[0].Gender).To(BeEmpty()) + }) + + It("returns no faces for an empty document", func() { + faces, err := parseAnalyzeJSON(`{"faces":[]}`) + Expect(err).ToNot(HaveOccurred()) + Expect(faces).To(BeEmpty()) + }) + + It("returns an error on malformed JSON", func() { + _, err := parseAnalyzeJSON(`{not-json`) + Expect(err).To(HaveOccurred()) + }) +}) + +var _ = Describe("materializeImage", func() { + It("decodes a base64 payload to a temp file", func() { + payload := base64.StdEncoding.EncodeToString([]byte("\xff\xd8\xff\xe0fake-jpeg")) + path, cleanup, err := materializeImage(payload) + Expect(err).ToNot(HaveOccurred()) + defer cleanup() + data, rerr := os.ReadFile(path) + Expect(rerr).ToNot(HaveOccurred()) + Expect(data).To(Equal([]byte("\xff\xd8\xff\xe0fake-jpeg"))) + }) + + It("strips a data: URI prefix before decoding", func() { + payload := "data:image/png;base64," + base64.StdEncoding.EncodeToString([]byte("hello")) + path, cleanup, err := materializeImage(payload) + Expect(err).ToNot(HaveOccurred()) + defer cleanup() + data, rerr := os.ReadFile(path) + Expect(rerr).ToNot(HaveOccurred()) + Expect(data).To(Equal([]byte("hello"))) + }) + + It("uses an existing path as-is", func() { + tmp, err := os.CreateTemp("", "face-detect-fixture-*.bin") + Expect(err).ToNot(HaveOccurred()) + defer func() { _ = os.Remove(tmp.Name()) }() + Expect(tmp.Close()).To(Succeed()) + + path, cleanup, err := materializeImage(tmp.Name()) + Expect(err).ToNot(HaveOccurred()) + defer cleanup() + Expect(path).To(Equal(tmp.Name())) + }) + + It("errors on input that is neither a path nor base64", func() { + _, _, err := materializeImage("not base64!!!") + Expect(err).To(HaveOccurred()) + }) +}) + +// The specs below exercise the real C-API end to end. They run only when both a +// model GGUF and a test image are provided, and skip cleanly otherwise so the +// suite stays green without large assets. +var _ = Describe("FaceDetect end-to-end", Ordered, func() { + var ( + f *FaceDetect + modelPath = os.Getenv("FACEDETECT_BACKEND_TEST_MODEL") + imagePath = os.Getenv("FACEDETECT_BACKEND_TEST_IMAGE") + ) + + BeforeAll(func() { + if modelPath == "" || imagePath == "" { + Skip("set FACEDETECT_BACKEND_TEST_MODEL and FACEDETECT_BACKEND_TEST_IMAGE to run the e2e specs") + } + if err := ensureLibLoaded(); err != nil { + Skip("libfacedetect.so not loadable: " + err.Error()) + } + f = &FaceDetect{} + Expect(f.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed()) + }) + + It("embeds the primary face in an image", func() { + emb, err := f.Embeddings(&pb.PredictOptions{Images: []string{imagePath}}) + Expect(err).ToNot(HaveOccurred()) + Expect(emb).ToNot(BeEmpty()) + }) + + It("detects at least one face", func() { + resp, err := f.Detect(&pb.DetectOptions{Src: imagePath}) + Expect(err).ToNot(HaveOccurred()) + Expect(resp.Detections).ToNot(BeEmpty()) + Expect(resp.Detections[0].ClassName).To(Equal("face")) + }) + + It("verifies an image against itself as the same identity", func() { + resp, err := f.FaceVerify(&pb.FaceVerifyRequest{Img1: imagePath, Img2: imagePath}) + Expect(err).ToNot(HaveOccurred()) + Expect(resp.Verified).To(BeTrue()) + Expect(resp.Distance).To(BeNumerically("<=", resp.Threshold)) + }) + + It("analyzes age/gender for each face", func() { + resp, err := f.FaceAnalyze(&pb.FaceAnalyzeRequest{Img: imagePath}) + Expect(err).ToNot(HaveOccurred()) + Expect(resp.Faces).ToNot(BeEmpty()) + }) +}) diff --git a/backend/go/face-detect/main.go b/backend/go/face-detect/main.go new file mode 100644 index 000000000..dc52f1e60 --- /dev/null +++ b/backend/go/face-detect/main.go @@ -0,0 +1,65 @@ +package main + +// Started internally by LocalAI - one gRPC server per loaded model. +// +// Loads libfacedetect.so via purego and registers the flat C-API entry points +// declared in facedetect_capi.h. The library name can be overridden with +// FACEDETECT_LIBRARY (mirrors the VOICEDETECT_LIBRARY / PARAKEET_LIBRARY +// convention in the sibling backends); the default looks for the .so next to +// this binary (resolved via LD_LIBRARY_PATH by run.sh). +import ( + "flag" + "fmt" + "os" + + "github.com/ebitengine/purego" + grpc "github.com/mudler/LocalAI/pkg/grpc" +) + +var ( + addr = flag.String("addr", "localhost:50051", "the address to connect to") +) + +type LibFuncs struct { + FuncPtr any + Name string +} + +func main() { + libName := os.Getenv("FACEDETECT_LIBRARY") + if libName == "" { + libName = "libfacedetect.so" + } + + lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL) + if err != nil { + panic(fmt.Errorf("face-detect: dlopen %q: %w", libName, err)) + } + + // Bound 1:1 to facedetect_capi.h. char*/float* returns are registered as + // uintptr so the raw pointer can be freed via the matching capi free fn. + libFuncs := []LibFuncs{ + {&CppAbiVersion, "facedetect_capi_abi_version"}, + {&CppLoad, "facedetect_capi_load"}, + {&CppFree, "facedetect_capi_free"}, + {&CppLastError, "facedetect_capi_last_error"}, + {&CppFreeString, "facedetect_capi_free_string"}, + {&CppFreeVec, "facedetect_capi_free_vec"}, + {&CppEmbedPath, "facedetect_capi_embed_path"}, + {&CppEmbedRGB, "facedetect_capi_embed_rgb"}, + {&CppDetectJSON, "facedetect_capi_detect_path_json"}, + {&CppVerifyPaths, "facedetect_capi_verify_paths"}, + {&CppAnalyzeJSON, "facedetect_capi_analyze_path_json"}, + } + for _, lf := range libFuncs { + purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name) + } + + fmt.Fprintf(os.Stderr, "[face-detect] ABI=%d\n", CppAbiVersion()) + + flag.Parse() + + if err := grpc.StartServer(*addr, &FaceDetect{}); err != nil { + panic(err) + } +} diff --git a/backend/go/face-detect/options.go b/backend/go/face-detect/options.go new file mode 100644 index 000000000..51951bfd7 --- /dev/null +++ b/backend/go/face-detect/options.go @@ -0,0 +1,47 @@ +package main + +import ( + "strconv" + "strings" +) + +// defaultVerifyThreshold is the cosine-distance cutoff used when a request does +// not set one. Matches the insightface buffalo_l ArcFace R50 default the Python +// face backend ships with so the two implementations agree on verdicts out of +// the box. +const defaultVerifyThreshold float32 = 0.35 + +// loadOptions holds the parsed model-level options for face-detect. +type loadOptions struct { + verifyThreshold float32 + modelName string +} + +func splitOption(o string) (key, value string, ok bool) { + i := strings.Index(o, ":") + if i < 0 { + return "", "", false + } + return strings.TrimSpace(o[:i]), strings.TrimSpace(o[i+1:]), true +} + +// parseOptions reads the backend "key:value" option slice. Unknown keys are +// ignored. Defaults: verify_threshold 0.35, model_name derived from the file. +func parseOptions(opts []string) loadOptions { + o := loadOptions{verifyThreshold: defaultVerifyThreshold} + for _, oo := range opts { + key, value, ok := splitOption(oo) + if !ok { + continue + } + switch key { + case "verify_threshold", "threshold": + if f, err := strconv.ParseFloat(value, 32); err == nil && f > 0 { + o.verifyThreshold = float32(f) + } + case "model_name": + o.modelName = value + } + } + return o +} diff --git a/backend/go/face-detect/package.sh b/backend/go/face-detect/package.sh new file mode 100644 index 000000000..36ffa8993 --- /dev/null +++ b/backend/go/face-detect/package.sh @@ -0,0 +1,68 @@ +#!/bin/bash +# +# Bundle the face-detect-grpc binary, libfacedetect.so, the core runtime libs +# (libc/libstdc++/libgomp + ld.so) and the GPU runtime for the active BUILD_TYPE +# so the package is self-contained. Mirrors backend/go/voice-detect/package.sh; +# run.sh routes the (CGO_ENABLED=0) binary through lib/ld.so so the packaged libc +# is used instead of the host's. + +set -e + +CURDIR=$(dirname "$(realpath "$0")") +REPO_ROOT="${CURDIR}/../../.." + +mkdir -p "$CURDIR/package/lib" + +cp -avf "$CURDIR/face-detect-grpc" "$CURDIR/package/" +cp -avf "$CURDIR/run.sh" "$CURDIR/package/" + +# libfacedetect.so + any soname symlinks. purego.Dlopen resolves it via +# LD_LIBRARY_PATH, which run.sh points at lib/. +cp -avf "$CURDIR"/libfacedetect.so* "$CURDIR/package/lib/" 2>/dev/null || { + echo "ERROR: libfacedetect.so not found in $CURDIR, run 'make' first" >&2 + exit 1 +} + +# Detect architecture and copy the core runtime libs libfacedetect.so links +# against, plus the matching dynamic loader as lib/ld.so. +if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then + echo "Detected x86_64 architecture, copying x86_64 libraries..." + cp -arfLv /lib64/ld-linux-x86-64.so.2 "$CURDIR/package/lib/ld.so" + cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 "$CURDIR/package/lib/libc.so.6" + cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 "$CURDIR/package/lib/libgcc_s.so.1" + cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 "$CURDIR/package/lib/libstdc++.so.6" + cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 "$CURDIR/package/lib/libm.so.6" + cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 "$CURDIR/package/lib/libgomp.so.1" + cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 "$CURDIR/package/lib/libdl.so.2" + cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 "$CURDIR/package/lib/librt.so.1" + cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 "$CURDIR/package/lib/libpthread.so.0" +elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then + echo "Detected ARM64 architecture, copying ARM64 libraries..." + cp -arfLv /lib/ld-linux-aarch64.so.1 "$CURDIR/package/lib/ld.so" + cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 "$CURDIR/package/lib/libc.so.6" + cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 "$CURDIR/package/lib/libgcc_s.so.1" + cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 "$CURDIR/package/lib/libstdc++.so.6" + cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 "$CURDIR/package/lib/libm.so.6" + cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 "$CURDIR/package/lib/libgomp.so.1" + cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 "$CURDIR/package/lib/libdl.so.2" + cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 "$CURDIR/package/lib/librt.so.1" + cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 "$CURDIR/package/lib/libpthread.so.0" +elif [ "$(uname -s)" = "Darwin" ]; then + echo "Detected Darwin" +else + echo "Error: Could not detect architecture" + exit 1 +fi + +# Package GPU libraries (CUDA/ROCm/Intel/Vulkan loader + ICDs + drivers) based on +# BUILD_TYPE so the backend can reach the GPU without the runtime base image +# shipping those drivers. +GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh" +if [ -f "$GPU_LIB_SCRIPT" ]; then + echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..." + source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib" + package_gpu_libs +fi + +echo "Packaging completed successfully" +ls -liah "$CURDIR/package/" "$CURDIR/package/lib/" diff --git a/backend/go/face-detect/run.sh b/backend/go/face-detect/run.sh new file mode 100644 index 000000000..a6cc59034 --- /dev/null +++ b/backend/go/face-detect/run.sh @@ -0,0 +1,16 @@ +#!/bin/bash +set -e + +CURDIR=$(dirname "$(realpath "$0")") + +export LD_LIBRARY_PATH="$CURDIR/lib:$CURDIR:${LD_LIBRARY_PATH:-}" + +# If a self-contained ld.so was packaged, route through it so the packaged +# libc / libstdc++ are used instead of the host's (matches the voice-detect / +# whisper / parakeet backends' runtime layout). +if [ -f "$CURDIR/lib/ld.so" ]; then + echo "Using lib/ld.so" + exec "$CURDIR/lib/ld.so" "$CURDIR/face-detect-grpc" "$@" +fi + +exec "$CURDIR/face-detect-grpc" "$@" diff --git a/backend/go/face-detect/test.sh b/backend/go/face-detect/test.sh new file mode 100644 index 000000000..da290c343 --- /dev/null +++ b/backend/go/face-detect/test.sh @@ -0,0 +1,15 @@ +#!/bin/bash +set -e + +CURDIR=$(dirname "$(realpath "$0")") +cd "$CURDIR" + +echo "Running face-detect backend tests..." + +# The pure-Go parsing specs always run. The embed/detect/verify/analyze smoke +# specs run only when a model + image are provided via +# FACEDETECT_BACKEND_TEST_MODEL and FACEDETECT_BACKEND_TEST_IMAGE; otherwise they +# auto-skip. +LD_LIBRARY_PATH="$CURDIR:${LD_LIBRARY_PATH:-}" go test -v -timeout 1200s . + +echo "face-detect tests completed." diff --git a/backend/go/voice-detect/.gitignore b/backend/go/voice-detect/.gitignore new file mode 100644 index 000000000..812afb9b2 --- /dev/null +++ b/backend/go/voice-detect/.gitignore @@ -0,0 +1,18 @@ +# Fetched upstream sources +sources/ + +# CMake build directories +build*/ + +# build artifacts staged in-tree by the Makefile (cp from sources/) or +# symlinked for local dev; the real sources live in voice-detect.cpp upstream. +*.so +*.so.* +voicedetect_capi.h +compile_commands.json + +# Compiled backend binary +voice-detect-grpc + +# Packaging output +package/ diff --git a/backend/go/voice-detect/Makefile b/backend/go/voice-detect/Makefile new file mode 100644 index 000000000..f172da4e6 --- /dev/null +++ b/backend/go/voice-detect/Makefile @@ -0,0 +1,107 @@ +# voice-detect backend Makefile. +# +# Upstream pin lives below as VOICEDETECT_VERSION?=3d51077... (.github/bump_deps.sh +# can find and update it - matches the parakeet.cpp / whisper.cpp / ds4 convention). +# +# Local dev shortcut: if you already have an out-of-tree voice-detect.cpp build, +# symlink the .so + header into this directory and skip the clone/cmake steps: +# +# ln -sf /path/to/voice-detect.cpp/build-shared/libvoicedetect.so . +# ln -sf /path/to/voice-detect.cpp/include/voicedetect_capi.h . +# go build -o voice-detect-grpc . +# +# The default target below does the proper clone-at-pin + cmake build so CI does +# not need a side-checkout. + +VOICEDETECT_VERSION?=3d510772357538c5182808ac7de2278b84824e24 +VOICEDETECT_REPO?=https://github.com/mudler/voice-detect.cpp + +GOCMD?=go +GO_TAGS?= +JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4) + +BUILD_TYPE?= +NATIVE?=false + +# Resolve the target arch. The backend matrix / Docker build pass TARGETARCH +# (amd64|arm64); fall back to uname -m (aarch64|x86_64) for a local build. +RECON_ARCH?=$(or $(TARGETARCH),$(shell uname -m)) + +# Build ggml statically into libvoicedetect.so (PIC) so the shared lib is +# self-contained: dlopen needs no libggml*.so alongside it, only system libs +# (libstdc++/libgomp/libc) that the runtime image already provides. +CMAKE_ARGS?=-DCMAKE_BUILD_TYPE=Release -DVOICEDETECT_SHARED=ON -DVOICEDETECT_BUILD_CLI=OFF -DVOICEDETECT_BUILD_TESTS=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON + +ifeq ($(NATIVE),false) + CMAKE_ARGS+=-DGGML_NATIVE=OFF +endif + +# voice-detect.cpp gates its GGML backends behind VOICEDETECT_GGML_* options and +# does set(GGML_CUDA ${VOICEDETECT_GGML_CUDA} CACHE BOOL "" FORCE), so a bare +# -DGGML_CUDA=ON is overwritten back to OFF. Forward the VOICEDETECT_GGML_* +# options instead. (openblas is not gated, so -DGGML_BLAS passes through.) +ifeq ($(BUILD_TYPE),cublas) + CMAKE_ARGS+=-DVOICEDETECT_GGML_CUDA=ON + # Opt-in cuDNN implicit-GEMM conv path (kills im2col on GPU, reaches + # torch-cuDNN parity). Only the arm64 + CUDA 13 image (GB10/Jetson/L4T) + # ships libcudnn9 + the -dev headers, so gate cuDNN to that variant. + # x86 CUDA images carry no cuDNN -> enabling it there is a link failure. + ifeq ($(CUDA_MAJOR_VERSION),13) + ifneq (,$(filter arm64 aarch64,$(RECON_ARCH))) + CMAKE_ARGS+=-DVOICEDETECT_GGML_CUDNN=ON + endif + endif +else ifeq ($(BUILD_TYPE),openblas) + CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS +else ifeq ($(BUILD_TYPE),hipblas) + CMAKE_ARGS+=-DVOICEDETECT_GGML_HIP=ON +else ifeq ($(BUILD_TYPE),vulkan) + CMAKE_ARGS+=-DVOICEDETECT_GGML_VULKAN=ON +else ifeq ($(BUILD_TYPE),metal) + CMAKE_ARGS+=-DVOICEDETECT_GGML_METAL=ON +endif + +.PHONY: voice-detect-grpc package build clean purge test all + +all: voice-detect-grpc + +# Clone the upstream voice-detect.cpp source at the pinned commit. Directory acts +# as the target so make only re-clones when missing. After a VOICEDETECT_VERSION +# bump, run 'make purge && make' to refetch. +sources/voice-detect.cpp: + mkdir -p sources/voice-detect.cpp + cd sources/voice-detect.cpp && \ + git init -q && \ + git remote add origin $(VOICEDETECT_REPO) && \ + git fetch --depth 1 origin $(VOICEDETECT_VERSION) && \ + git checkout FETCH_HEAD && \ + git submodule update --init --recursive --depth 1 --single-branch + +# Build the shared lib + header out-of-tree, then stage them next to the Go +# sources so purego.Dlopen("libvoicedetect.so") and the cgo-less build both pick +# them up. +libvoicedetect.so: sources/voice-detect.cpp + cmake -B sources/voice-detect.cpp/build-shared -S sources/voice-detect.cpp $(CMAKE_ARGS) + cmake --build sources/voice-detect.cpp/build-shared --config Release -j$(JOBS) --target voicedetect + cp -fv sources/voice-detect.cpp/build-shared/libvoicedetect.so* ./ 2>/dev/null || true + cp -fv sources/voice-detect.cpp/include/voicedetect_capi.h ./ + +voice-detect-grpc: libvoicedetect.so main.go govoicedetect.go options.go + CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o voice-detect-grpc . + +package: voice-detect-grpc + bash package.sh + +build: package + +# Test target. The embed/verify/analyze smoke specs are gated on +# VOICEDETECT_BACKEND_TEST_MODEL + VOICEDETECT_BACKEND_TEST_WAV; without them the +# heavy specs auto-skip and only the pure-Go parsing specs run. +test: + LD_LIBRARY_PATH=$(CURDIR):$$LD_LIBRARY_PATH $(GOCMD) test ./... -count=1 + +clean: purge + rm -rf libvoicedetect.so* voicedetect_capi.h package voice-detect-grpc + +purge: + rm -rf sources/voice-detect.cpp diff --git a/backend/go/voice-detect/govoicedetect.go b/backend/go/voice-detect/govoicedetect.go new file mode 100644 index 000000000..2bbe74bd0 --- /dev/null +++ b/backend/go/voice-detect/govoicedetect.go @@ -0,0 +1,273 @@ +package main + +import ( + "encoding/json" + "errors" + "fmt" + "math" + "os" + "path/filepath" + "strconv" + "strings" + "time" + "unsafe" + + "github.com/mudler/LocalAI/pkg/grpc/base" + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + "github.com/mudler/xlog" +) + +// purego-bound entry points from libvoicedetect.so. Names match +// voicedetect_capi.h exactly so a `nm libvoicedetect.so | grep voicedetect_capi` +// is enough to spot drift. +// +// The opaque ctx and the malloc'd char*/float* return values are declared as +// uintptr so we get the raw pointer back and can release it via the matching +// capi free function. purego's native string/[]float32 returns would copy and +// forget the original pointer, leaking the C-owned buffer on every call. +var ( + CppAbiVersion func() int32 + CppLoad func(ggufPath string) uintptr + CppFree func(ctx uintptr) + CppLastError func(ctx uintptr) string + CppFreeString func(s uintptr) + CppFreeVec func(v uintptr) + CppEmbedPath func(ctx uintptr, wavPath string, outVec, outDim unsafe.Pointer) int32 + CppEmbedPCM func(ctx uintptr, pcm []float32, nSamples, sampleRate int32, outVec, outDim unsafe.Pointer) int32 + CppVerifyPaths func(ctx uintptr, a, b string, threshold float32, outDistance, outVerified unsafe.Pointer) int32 + CppAnalyzeJSON func(ctx uintptr, wavPath string) uintptr +) + +// VoiceDetect implements the speaker-recognition voice subset of the Backend +// gRPC service over libvoicedetect.so. The C side keeps a single loaded model +// plus a per-ctx last-error buffer and is not reentrant, so base.SingleThread +// serializes every call. +type VoiceDetect struct { + base.SingleThread + opts loadOptions + ctxPtr uintptr +} + +func (v *VoiceDetect) Load(opts *pb.ModelOptions) error { + model := opts.ModelFile + if model == "" { + model = opts.ModelPath + } + if !filepath.IsAbs(model) && opts.ModelPath != "" { + model = filepath.Join(opts.ModelPath, model) + } + if model == "" { + return errors.New("voice-detect: ModelFile is required") + } + + v.opts = parseOptions(opts.Options) + if v.opts.modelName == "" { + v.opts.modelName = filepath.Base(model) + } + + // Propagate LocalAI's per-model thread budget to the engine. LocalAI spawns + // one backend process per model and serves requests concurrently, so the + // engine's own min(hardware_concurrency, 8) default can oversubscribe cores. + // VOICEDETECT_THREADS is read by the engine at backend construction, so it + // must be set before the capi load. A non-positive Threads means "unset": + // leave the env alone so the engine keeps its sane default. + threads := opts.Threads + if threads > 0 { + if err := os.Setenv("VOICEDETECT_THREADS", strconv.Itoa(int(threads))); err != nil { + return fmt.Errorf("voice-detect: set VOICEDETECT_THREADS: %w", err) + } + xlog.Info("voice-detect: applying LocalAI thread budget", "threads", threads) + } + + xlog.Info("voice-detect: loading model", "model", model, + "verify_threshold", v.opts.verifyThreshold, "abi", CppAbiVersion()) + + ctx := CppLoad(model) + if ctx == 0 { + // The last-error buffer lives on the ctx that was never returned, so + // surface the path the operator tried to load instead. + return fmt.Errorf("voice-detect: voicedetect_capi_load failed for %q", model) + } + v.ctxPtr = ctx + return nil +} + +// VoiceEmbed returns the L2-normalized speaker embedding for an audio clip. +// The request carries a filesystem PATH; the HTTP layer materializes +// base64/URL/data-URI inputs to a temp file before the gRPC call. +func (v *VoiceDetect) VoiceEmbed(req *pb.VoiceEmbedRequest) (pb.VoiceEmbedResponse, error) { + if v.ctxPtr == 0 { + return pb.VoiceEmbedResponse{}, errors.New("voice-detect: model not loaded") + } + if req.Audio == "" { + return pb.VoiceEmbedResponse{}, errors.New("voice-detect: audio path is required") + } + emb, err := v.embedPath(req.Audio) + if err != nil { + return pb.VoiceEmbedResponse{}, err + } + return pb.VoiceEmbedResponse{Embedding: emb, Model: v.opts.modelName}, nil +} + +func (v *VoiceDetect) embedPath(path string) ([]float32, error) { + var vec uintptr + var dim int32 + rc := CppEmbedPath(v.ctxPtr, path, unsafe.Pointer(&vec), unsafe.Pointer(&dim)) + if rc != 0 || vec == 0 || dim <= 0 { + return nil, v.lastErr("embed", path) + } + defer CppFreeVec(vec) + // Copy out of the C-owned malloc'd buffer before freeing it. The + // uintptr->Pointer conversion trips vet's unsafeptr check, which can't tell + // a C heap pointer from Go-managed memory; safe here, the GC neither tracks + // nor moves this buffer and we copy immediately. + src := unsafe.Slice((*float32)(unsafe.Pointer(vec)), int(dim)) //nolint:govet // C-owned malloc'd vector, copied out before free + out := make([]float32, int(dim)) + copy(out, src) + return out, nil +} + +// VoiceVerify embeds two clips and reports whether they are the same speaker by +// cosine distance against a threshold. A request threshold <= 0 falls back to +// the model-configured default (verify_threshold option, 0.25 if unset). +func (v *VoiceDetect) VoiceVerify(req *pb.VoiceVerifyRequest) (pb.VoiceVerifyResponse, error) { + if v.ctxPtr == 0 { + return pb.VoiceVerifyResponse{}, errors.New("voice-detect: model not loaded") + } + if req.Audio1 == "" || req.Audio2 == "" { + return pb.VoiceVerifyResponse{}, errors.New("voice-detect: audio1 and audio2 are required") + } + + threshold := req.Threshold + if threshold <= 0 { + threshold = v.opts.verifyThreshold + } + + started := time.Now() + var distance float32 + var verified int32 + rc := CppVerifyPaths(v.ctxPtr, req.Audio1, req.Audio2, threshold, + unsafe.Pointer(&distance), unsafe.Pointer(&verified)) + if rc != 0 { + return pb.VoiceVerifyResponse{}, v.lastErr("verify", req.Audio1+","+req.Audio2) + } + elapsedMs := float32(time.Since(started).Seconds() * 1000.0) + + // Confidence decays linearly from 100 at distance 0 to 0 at the threshold, + // matching the Python speaker-recognition backend's reporting. + confidence := float32(0) + if threshold > 0 { + confidence = float32(math.Max(0, math.Min(100, (1.0-float64(distance)/float64(threshold))*100.0))) + } + + return pb.VoiceVerifyResponse{ + Verified: verified != 0, + Distance: distance, + Threshold: threshold, + Confidence: confidence, + Model: v.opts.modelName, + ProcessingTimeMs: elapsedMs, + }, nil +} + +// VoiceAnalyze runs the age/gender/emotion heads on a single clip. The C-API +// always evaluates every supported head, so the request's actions filter is +// advisory and the full analysis is returned as a single segment (the engine +// does not produce time-bounded segments). +func (v *VoiceDetect) VoiceAnalyze(req *pb.VoiceAnalyzeRequest) (pb.VoiceAnalyzeResponse, error) { + if v.ctxPtr == 0 { + return pb.VoiceAnalyzeResponse{}, errors.New("voice-detect: model not loaded") + } + if req.Audio == "" { + return pb.VoiceAnalyzeResponse{}, errors.New("voice-detect: audio path is required") + } + + ptr := CppAnalyzeJSON(v.ctxPtr, req.Audio) + if ptr == 0 { + return pb.VoiceAnalyzeResponse{}, v.lastErr("analyze", req.Audio) + } + defer CppFreeString(ptr) + + seg, err := parseAnalyzeJSON(goStringFromCPtr(ptr)) + if err != nil { + return pb.VoiceAnalyzeResponse{}, fmt.Errorf("voice-detect: analyze JSON for %q: %w", req.Audio, err) + } + return pb.VoiceAnalyzeResponse{Segments: []*pb.VoiceAnalysis{seg}}, nil +} + +// analyzeJSON mirrors the document returned by voicedetect_capi_analyze_path_json: +// +// {"age":42.0, +// "gender":{"label":"female","female":0.88,"male":0.12}, +// "emotion":{"label":"neutral","scores":{"neutral":0.7, ...}}} +// +// gender is a mixed object (a "label" string plus per-class float scores), so +// it is decoded into raw messages and split in parseAnalyzeJSON. +type analyzeJSON struct { + Age float32 `json:"age"` + Gender map[string]json.RawMessage `json:"gender"` + Emotion struct { + Label string `json:"label"` + Scores map[string]float32 `json:"scores"` + } `json:"emotion"` +} + +// parseAnalyzeJSON maps the engine's analyze document onto a VoiceAnalysis. +// start/end stay 0: the model emits a single whole-utterance result, not +// time-bounded segments. +func parseAnalyzeJSON(doc string) (*pb.VoiceAnalysis, error) { + var a analyzeJSON + if err := json.Unmarshal([]byte(doc), &a); err != nil { + return nil, err + } + + seg := &pb.VoiceAnalysis{ + Age: a.Age, + DominantEmotion: a.Emotion.Label, + Emotion: a.Emotion.Scores, + } + + if len(a.Gender) > 0 { + gender := make(map[string]float32, len(a.Gender)) + for k, raw := range a.Gender { + if k == "label" { + _ = json.Unmarshal(raw, &seg.DominantGender) + continue + } + var score float32 + if err := json.Unmarshal(raw, &score); err == nil { + gender[k] = score + } + } + seg.Gender = gender + } + + return seg, nil +} + +// lastErr wraps the C-API's per-ctx last-error buffer into a Go error. +func (v *VoiceDetect) lastErr(op, subject string) error { + msg := strings.TrimSpace(CppLastError(v.ctxPtr)) + if msg == "" { + msg = "no error detail" + } + return fmt.Errorf("voice-detect: %s failed for %q: %s", op, subject, msg) +} + +// goStringFromCPtr copies a NUL-terminated C string into Go memory. cptr is a +// malloc'd buffer the caller owns; release it via CppFreeString after the copy. +// +// The uintptr->Pointer conversion trips vet's unsafeptr check, which can't tell +// a C heap pointer from Go-managed memory. Safe here: the GC neither tracks nor +// moves the buffer and we dereference it immediately to copy the bytes out. +func goStringFromCPtr(cptr uintptr) string { + if cptr == 0 { + return "" + } + p := unsafe.Pointer(cptr) //nolint:govet // C-owned malloc'd buffer, not Go-GC memory (see doc above) + n := 0 + for *(*byte)(unsafe.Add(p, n)) != 0 { + n++ + } + return string(unsafe.Slice((*byte)(p), n)) +} diff --git a/backend/go/voice-detect/govoicedetect_test.go b/backend/go/voice-detect/govoicedetect_test.go new file mode 100644 index 000000000..2de7fcc8a --- /dev/null +++ b/backend/go/voice-detect/govoicedetect_test.go @@ -0,0 +1,144 @@ +package main + +import ( + "os" + "sync" + "testing" + + "github.com/ebitengine/purego" + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestVoiceDetect(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "voice-detect Backend Suite") +} + +var ( + libLoadOnce sync.Once + libLoadErr error +) + +// ensureLibLoaded mirrors main.go's bootstrap so a Go test can drive the C-API +// bridge without spinning up the gRPC server. Records the error (the smoke +// specs skip themselves) when libvoicedetect.so is not loadable from cwd +// (LD_LIBRARY_PATH or a symlink in ./). +func ensureLibLoaded() error { + libLoadOnce.Do(func() { + libName := os.Getenv("VOICEDETECT_LIBRARY") + if libName == "" { + libName = "libvoicedetect.so" + } + lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL) + if err != nil { + libLoadErr = err + return + } + purego.RegisterLibFunc(&CppAbiVersion, lib, "voicedetect_capi_abi_version") + purego.RegisterLibFunc(&CppLoad, lib, "voicedetect_capi_load") + purego.RegisterLibFunc(&CppFree, lib, "voicedetect_capi_free") + purego.RegisterLibFunc(&CppLastError, lib, "voicedetect_capi_last_error") + purego.RegisterLibFunc(&CppFreeString, lib, "voicedetect_capi_free_string") + purego.RegisterLibFunc(&CppFreeVec, lib, "voicedetect_capi_free_vec") + purego.RegisterLibFunc(&CppEmbedPath, lib, "voicedetect_capi_embed_path") + purego.RegisterLibFunc(&CppEmbedPCM, lib, "voicedetect_capi_embed_pcm") + purego.RegisterLibFunc(&CppVerifyPaths, lib, "voicedetect_capi_verify_paths") + purego.RegisterLibFunc(&CppAnalyzeJSON, lib, "voicedetect_capi_analyze_path_json") + }) + return libLoadErr +} + +var _ = Describe("parseOptions", func() { + It("defaults verify_threshold to 0.25", func() { + o := parseOptions(nil) + Expect(o.verifyThreshold).To(Equal(float32(0.25))) + Expect(o.modelName).To(Equal("")) + }) + + It("parses verify_threshold, threshold alias and model_name", func() { + o := parseOptions([]string{"verify_threshold:0.4", "model_name:ecapa", "unknown:x"}) + Expect(o.verifyThreshold).To(Equal(float32(0.4))) + Expect(o.modelName).To(Equal("ecapa")) + + o2 := parseOptions([]string{"threshold:0.3"}) + Expect(o2.verifyThreshold).To(Equal(float32(0.3))) + }) + + It("ignores non-positive thresholds and keeps the default", func() { + o := parseOptions([]string{"verify_threshold:0", "threshold:-1"}) + Expect(o.verifyThreshold).To(Equal(float32(0.25))) + }) +}) + +var _ = Describe("parseAnalyzeJSON", func() { + It("maps age, gender label+scores and emotion label+scores", func() { + doc := `{"age":42.0, + "gender":{"label":"female","female":0.88,"male":0.12}, + "emotion":{"label":"neutral","scores":{"neutral":0.7,"happy":0.2,"sad":0.1}}}` + seg, err := parseAnalyzeJSON(doc) + Expect(err).ToNot(HaveOccurred()) + Expect(seg.Age).To(BeNumerically("~", 42.0, 1e-4)) + Expect(seg.Start).To(Equal(float32(0))) + Expect(seg.End).To(Equal(float32(0))) + + Expect(seg.DominantGender).To(Equal("female")) + Expect(seg.Gender).To(HaveKeyWithValue("female", BeNumerically("~", 0.88, 1e-4))) + Expect(seg.Gender).To(HaveKeyWithValue("male", BeNumerically("~", 0.12, 1e-4))) + // The "label" entry is consumed into DominantGender, not the score map. + Expect(seg.Gender).ToNot(HaveKey("label")) + + Expect(seg.DominantEmotion).To(Equal("neutral")) + Expect(seg.Emotion).To(HaveKeyWithValue("neutral", BeNumerically("~", 0.7, 1e-4))) + Expect(seg.Emotion).To(HaveKeyWithValue("happy", BeNumerically("~", 0.2, 1e-4))) + }) + + It("tolerates a missing gender block", func() { + seg, err := parseAnalyzeJSON(`{"age":30.0,"emotion":{"label":"happy","scores":{"happy":1.0}}}`) + Expect(err).ToNot(HaveOccurred()) + Expect(seg.DominantGender).To(Equal("")) + Expect(seg.DominantEmotion).To(Equal("happy")) + }) + + It("returns an error on malformed JSON", func() { + _, err := parseAnalyzeJSON(`{not-json`) + Expect(err).To(HaveOccurred()) + }) +}) + +// The specs below exercise the real C-API end to end. They run only when both a +// model GGUF and a test WAV are provided, and skip cleanly otherwise so the +// suite stays green without large assets. +var _ = Describe("VoiceDetect end-to-end", Ordered, func() { + var ( + v *VoiceDetect + modelPath = os.Getenv("VOICEDETECT_BACKEND_TEST_MODEL") + wavPath = os.Getenv("VOICEDETECT_BACKEND_TEST_WAV") + ) + + BeforeAll(func() { + if modelPath == "" || wavPath == "" { + Skip("set VOICEDETECT_BACKEND_TEST_MODEL and VOICEDETECT_BACKEND_TEST_WAV to run the e2e specs") + } + if err := ensureLibLoaded(); err != nil { + Skip("libvoicedetect.so not loadable: " + err.Error()) + } + v = &VoiceDetect{} + Expect(v.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed()) + }) + + It("embeds an audio clip", func() { + resp, err := v.VoiceEmbed(&pb.VoiceEmbedRequest{Audio: wavPath}) + Expect(err).ToNot(HaveOccurred()) + Expect(resp.Embedding).ToNot(BeEmpty()) + Expect(resp.Model).ToNot(BeEmpty()) + }) + + It("verifies a clip against itself as the same speaker", func() { + resp, err := v.VoiceVerify(&pb.VoiceVerifyRequest{Audio1: wavPath, Audio2: wavPath}) + Expect(err).ToNot(HaveOccurred()) + Expect(resp.Verified).To(BeTrue()) + Expect(resp.Distance).To(BeNumerically("<=", resp.Threshold)) + }) +}) diff --git a/backend/go/voice-detect/main.go b/backend/go/voice-detect/main.go new file mode 100644 index 000000000..35421b5c3 --- /dev/null +++ b/backend/go/voice-detect/main.go @@ -0,0 +1,64 @@ +package main + +// Started internally by LocalAI - one gRPC server per loaded model. +// +// Loads libvoicedetect.so via purego and registers the flat C-API entry points +// declared in voicedetect_capi.h. The library name can be overridden with +// VOICEDETECT_LIBRARY (mirrors the PARAKEET_LIBRARY / OMNIVOICE_LIBRARY +// convention in the sibling backends); the default looks for the .so next to +// this binary (resolved via LD_LIBRARY_PATH by run.sh). +import ( + "flag" + "fmt" + "os" + + "github.com/ebitengine/purego" + grpc "github.com/mudler/LocalAI/pkg/grpc" +) + +var ( + addr = flag.String("addr", "localhost:50051", "the address to connect to") +) + +type LibFuncs struct { + FuncPtr any + Name string +} + +func main() { + libName := os.Getenv("VOICEDETECT_LIBRARY") + if libName == "" { + libName = "libvoicedetect.so" + } + + lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL) + if err != nil { + panic(fmt.Errorf("voice-detect: dlopen %q: %w", libName, err)) + } + + // Bound 1:1 to voicedetect_capi.h. char*/float* returns are registered as + // uintptr so the raw pointer can be freed via the matching capi free fn. + libFuncs := []LibFuncs{ + {&CppAbiVersion, "voicedetect_capi_abi_version"}, + {&CppLoad, "voicedetect_capi_load"}, + {&CppFree, "voicedetect_capi_free"}, + {&CppLastError, "voicedetect_capi_last_error"}, + {&CppFreeString, "voicedetect_capi_free_string"}, + {&CppFreeVec, "voicedetect_capi_free_vec"}, + {&CppEmbedPath, "voicedetect_capi_embed_path"}, + {&CppEmbedPCM, "voicedetect_capi_embed_pcm"}, + {&CppVerifyPaths, "voicedetect_capi_verify_paths"}, + {&CppAnalyzeJSON, "voicedetect_capi_analyze_path_json"}, + } + for _, lf := range libFuncs { + purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name) + } + + fmt.Fprintf(os.Stderr, "[voice-detect] ABI=%d\n", CppAbiVersion()) + + flag.Parse() + + if err := grpc.StartServer(*addr, &VoiceDetect{}); err != nil { + panic(err) + } +} diff --git a/backend/go/voice-detect/options.go b/backend/go/voice-detect/options.go new file mode 100644 index 000000000..c5a6e2595 --- /dev/null +++ b/backend/go/voice-detect/options.go @@ -0,0 +1,46 @@ +package main + +import ( + "strconv" + "strings" +) + +// defaultVerifyThreshold is the cosine-distance cutoff used when a request does +// not set one. Matches the Python speaker-recognition backend's default so the +// two implementations agree on verdicts out of the box. +const defaultVerifyThreshold float32 = 0.25 + +// loadOptions holds the parsed model-level options for voice-detect. +type loadOptions struct { + verifyThreshold float32 + modelName string +} + +func splitOption(o string) (key, value string, ok bool) { + i := strings.Index(o, ":") + if i < 0 { + return "", "", false + } + return strings.TrimSpace(o[:i]), strings.TrimSpace(o[i+1:]), true +} + +// parseOptions reads the backend "key:value" option slice. Unknown keys are +// ignored. Defaults: verify_threshold 0.25, model_name derived from the file. +func parseOptions(opts []string) loadOptions { + o := loadOptions{verifyThreshold: defaultVerifyThreshold} + for _, oo := range opts { + key, value, ok := splitOption(oo) + if !ok { + continue + } + switch key { + case "verify_threshold", "threshold": + if f, err := strconv.ParseFloat(value, 32); err == nil && f > 0 { + o.verifyThreshold = float32(f) + } + case "model_name": + o.modelName = value + } + } + return o +} diff --git a/backend/go/voice-detect/package.sh b/backend/go/voice-detect/package.sh new file mode 100755 index 000000000..de95c8ce2 --- /dev/null +++ b/backend/go/voice-detect/package.sh @@ -0,0 +1,68 @@ +#!/bin/bash +# +# Bundle the voice-detect-grpc binary, libvoicedetect.so, the core runtime libs +# (libc/libstdc++/libgomp + ld.so) and the GPU runtime for the active BUILD_TYPE +# so the package is self-contained. Mirrors backend/go/parakeet-cpp/package.sh; +# run.sh routes the (CGO_ENABLED=0) binary through lib/ld.so so the packaged libc +# is used instead of the host's. + +set -e + +CURDIR=$(dirname "$(realpath "$0")") +REPO_ROOT="${CURDIR}/../../.." + +mkdir -p "$CURDIR/package/lib" + +cp -avf "$CURDIR/voice-detect-grpc" "$CURDIR/package/" +cp -avf "$CURDIR/run.sh" "$CURDIR/package/" + +# libvoicedetect.so + any soname symlinks. purego.Dlopen resolves it via +# LD_LIBRARY_PATH, which run.sh points at lib/. +cp -avf "$CURDIR"/libvoicedetect.so* "$CURDIR/package/lib/" 2>/dev/null || { + echo "ERROR: libvoicedetect.so not found in $CURDIR, run 'make' first" >&2 + exit 1 +} + +# Detect architecture and copy the core runtime libs libvoicedetect.so links +# against, plus the matching dynamic loader as lib/ld.so. +if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then + echo "Detected x86_64 architecture, copying x86_64 libraries..." + cp -arfLv /lib64/ld-linux-x86-64.so.2 "$CURDIR/package/lib/ld.so" + cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 "$CURDIR/package/lib/libc.so.6" + cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 "$CURDIR/package/lib/libgcc_s.so.1" + cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 "$CURDIR/package/lib/libstdc++.so.6" + cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 "$CURDIR/package/lib/libm.so.6" + cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 "$CURDIR/package/lib/libgomp.so.1" + cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 "$CURDIR/package/lib/libdl.so.2" + cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 "$CURDIR/package/lib/librt.so.1" + cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 "$CURDIR/package/lib/libpthread.so.0" +elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then + echo "Detected ARM64 architecture, copying ARM64 libraries..." + cp -arfLv /lib/ld-linux-aarch64.so.1 "$CURDIR/package/lib/ld.so" + cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 "$CURDIR/package/lib/libc.so.6" + cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 "$CURDIR/package/lib/libgcc_s.so.1" + cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 "$CURDIR/package/lib/libstdc++.so.6" + cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 "$CURDIR/package/lib/libm.so.6" + cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 "$CURDIR/package/lib/libgomp.so.1" + cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 "$CURDIR/package/lib/libdl.so.2" + cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 "$CURDIR/package/lib/librt.so.1" + cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 "$CURDIR/package/lib/libpthread.so.0" +elif [ "$(uname -s)" = "Darwin" ]; then + echo "Detected Darwin" +else + echo "Error: Could not detect architecture" + exit 1 +fi + +# Package GPU libraries (CUDA/ROCm/Intel/Vulkan loader + ICDs + drivers) based on +# BUILD_TYPE so the backend can reach the GPU without the runtime base image +# shipping those drivers. +GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh" +if [ -f "$GPU_LIB_SCRIPT" ]; then + echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..." + source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib" + package_gpu_libs +fi + +echo "Packaging completed successfully" +ls -liah "$CURDIR/package/" "$CURDIR/package/lib/" diff --git a/backend/go/voice-detect/run.sh b/backend/go/voice-detect/run.sh new file mode 100755 index 000000000..ea5fef508 --- /dev/null +++ b/backend/go/voice-detect/run.sh @@ -0,0 +1,16 @@ +#!/bin/bash +set -e + +CURDIR=$(dirname "$(realpath "$0")") + +export LD_LIBRARY_PATH="$CURDIR/lib:$CURDIR:${LD_LIBRARY_PATH:-}" + +# If a self-contained ld.so was packaged, route through it so the packaged +# libc / libstdc++ are used instead of the host's (matches the whisper / +# parakeet backends' runtime layout). +if [ -f "$CURDIR/lib/ld.so" ]; then + echo "Using lib/ld.so" + exec "$CURDIR/lib/ld.so" "$CURDIR/voice-detect-grpc" "$@" +fi + +exec "$CURDIR/voice-detect-grpc" "$@" diff --git a/backend/go/voice-detect/test.sh b/backend/go/voice-detect/test.sh new file mode 100755 index 000000000..17addfebf --- /dev/null +++ b/backend/go/voice-detect/test.sh @@ -0,0 +1,14 @@ +#!/bin/bash +set -e + +CURDIR=$(dirname "$(realpath "$0")") +cd "$CURDIR" + +echo "Running voice-detect backend tests..." + +# The pure-Go parsing specs always run. The embed/verify/analyze smoke specs run +# only when a model + WAV are provided via VOICEDETECT_BACKEND_TEST_MODEL and +# VOICEDETECT_BACKEND_TEST_WAV; otherwise they auto-skip. +LD_LIBRARY_PATH="$CURDIR:${LD_LIBRARY_PATH:-}" go test -v -timeout 1200s . + +echo "voice-detect tests completed." diff --git a/backend/index.yaml b/backend/index.yaml index 8c0f1bc8d..1eb02e1ba 100644 --- a/backend/index.yaml +++ b/backend/index.yaml @@ -209,6 +209,78 @@ nvidia-cuda-12: "cuda12-ced" nvidia-l4t-cuda-12: "nvidia-l4t-arm64-ced" nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-ced" +- &voicedetect + name: "voice-detect" + alias: "voice-detect" + license: mit + icon: https://avatars.githubusercontent.com/u/95302084 + description: | + voice-detect speaker recognition and voice analysis. + voice-detect.cpp is a C++/ggml engine that produces L2-normalised + speaker embeddings (ECAPA-TDNN, WeSpeaker ResNet34, 3D-Speaker + ERes2Net, CAM++) for voice verification and 1:N identification, plus + a wav2vec2 age / gender / emotion analysis head. It replaces the + Python speaker-recognition backend and is exposed through the Voice* + gRPC rpcs and the /v1/voice/* REST endpoints. It runs on CPU, NVIDIA + CUDA, AMD ROCm/HIP, Intel SYCL, Vulkan and NVIDIA Jetson (L4T) targets. + urls: + - https://github.com/mudler/voice-detect.cpp + tags: + - voice-recognition + - speaker-verification + - speaker-embedding + - CPU + - GPU + - CUDA + - HIP + capabilities: + default: "cpu-voice-detect" + nvidia: "cuda12-voice-detect" + intel: "intel-sycl-f16-voice-detect" + metal: "metal-voice-detect" + amd: "rocm-voice-detect" + vulkan: "vulkan-voice-detect" + nvidia-l4t: "nvidia-l4t-arm64-voice-detect" + nvidia-cuda-13: "cuda13-voice-detect" + nvidia-cuda-12: "cuda12-voice-detect" + nvidia-l4t-cuda-12: "nvidia-l4t-arm64-voice-detect" + nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-voice-detect" +- &facedetect + name: "face-detect" + alias: "face-detect" + license: mit + icon: https://avatars.githubusercontent.com/u/95302084 + description: | + face-detect face detection, embedding, verification and analysis. + face-detect.cpp is a C++/ggml engine that runs SCRFD / YuNet face + detection and ArcFace / SFace 512-d (or 128-d) L2-normalised face + embeddings for verification and 1:N identification, plus a landmark / + age / gender analysis head. It replaces the Python insightface backend + and is exposed through the Embedding, Detect and Face* gRPC rpcs and + the /v1/face/* REST endpoints. It runs on CPU, NVIDIA CUDA, AMD + ROCm/HIP, Intel SYCL, Vulkan and NVIDIA Jetson (L4T) targets. + urls: + - https://github.com/mudler/face-detect.cpp + tags: + - face-recognition + - face-verification + - face-embedding + - CPU + - GPU + - CUDA + - HIP + capabilities: + default: "cpu-face-detect" + nvidia: "cuda12-face-detect" + intel: "intel-sycl-f16-face-detect" + metal: "metal-face-detect" + amd: "rocm-face-detect" + vulkan: "vulkan-face-detect" + nvidia-l4t: "nvidia-l4t-arm64-face-detect" + nvidia-cuda-13: "cuda13-face-detect" + nvidia-cuda-12: "cuda12-face-detect" + nvidia-l4t-cuda-12: "nvidia-l4t-arm64-face-detect" + nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-face-detect" - &voxtral name: "voxtral" alias: "voxtral" @@ -2827,6 +2899,236 @@ uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-ced" mirrors: - localai/localai-backends:master-gpu-nvidia-cuda-13-ced +## voice-detect +- !!merge <<: *voicedetect + name: "voice-detect-development" + capabilities: + default: "cpu-voice-detect-development" + nvidia: "cuda12-voice-detect-development" + intel: "intel-sycl-f16-voice-detect-development" + metal: "metal-voice-detect-development" + amd: "rocm-voice-detect-development" + vulkan: "vulkan-voice-detect-development" + nvidia-l4t: "nvidia-l4t-arm64-voice-detect-development" + nvidia-cuda-13: "cuda13-voice-detect-development" + nvidia-cuda-12: "cuda12-voice-detect-development" + nvidia-l4t-cuda-12: "nvidia-l4t-arm64-voice-detect-development" + nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-voice-detect-development" +- !!merge <<: *voicedetect + name: "nvidia-l4t-arm64-voice-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-voice-detect" + mirrors: + - localai/localai-backends:latest-nvidia-l4t-arm64-voice-detect +- !!merge <<: *voicedetect + name: "nvidia-l4t-arm64-voice-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-arm64-voice-detect" + mirrors: + - localai/localai-backends:master-nvidia-l4t-arm64-voice-detect +- !!merge <<: *voicedetect + name: "cuda13-nvidia-l4t-arm64-voice-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-voice-detect" + mirrors: + - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-voice-detect +- !!merge <<: *voicedetect + name: "cuda13-nvidia-l4t-arm64-voice-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-voice-detect" + mirrors: + - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-voice-detect +- !!merge <<: *voicedetect + name: "cpu-voice-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-voice-detect" + mirrors: + - localai/localai-backends:latest-cpu-voice-detect +- !!merge <<: *voicedetect + name: "cpu-voice-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-cpu-voice-detect" + mirrors: + - localai/localai-backends:master-cpu-voice-detect +- !!merge <<: *voicedetect + name: "metal-voice-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-voice-detect" + mirrors: + - localai/localai-backends:latest-metal-darwin-arm64-voice-detect +- !!merge <<: *voicedetect + name: "metal-voice-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-voice-detect" + mirrors: + - localai/localai-backends:master-metal-darwin-arm64-voice-detect +- !!merge <<: *voicedetect + name: "cuda12-voice-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-voice-detect" + mirrors: + - localai/localai-backends:latest-gpu-nvidia-cuda-12-voice-detect +- !!merge <<: *voicedetect + name: "cuda12-voice-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-voice-detect" + mirrors: + - localai/localai-backends:master-gpu-nvidia-cuda-12-voice-detect +- !!merge <<: *voicedetect + name: "rocm-voice-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-voice-detect" + mirrors: + - localai/localai-backends:latest-gpu-rocm-hipblas-voice-detect +- !!merge <<: *voicedetect + name: "rocm-voice-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-voice-detect" + mirrors: + - localai/localai-backends:master-gpu-rocm-hipblas-voice-detect +- !!merge <<: *voicedetect + name: "intel-sycl-f32-voice-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-voice-detect" + mirrors: + - localai/localai-backends:latest-gpu-intel-sycl-f32-voice-detect +- !!merge <<: *voicedetect + name: "intel-sycl-f32-voice-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-voice-detect" + mirrors: + - localai/localai-backends:master-gpu-intel-sycl-f32-voice-detect +- !!merge <<: *voicedetect + name: "intel-sycl-f16-voice-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f16-voice-detect" + mirrors: + - localai/localai-backends:latest-gpu-intel-sycl-f16-voice-detect +- !!merge <<: *voicedetect + name: "intel-sycl-f16-voice-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f16-voice-detect" + mirrors: + - localai/localai-backends:master-gpu-intel-sycl-f16-voice-detect +- !!merge <<: *voicedetect + name: "vulkan-voice-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-voice-detect" + mirrors: + - localai/localai-backends:latest-gpu-vulkan-voice-detect +- !!merge <<: *voicedetect + name: "vulkan-voice-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-voice-detect" + mirrors: + - localai/localai-backends:master-gpu-vulkan-voice-detect +- !!merge <<: *voicedetect + name: "cuda13-voice-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-voice-detect" + mirrors: + - localai/localai-backends:latest-gpu-nvidia-cuda-13-voice-detect +- !!merge <<: *voicedetect + name: "cuda13-voice-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-voice-detect" + mirrors: + - localai/localai-backends:master-gpu-nvidia-cuda-13-voice-detect +## face-detect +- !!merge <<: *facedetect + name: "face-detect-development" + capabilities: + default: "cpu-face-detect-development" + nvidia: "cuda12-face-detect-development" + intel: "intel-sycl-f16-face-detect-development" + metal: "metal-face-detect-development" + amd: "rocm-face-detect-development" + vulkan: "vulkan-face-detect-development" + nvidia-l4t: "nvidia-l4t-arm64-face-detect-development" + nvidia-cuda-13: "cuda13-face-detect-development" + nvidia-cuda-12: "cuda12-face-detect-development" + nvidia-l4t-cuda-12: "nvidia-l4t-arm64-face-detect-development" + nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-face-detect-development" +- !!merge <<: *facedetect + name: "nvidia-l4t-arm64-face-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-face-detect" + mirrors: + - localai/localai-backends:latest-nvidia-l4t-arm64-face-detect +- !!merge <<: *facedetect + name: "nvidia-l4t-arm64-face-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-arm64-face-detect" + mirrors: + - localai/localai-backends:master-nvidia-l4t-arm64-face-detect +- !!merge <<: *facedetect + name: "cuda13-nvidia-l4t-arm64-face-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-face-detect" + mirrors: + - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-face-detect +- !!merge <<: *facedetect + name: "cuda13-nvidia-l4t-arm64-face-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-face-detect" + mirrors: + - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-face-detect +- !!merge <<: *facedetect + name: "cpu-face-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-face-detect" + mirrors: + - localai/localai-backends:latest-cpu-face-detect +- !!merge <<: *facedetect + name: "cpu-face-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-cpu-face-detect" + mirrors: + - localai/localai-backends:master-cpu-face-detect +- !!merge <<: *facedetect + name: "metal-face-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-face-detect" + mirrors: + - localai/localai-backends:latest-metal-darwin-arm64-face-detect +- !!merge <<: *facedetect + name: "metal-face-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-face-detect" + mirrors: + - localai/localai-backends:master-metal-darwin-arm64-face-detect +- !!merge <<: *facedetect + name: "cuda12-face-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-face-detect" + mirrors: + - localai/localai-backends:latest-gpu-nvidia-cuda-12-face-detect +- !!merge <<: *facedetect + name: "cuda12-face-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-face-detect" + mirrors: + - localai/localai-backends:master-gpu-nvidia-cuda-12-face-detect +- !!merge <<: *facedetect + name: "rocm-face-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-face-detect" + mirrors: + - localai/localai-backends:latest-gpu-rocm-hipblas-face-detect +- !!merge <<: *facedetect + name: "rocm-face-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-face-detect" + mirrors: + - localai/localai-backends:master-gpu-rocm-hipblas-face-detect +- !!merge <<: *facedetect + name: "intel-sycl-f32-face-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-face-detect" + mirrors: + - localai/localai-backends:latest-gpu-intel-sycl-f32-face-detect +- !!merge <<: *facedetect + name: "intel-sycl-f32-face-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-face-detect" + mirrors: + - localai/localai-backends:master-gpu-intel-sycl-f32-face-detect +- !!merge <<: *facedetect + name: "intel-sycl-f16-face-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f16-face-detect" + mirrors: + - localai/localai-backends:latest-gpu-intel-sycl-f16-face-detect +- !!merge <<: *facedetect + name: "intel-sycl-f16-face-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f16-face-detect" + mirrors: + - localai/localai-backends:master-gpu-intel-sycl-f16-face-detect +- !!merge <<: *facedetect + name: "vulkan-face-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-face-detect" + mirrors: + - localai/localai-backends:latest-gpu-vulkan-face-detect +- !!merge <<: *facedetect + name: "vulkan-face-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-face-detect" + mirrors: + - localai/localai-backends:master-gpu-vulkan-face-detect +- !!merge <<: *facedetect + name: "cuda13-face-detect" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-face-detect" + mirrors: + - localai/localai-backends:latest-gpu-nvidia-cuda-13-face-detect +- !!merge <<: *facedetect + name: "cuda13-face-detect-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-face-detect" + mirrors: + - localai/localai-backends:master-gpu-nvidia-cuda-13-face-detect ## stablediffusion-ggml - !!merge <<: *stablediffusionggml name: "cpu-stablediffusion-ggml" diff --git a/core/config/backend_capabilities.go b/core/config/backend_capabilities.go index cc9567887..d54463a8e 100644 --- a/core/config/backend_capabilities.go +++ b/core/config/backend_capabilities.go @@ -542,6 +542,19 @@ var BackendCapabilities = map[string]BackendCapability{ DefaultUsecases: []string{UsecaseSpeakerRecognition}, Description: "Speaker recognition — voice identity verification and analysis", }, + "voice-detect": { + GRPCMethods: []GRPCMethod{MethodVoiceVerify, MethodVoiceEmbed, MethodVoiceAnalyze}, + PossibleUsecases: []string{UsecaseSpeakerRecognition}, + DefaultUsecases: []string{UsecaseSpeakerRecognition}, + Description: "voice-detect.cpp: C++/ggml speaker embedding, verification and voice analysis (age/gender/emotion)", + }, + "face-detect": { + GRPCMethods: []GRPCMethod{MethodEmbedding, MethodDetect, MethodFaceVerify, MethodFaceAnalyze}, + PossibleUsecases: []string{UsecaseEmbeddings, UsecaseDetection, UsecaseFaceRecognition}, + DefaultUsecases: []string{UsecaseFaceRecognition}, + AcceptsImages: true, + Description: "face-detect.cpp: C++/ggml face detection, embedding, verification and attribute analysis", + }, "silero-vad": { GRPCMethods: []GRPCMethod{MethodVAD}, PossibleUsecases: []string{UsecaseVAD}, diff --git a/docs/content/features/face-recognition.md b/docs/content/features/face-recognition.md index ecc3e7213..7bddc702f 100644 --- a/docs/content/features/face-recognition.md +++ b/docs/content/features/face-recognition.md @@ -7,16 +7,93 @@ url = "/features/face-recognition/" ![Face recognition: 1:N match against a vector store, with an anti-spoofing liveness gate that can veto a verification](/images/diagrams/face-recognition-flow.png) -LocalAI supports face recognition through the `insightface` backend: -face verification (1:1), face identification (1:N) against a built-in -vector store, face embedding, face detection, demographic analysis -(age / gender), and antispoofing / liveness detection. +LocalAI supports face recognition: face verification (1:1), face +identification (1:N) against a built-in vector store, face embedding, +face detection, demographic analysis (age / gender), and antispoofing / +liveness detection. -The backend ships **two interchangeable engines** under one image, each -paired with a distinct gallery entry so users can pick by license and -accuracy needs. +The same `/v1/face/*` HTTP API is served by two backends: -## Licensing — read this first +- **`face-detect` (recommended, default).** A standalone C++/ggml + engine ([face-detect.cpp](https://github.com/mudler/face-detect.cpp)): + no Python, no onnxruntime, no torch runtime. Each gallery entry is a + single self-describing GGUF. This is the recommended option for new + deployments. +- **`insightface` (Python).** The original ONNX Runtime backend. Still + supported; see [the Python backend](#insightface-python-backend) below. + +Both backends expose the identical wire format, so the API examples in +this page work with either - only the gallery entry name (the `model` +field) changes. + +## face-detect (ggml) backend + +The `face-detect` backend reads the detector and recognizer architecture +(`facedetect.arch`) directly from the GGUF metadata, so installing a +gallery entry is all that is needed to select an engine. It drives the +Embeddings / Detect / FaceVerify / FaceAnalyze gRPC rpcs behind the +`/v1/face/{embed,verify,analyze,detect,register,identify,forget}` +endpoints. + +### Licensing - read this first + +| Gallery entry | Detector + recognizer | Embedding dim | License | +|---|---|---|---| +| `face-detect-buffalo-l` | SCRFD-10GF + ArcFace R50 + GenderAge | 512 | **Non-commercial research only** (upstream insightface weights) | +| `face-detect-buffalo-m` | SCRFD-2.5GF + ArcFace R50 + GenderAge | 512 | **Non-commercial research only** | +| `face-detect-buffalo-s` | SCRFD-500MF + MBF + GenderAge | 512 | **Non-commercial research only** | +| `face-detect-yunet-sface` | YuNet + SFace (OpenCV Zoo) | 128 | **Apache 2.0 - commercial-safe** | + +The insightface buffalo packs (buffalo_l / buffalo_m / buffalo_s) are +released by the upstream maintainers for **non-commercial research use +only**. Pick the `face-detect-yunet-sface` entry for production / +commercial deployments. + +### Quickstart + +Install the commercial-safe entry (recommended for copy-paste): + +```bash +local-ai models install face-detect-yunet-sface +``` + +Verify that two images depict the same person: + +```bash +curl -sX POST http://localhost:8080/v1/face/verify \ + -H "Content-Type: application/json" \ + -d '{ + "model": "face-detect-yunet-sface", + "img1": "https://example.com/alice_1.jpg", + "img2": "https://example.com/alice_2.jpg" + }' +``` + +Detect faces and analyze demographics (buffalo entries populate +age / gender; YuNet + SFace returns regions only): + +```bash +curl -sX POST http://localhost:8080/v1/face/detect \ + -H "Content-Type: application/json" \ + -d '{"model": "face-detect-buffalo-l", "img": "https://example.com/group.jpg"}' + +curl -sX POST http://localhost:8080/v1/face/analyze \ + -H "Content-Type: application/json" \ + -d '{"model": "face-detect-buffalo-l", "img": "https://example.com/alice.jpg"}' +``` + +The 1:N register / identify / forget workflow and the rest of the API +are identical to the [API reference](#api-reference) below - just pass a +`face-detect-*` model name. The per-engine verify thresholds are ~0.35 +for the buffalo ArcFace/MBF recognizers and ~0.363 for SFace. + +## insightface (Python) backend + +The `insightface` backend ships **two interchangeable engines** under +one image, each paired with a distinct gallery entry so users can pick +by license and accuracy needs. + +### Licensing - read this first | Gallery entry | Detector + recognizer | Size | License | |---|---|---|---| diff --git a/docs/content/features/voice-recognition.md b/docs/content/features/voice-recognition.md index 20728a28f..aed5d5bf6 100644 --- a/docs/content/features/voice-recognition.md +++ b/docs/content/features/voice-recognition.md @@ -7,16 +7,92 @@ url = "/features/voice-recognition/" ![Voice recognition: register, identify, and forget voiceprints in a vector store, for 1:1 verify or 1:N identify](/images/diagrams/voice-recognition-flow.png) -LocalAI supports voice (speaker) recognition through the -`speaker-recognition` backend: speaker verification (1:1), speaker -identification (1:N) against a built-in vector store, speaker -embedding, and demographic analysis (age / gender / emotion from -voice). +LocalAI supports voice (speaker) recognition: speaker verification +(1:1), speaker identification (1:N) against a built-in vector store, +speaker embedding, and demographic analysis (age / gender / emotion +from voice). The audio analog to [Face Recognition](/features/face-recognition/), -following the same two-engine pattern under one image. +served over the same `/v1/voice/*` HTTP API by two backends: -## Engines +- **`voice-detect` (recommended, default).** A standalone C++/ggml + engine ([voice-detect.cpp](https://github.com/mudler/voice-detect.cpp)): + no Python, no onnxruntime, no torch runtime. Each gallery entry is a + single self-describing GGUF. This is the recommended option for new + deployments. +- **`speaker-recognition` (Python).** The original SpeechBrain / ONNX + backend. Still supported; see [the Python backend](#speaker-recognition-python-backend) + below. + +Both backends expose the identical wire format, so the API examples on +this page work with either - only the gallery entry name (the `model` +field) changes. + +## voice-detect (ggml) backend + +The `voice-detect` backend reads the embedding (or analysis) +architecture (`voicedetect.arch`) directly from the GGUF metadata, so +installing a gallery entry is all that is needed to select an engine. It +drives the VoiceEmbed / VoiceVerify / VoiceAnalyze gRPC rpcs behind the +`/v1/voice/{embed,verify,analyze,register,identify,forget}` endpoints. + +### Gallery entries + +| Gallery entry | Model | Embedding dim | License | +|---|---|---|---| +| `voice-detect-ecapa-tdnn` | SpeechBrain ECAPA-TDNN (VoxCeleb) | 192 | **Apache 2.0 - commercial-safe** | +| `voice-detect-wespeaker-resnet34` | WeSpeaker ResNet34 (VoxCeleb) | 256 | CC-BY-4.0 | +| `voice-detect-eres2net` | 3D-Speaker ERes2Net (VoxCeleb) | 192 | **Apache 2.0 - commercial-safe** | +| `voice-detect-campplus` | 3D-Speaker CAM++ (VoxCeleb) | 192 | **Apache 2.0 - commercial-safe** | +| `voice-detect-emotion-wav2vec2` | audEERING wav2vec2 (age / gender / emotion) | analyze head | **CC-BY-NC-SA-4.0 - non-commercial** | + +The four speaker-recognition entries drive verify / embed / identify. +`voice-detect-emotion-wav2vec2` is the analysis head behind +`/v1/voice/analyze` (continuous age estimate plus gender and emotion +class scores) and is **non-commercial / research use only**. + +### Quickstart + +Install the default entry (recommended for copy-paste): + +```bash +local-ai models install voice-detect-ecapa-tdnn +``` + +Verify that two audio clips were spoken by the same person: + +```bash +curl -sX POST http://localhost:8080/v1/voice/verify \ + -H "Content-Type: application/json" \ + -d '{ + "model": "voice-detect-ecapa-tdnn", + "audio1": "https://example.com/alice_1.wav", + "audio2": "https://example.com/alice_2.wav" + }' +``` + +Analyze age / gender / emotion (install the analyze entry first): + +```bash +local-ai models install voice-detect-emotion-wav2vec2 + +curl -sX POST http://localhost:8080/v1/voice/analyze \ + -H "Content-Type: application/json" \ + -d '{"model": "voice-detect-emotion-wav2vec2", "audio": "https://example.com/alice.wav"}' +``` + +The 1:N register / identify / forget workflow and the rest of the API +are identical to the [API reference](#api-reference) below - just pass a +`voice-detect-*` model name. The default verify threshold is ~0.25 for +the ECAPA-TDNN / ERes2Net / CAM++ recognizers and ~0.30 for WeSpeaker +ResNet34. + +## speaker-recognition (Python) backend + +The `speaker-recognition` backend follows the same two-engine pattern +under one image. + +### Engines | Gallery entry | Model | Size | License | |---|---|---|---| diff --git a/docs/content/reference/compatibility-table.md b/docs/content/reference/compatibility-table.md index 21971ff45..0e9551b3b 100644 --- a/docs/content/reference/compatibility-table.md +++ b/docs/content/reference/compatibility-table.md @@ -97,6 +97,8 @@ All backends listed here can be installed on demand from the [Backend Gallery]({ | [locate-anything.cpp](https://github.com/mudler/locate-anything.cpp) | Open-vocabulary object detection and visual grounding (LocateAnything-3B) in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T | | [depth-anything.cpp](https://github.com/mudler/depth-anything.cpp) | Depth Anything 3 monocular metric depth + camera pose in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T | | [sam3.cpp](https://github.com/PABannier/sam3.cpp) | Segment Anything (SAM 3/2/EdgeTAM) with text/point/box prompts in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T | +| [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Native face detection, recognition, embedding, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace) in C/C++ using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T | +| [voice-detect.cpp](https://github.com/mudler/voice-detect.cpp) | Native speaker (voice) recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2) in C/C++ using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T | | [insightface](https://github.com/deepinsight/insightface) | Face verification, embedding, and anti-spoofing liveness (ONNX Runtime) | CPU, CUDA 12 | | [speaker-recognition](https://speechbrain.github.io/) | Speaker (voice) recognition via SpeechBrain ECAPA-TDNN | CPU, CUDA 12, Metal | diff --git a/gallery/index.yaml b/gallery/index.yaml index f39993333..a52568405 100644 --- a/gallery/index.yaml +++ b/gallery/index.yaml @@ -9060,6 +9060,248 @@ - filename: MiniFASNetV1SE.onnx sha256: ebab7f90c7833fbccd46d3a555410e78d969db5438e169b6524be444862b3676 uri: https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV1SE.onnx +- name: face-detect-buffalo-l + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/face-detect.cpp + - https://github.com/deepinsight/insightface + description: | + Face recognition with insightface's `buffalo_l` pack (SCRFD-10GF + detector + ResNet50 ArcFace 512-d embedder), ported to C++/ggml and + shipped as a single GGUF for the `face-detect` backend. Highest + accuracy of the buffalo line. + + No Python / onnxruntime / torch runtime: face-detect.cpp reads the + detector and embedder architecture (`facedetect.arch`) directly from + the GGUF metadata, so installing this entry is all that is needed to + select buffalo_l. Drives the Embedding / Detect / FaceVerify / + FaceAnalyze gRPC rpcs and the /v1/face/{verify,analyze,embed,detect} + REST endpoints. This GGUF also embeds the MiniFASNet anti-spoof + ensemble, available via the FaceVerify `anti_spoof` request flag. + NON-COMMERCIAL RESEARCH USE ONLY: for commercial use see + `face-detect-yunet-sface`. + license: insightface-non-commercial + icon: https://avatars.githubusercontent.com/u/53104118?s=200&v=4 + tags: + - face-recognition + - face-verification + - face-embedding + - research-only + - gpu + - cpu + last_checked: "2026-06-22" + overrides: + backend: face-detect + known_usecases: + - face_recognition + - detection + - embeddings + options: + - verify_threshold:0.35 + parameters: + model: face-detect-buffalo-l.gguf + files: + - filename: face-detect-buffalo-l.gguf + sha256: 6ed070f6e569beeed542ddd5603bcbc9eb8ea57f728f7d8013d6a90b2b952116 + uri: https://huggingface.co/mudler/face-detect-gguf/resolve/main/buffalo_l.gguf +- name: face-detect-buffalo-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/face-detect.cpp + - https://github.com/deepinsight/insightface + description: | + Face recognition with insightface's `buffalo_m` pack (SCRFD-2.5GF + detector + ResNet50 ArcFace embedder), converted to a C++/ggml GGUF + for the `face-detect` backend. Same recognition accuracy as + `buffalo_l` with a cheaper detector: a good balance on mid-range + hardware. + + The architecture (`facedetect.arch`) is read from the GGUF metadata, + so this entry alone selects the buffalo_m engine. This GGUF also + embeds the MiniFASNet anti-spoof ensemble, available via the + FaceVerify `anti_spoof` request flag. NON-COMMERCIAL RESEARCH USE + ONLY. + license: insightface-non-commercial + icon: https://avatars.githubusercontent.com/u/53104118?s=200&v=4 + tags: + - face-recognition + - face-verification + - face-embedding + - research-only + - gpu + - cpu + last_checked: "2026-06-22" + overrides: + backend: face-detect + known_usecases: + - face_recognition + - detection + - embeddings + options: + - verify_threshold:0.35 + parameters: + model: face-detect-buffalo-m.gguf + files: + - filename: face-detect-buffalo-m.gguf + sha256: 0f7527eeb97b88719bf7e11e43ab8af6f05999357d767f8dde53db3c586c1c3f + uri: https://huggingface.co/mudler/face-detect-gguf/resolve/main/buffalo_m.gguf +- name: face-detect-buffalo-s + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/face-detect.cpp + - https://github.com/deepinsight/insightface + description: | + Face recognition with insightface's `buffalo_s` pack (SCRFD-500MF + detector + MBF 512-d embedder), converted to a C++/ggml GGUF for the + `face-detect` backend. Small and CPU-friendly: a good fit for + mid-range and edge deployments. + + The architecture (`facedetect.arch`) is read from the GGUF metadata, + so this entry alone selects the buffalo_s engine. This GGUF also + embeds the MiniFASNet anti-spoof ensemble, available via the + FaceVerify `anti_spoof` request flag. NON-COMMERCIAL RESEARCH USE + ONLY. + license: insightface-non-commercial + icon: https://avatars.githubusercontent.com/u/53104118?s=200&v=4 + tags: + - face-recognition + - face-verification + - face-embedding + - research-only + - edge + - cpu + last_checked: "2026-06-22" + overrides: + backend: face-detect + known_usecases: + - face_recognition + - detection + - embeddings + options: + - verify_threshold:0.35 + parameters: + model: face-detect-buffalo-s.gguf + files: + - filename: face-detect-buffalo-s.gguf + sha256: 7490b1efbc8746b188a5aef0adf5e3d1a2dc9607abd474018893f95571999969 + uri: https://huggingface.co/mudler/face-detect-gguf/resolve/main/buffalo_s.gguf +- name: face-detect-buffalo-sc + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/face-detect.cpp + - https://github.com/deepinsight/insightface + description: | + Face recognition with insightface's `buffalo_sc` pack (SCRFD-500M + detector + a small ArcFace embedder), converted to a C++/ggml GGUF + for the `face-detect` backend. This is the smallest insightface + pack: the lightest option for low-resource and edge deployments. + + The architecture (`facedetect.arch`) is read from the GGUF metadata, + so this entry alone selects the buffalo_sc engine. If this GGUF + embeds the MiniFASNet anti-spoof ensemble, it is available via the + FaceVerify `anti_spoof` request flag. NON-COMMERCIAL RESEARCH USE + ONLY. + license: insightface-non-commercial + icon: https://avatars.githubusercontent.com/u/53104118?s=200&v=4 + tags: + - face-recognition + - face-verification + - face-embedding + - research-only + - edge + - cpu + last_checked: "2026-06-22" + overrides: + backend: face-detect + known_usecases: + - face_recognition + - detection + - embeddings + options: + - verify_threshold:0.35 + parameters: + model: face-detect-buffalo-sc.gguf + files: + - filename: face-detect-buffalo-sc.gguf + sha256: f754c0e32d5efbbc53d7efca13be2807676bf5db20a8594ef96b32afa2c482b1 + uri: https://huggingface.co/mudler/face-detect-gguf/resolve/main/buffalo_sc.gguf +- name: face-detect-antelopev2 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/face-detect.cpp + - https://github.com/deepinsight/insightface + description: | + Face recognition with insightface's `antelopev2` pack (SCRFD-10G + detector + ArcFace glint360k R100, 512-d embedder), converted to a + C++/ggml GGUF for the `face-detect` backend. The higher-accuracy + insightface pack: heavier, but the best fit when recognition + quality matters more than speed. + + The architecture (`facedetect.arch`) is read from the GGUF metadata, + so this entry alone selects the antelopev2 engine. If this GGUF + embeds the MiniFASNet anti-spoof ensemble, it is available via the + FaceVerify `anti_spoof` request flag. NON-COMMERCIAL RESEARCH USE + ONLY. + license: insightface-non-commercial + icon: https://avatars.githubusercontent.com/u/53104118?s=200&v=4 + tags: + - face-recognition + - face-verification + - face-embedding + - research-only + last_checked: "2026-06-22" + overrides: + backend: face-detect + known_usecases: + - face_recognition + - detection + - embeddings + options: + - verify_threshold:0.35 + parameters: + model: face-detect-antelopev2.gguf + files: + - filename: face-detect-antelopev2.gguf + sha256: 245e657e51754fbf075dd43d80a80a2d14a60c2fc42a3220f63eef17a315e96c + uri: https://huggingface.co/mudler/face-detect-gguf/resolve/main/antelopev2.gguf +- name: face-detect-yunet-sface + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/face-detect.cpp + - https://github.com/opencv/opencv_zoo + description: | + Face recognition with OpenCV Zoo weights: YuNet detector + SFace + 128-d recognizer, converted to a C++/ggml GGUF for the `face-detect` + backend. APACHE 2.0: safe for commercial use. Lower accuracy than the + buffalo packs and no demographic head, but the commercial-friendly + alternative to the insightface buffalo line. + + The architecture (`facedetect.arch`) is read from the GGUF metadata, + so this entry alone selects the YuNet + SFace engine. + license: apache-2.0 + icon: https://avatars.githubusercontent.com/u/95302084 + tags: + - face-recognition + - face-verification + - face-embedding + - commercial-ok + - gpu + - cpu + last_checked: "2026-06-22" + overrides: + backend: face-detect + known_usecases: + - face_recognition + - detection + - embeddings + options: + - verify_threshold:0.363 + parameters: + model: face-detect-yunet-sface.gguf + files: + - filename: face-detect-yunet-sface.gguf + sha256: 9ce78d4ba0ae9d5e8c91a0e145d511558d1d90f5d9c1f4131cca9bb4bce60902 + uri: https://huggingface.co/mudler/face-detect-gguf/resolve/main/yunet-sface.gguf - name: speechbrain-ecapa-tdnn url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: @@ -9129,6 +9371,217 @@ - filename: wespeaker_voxceleb_resnet34.onnx sha256: 7bb2f06e9df17cdf1ef14ee8a15ab08ed28e8d0ef5054ee135741560df2ec068 uri: https://huggingface.co/Wespeaker/wespeaker-voxceleb-resnet34-LM/resolve/main/voxceleb_resnet34_LM.onnx +- name: voice-detect-ecapa-tdnn + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/voice-detect.cpp + - https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb + description: | + Speaker (voice) recognition with SpeechBrain's ECAPA-TDNN trained + on VoxCeleb, ported to C++/ggml and shipped as a single GGUF for the + `voice-detect` backend. 192-d L2-normalised embeddings, ~1.9% Equal + Error Rate on VoxCeleb1-O. APACHE 2.0 - commercial-safe. + + No Python / torch runtime: voice-detect.cpp reads the embedding + architecture (`voicedetect.arch`) directly from the GGUF metadata, + so installing this entry is all that is needed to select ECAPA-TDNN. + Drives the VoiceVerify / VoiceEmbed gRPC rpcs and the + /v1/voice/{verify,embed,register,identify,forget} REST endpoints. + license: apache-2.0 + icon: https://avatars.githubusercontent.com/u/95302084 + tags: + - voice-recognition + - speaker-verification + - speaker-embedding + - commercial-ok + - cpu + - gpu + last_checked: "2026-06-22" + overrides: + backend: voice-detect + known_usecases: + - speaker_recognition + options: + - verify_threshold:0.25 + parameters: + model: voice-detect-ecapa-tdnn-voxceleb.gguf + files: + - filename: voice-detect-ecapa-tdnn-voxceleb.gguf + sha256: 68046a1fdfb7843f460962db4739fbd381cc5c3ab93d1505e75e2f4c0dc19b8f + uri: https://huggingface.co/mudler/voice-detect-gguf/resolve/main/ecapa-tdnn-voxceleb.gguf +- name: voice-detect-wespeaker-resnet34 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/voice-detect.cpp + - https://github.com/wenet-e2e/wespeaker + description: | + Speaker recognition with WeSpeaker's ResNet34 trained on VoxCeleb, + converted to a C++/ggml GGUF for the `voice-detect` backend. 256-d + embeddings, CPU-friendly and runtime-free (no onnxruntime or torch). + CC-BY-4.0. + + Use when you want WeSpeaker's ResNet34 topology instead of + ECAPA-TDNN. The embedding architecture (`voicedetect.arch`) is read + from the GGUF metadata, so this entry alone selects the engine. + license: cc-by-4.0 + icon: https://avatars.githubusercontent.com/u/95302084 + tags: + - voice-recognition + - speaker-verification + - speaker-embedding + - commercial-ok + - edge + - cpu + last_checked: "2026-06-22" + overrides: + backend: voice-detect + known_usecases: + - speaker_recognition + options: + - verify_threshold:0.25 + parameters: + model: voice-detect-wespeaker-resnet34.gguf + files: + - filename: voice-detect-wespeaker-resnet34.gguf + sha256: 72040372494eafec299836bc1977cfc13c603cb486674ed59b0f4c03758d29da + uri: https://huggingface.co/mudler/voice-detect-gguf/resolve/main/wespeaker-resnet34-voxceleb.gguf +- name: voice-detect-eres2net + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/voice-detect.cpp + - https://huggingface.co/iic/speech_eres2net_sv_en_voxceleb_16k + description: | + Speaker recognition with 3D-Speaker's ERes2Net trained on VoxCeleb, + converted to a C++/ggml GGUF for the `voice-detect` backend. + 192-d embeddings with strong verification accuracy. APACHE 2.0. + + The embedding architecture (`voicedetect.arch`) is read from the + GGUF metadata, so this entry alone selects the ERes2Net engine. + license: apache-2.0 + icon: https://avatars.githubusercontent.com/u/95302084 + tags: + - voice-recognition + - speaker-verification + - speaker-embedding + - commercial-ok + - cpu + - gpu + last_checked: "2026-06-22" + overrides: + backend: voice-detect + known_usecases: + - speaker_recognition + options: + - verify_threshold:0.25 + parameters: + model: voice-detect-eres2net.gguf + files: + - filename: voice-detect-eres2net.gguf + sha256: d39f53c7a4d39734740a86a07521b9a819ee8ea56c1a9436eba611ab733a3d06 + uri: https://huggingface.co/mudler/voice-detect-gguf/resolve/main/eres2net-base-zh-cn.gguf +- name: voice-detect-campplus + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/voice-detect.cpp + - https://huggingface.co/iic/speech_campplus_sv_en_voxceleb_16k + description: | + Speaker recognition with 3D-Speaker's CAM++ trained on VoxCeleb, + converted to a C++/ggml GGUF for the `voice-detect` backend. 192-d + embeddings, a fast context-aware masking topology well-suited to + CPU and edge deployments. APACHE 2.0. + + The embedding architecture (`voicedetect.arch`) is read from the + GGUF metadata, so this entry alone selects the CAM++ engine. + license: apache-2.0 + icon: https://avatars.githubusercontent.com/u/95302084 + tags: + - voice-recognition + - speaker-verification + - speaker-embedding + - commercial-ok + - edge + - cpu + last_checked: "2026-06-22" + overrides: + backend: voice-detect + known_usecases: + - speaker_recognition + options: + - verify_threshold:0.25 + parameters: + model: voice-detect-campplus.gguf + files: + - filename: voice-detect-campplus.gguf + sha256: a6e34c6d230cff26e37b71a2df0907fde1de425654e28d9d5cacca32e02a13d3 + uri: https://huggingface.co/mudler/voice-detect-gguf/resolve/main/campplus-zh-cn.gguf +- name: voice-detect-emotion-wav2vec2 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://github.com/mudler/voice-detect.cpp + - https://huggingface.co/audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim + description: | + Voice analysis (age / gender / emotion) with audEERING's wav2vec2 + model, converted to a C++/ggml GGUF for the `voice-detect` backend. + Drives the VoiceAnalyze gRPC rpc and the /v1/voice/analyze REST + endpoint, returning a continuous age estimate plus gender and + emotion class scores for a single utterance. CC-BY-NC-SA-4.0 - + research / non-commercial use only. + + The analysis architecture (`voicedetect.arch`) is read from the + GGUF metadata, so this entry alone selects the wav2vec2 analyze head. + license: cc-by-nc-sa-4.0 + icon: https://avatars.githubusercontent.com/u/95302084 + tags: + - voice-recognition + - voice-analysis + - emotion-recognition + - cpu + - gpu + last_checked: "2026-06-22" + overrides: + backend: voice-detect + known_usecases: + - speaker_recognition + parameters: + model: voice-detect-emotion-wav2vec2.gguf + files: + - filename: voice-detect-emotion-wav2vec2.gguf + sha256: 9e9793e4f77a27f4ae068bcb29c2b6fe2f74881799e2cfea0f8e436ad3765e50 + uri: https://huggingface.co/mudler/voice-detect-gguf/resolve/main/emotion-wav2vec2-superb-er.gguf +- name: voice-detect-age-gender-wav2vec2 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://huggingface.co/audeering/wav2vec2-large-robust-24-ft-age-gender + - https://github.com/mudler/voice-detect.cpp + description: | + wav2vec2-large-robust age + gender analysis head + (audeering/wav2vec2-large-robust-24-ft-age-gender), converted to a + C++/ggml GGUF for the `voice-detect` backend. Drives the VoiceAnalyze + gRPC rpc and the /v1/voice/analyze REST endpoint, returning a + continuous age estimate plus gender class scores for a single + utterance. CC-BY-NC-SA-4.0 - research / non-commercial use only. + + The analysis architecture (`voicedetect.arch`) is read from the + GGUF metadata, so this entry alone selects the wav2vec2 analyze head. + license: cc-by-nc-sa-4.0 + icon: https://avatars.githubusercontent.com/u/95302084 + tags: + - voice-recognition + - voice-analysis + - research-only + - cpu + - gpu + last_checked: "2026-06-22" + overrides: + backend: voice-detect + known_usecases: + - speaker_recognition + parameters: + model: voice-detect-age-gender-wav2vec2.gguf + files: + - filename: voice-detect-age-gender-wav2vec2.gguf + sha256: d92486b3f1ea7baf6a90f1026b7b8e9848b3a8332bccfb01cc8889eed7069064 + uri: https://huggingface.co/mudler/voice-detect-gguf/resolve/main/age-gender-wav2vec2-audeering.gguf - name: rfdetr-base url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: