mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 17:18:49 -04:00
37c773c384ea9835a01eeb7208fc64db270fd616
479
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 mudler-agentandEttore Di Giacinto
|
2f4eaf1fad |
fix(sound): require tool calls for expansion (#12783)
Require one expand_sound call instead of parsing assistant prose. Reject unsupported templates and invalid arguments before audio generation. Keep installed configuration unchanged and preserve explicit controls. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
dce91ba7aa |
fix(backends): prefer installed development builds (#12782)
* fix(ci): capture and surface ginkgo errors in run-coverage.sh The coverage script ran ginkgo per test root with `|| fail=1`, which swallowed the exit code. Combined with `make --output-sync=target`, which buffers all output and flushes at the end, the actual ginkgo error was invisible in interleaved CI logs. Every test suite passed but the script still exited 1, and the error message was buried in 22 000 lines of buffered output. Each ginkgo invocation now captures combined stdout+stderr to a per-root log file. On failure the last 80 lines are printed to stderr so the error is visible in the CI log. On success the log is replayed to stdout so test output remains visible. The build flags (--tags, --coverpkg) are quoted correctly so values containing spaces (e.g. --tags="debug auth") are preserved through eval. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): create output dir before writing ginkgo logs The previous commit added per-root log files but forgot to create the output directory before the first ginkgo invocation. On CI the coverage/ dir did not exist yet, so the shell redirect to cover-pkg.out.log failed with "No such file or directory" before ginkgo even ran. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(core/http): add canvas preview route to auth coverage allowlist Commit |
||
|
|
36307005a9 |
feat(chat-history): server-side chat history with tombstones (follow-up to #9902) (#12663)
* feat(schema): add Conversation schema for chat history (#9432) Persist WebUI chat conversations server-side so browser refresh, private windows, or device changes preserve user history (issue #9432). This commit adds the on-disk representation: - Conversation holds id, name, model, opaque history, MCP / sampling settings, and timestamps. - ConversationsFile is the per-user list serialised to JSON. History is stored as json.RawMessage so the server stays agnostic to React message shapes (user / assistant / thinking / tool_call / tool_result mixed with text / image_url / audio_url / file attachments). Assisted-by: Claude:claude-opus-4-7 Signed-off-by: TLoE419 <tloemizuchizu@gmail.com> * feat(chathistory): add file-based per-user conversation store (#9432) File-backed persister for chat history. Each user's conversations live under {baseDir}/{userID}/conversations.json (anonymous/ when auth is disabled). Design choices: - In-memory cache backed by sync.Mutex so concurrent saves don't interleave writes. - Atomic write via tmp file + os.Rename so a crash mid-save never leaves a corrupted history. - ID validation regex blocks path-traversal payloads at the store boundary, in addition to the auth context constraint. Tests cover round-trip persistence across instances, user isolation, unsafe-ID rejection, bulk replace migration, and the anonymous fallback path. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: TLoE419 <tloemizuchizu@gmail.com> * feat(api): expose chat conversation CRUD endpoints (#9432) Wire the chathistory store into the HTTP layer. New endpoints under /api/conversations: - GET /api/conversations list - POST /api/conversations upsert (id in body) - DELETE /api/conversations delete all - PUT /api/conversations/bulk replace entire set (localStorage migration) - GET /api/conversations/:id fetch one - PUT /api/conversations/:id upsert (id in path; path id wins over body) - DELETE /api/conversations/:id delete one All endpoints scope queries to the authenticated user via getUserID(c) - callers cannot impersonate other users by passing a user_id in the body. The endpoints are gated behind the new chat_history feature permission (default ON in APIFeatures), registered through the existing RouteFeatureRegistry so the unified feature middleware picks them up automatically. Application.ChatHistoryStore() returns nil when DisableWebUI is set or no persistence path is configured, in which case route registration is skipped entirely rather than registering handlers that always 503. Also adds the chat-history instruction entry and Swagger tag so the endpoint surfaces in /api/instructions and /swagger. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: TLoE419 <tloemizuchizu@gmail.com> * feat(react-ui): sync chat history to server with localStorage fallback (#9432) useChat now probes /api/conversations on mount. When the endpoint responds 200, the server becomes the authoritative source: the hook merges remote conversations into local state and pushes per-chat PUT updates on each debounced save. When the endpoint 404s - older LocalAI deploys or the feature disabled - the hook silently keeps the old localStorage-only behaviour, so the change is backward-compatible. Migration: when the server is reachable but empty and the browser has local conversations with history, the hook fires a single PUT /api/conversations/bulk to seed the server. The atomic bulk endpoint avoids partial-upload states where a retry would skip already-uploaded entries. Delete operations propagate to the server when it's available; failures are silently swallowed so the local UI stays responsive on transient network errors. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: TLoE419 <tloemizuchizu@gmail.com> * test(chathistory): rewrite tests in Ginkgo/Gomega The initial test file used the stdlib testing package (t.Fatalf / t.Errorf), which is forbidden by the forbidigo linter rule in .golangci.yml — LocalAI mandates Ginkgo/Gomega for all Go tests (see .agents/coding-style.md). Restructured the same six scenarios into Describe / Context / It blocks, with a chathistory_suite_test.go bootstrap that registers the Ginkgo fail handler. Test coverage is unchanged: basic CRUD round-trip, cross- instance persistence, user isolation, unsafe ID rejection (now via DescribeTable), bulk replace overwrite, and the anonymous fallback path. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: TLoE419 <tloemizuchizu@gmail.com> * refactor(chathistory): switch store to GORM-backed authDB Addresses mudler's review on #9902. Chat history was the only per-user state in LocalAI sitting in a separate filesystem store instead of the shared GORM database — AgentStore and JobStore both live there. Move it onto Application.authDB so per-user state stays in one place. * ConversationRecord (GORM model) with composite primary key (user_id, conv_id) — React mints IDs locally with no global coordination so per-user uniqueness is the natural shape. * Store.New(db *gorm.DB) replaces Store.New(baseDir string). Save / Get / List / Delete / DeleteAll / ReplaceAll keep their signatures so HTTP handlers and the React UI are untouched. * ReplaceAll runs in a transaction with an Unscoped() pre-clear so a future retention sweep cannot resurrect a stale ID on re-upload. * gorm.DeletedAt in the schema means a future retention pruner is a one-line query — addresses the second half of the review (unbounded in-memory map). The map is gone: reads go straight to the DB. Trade-off worth flagging in the PR reply: chat history persistence now requires auth.Enabled = true (that's what initialises Application.authDB). With auth disabled the React UI transparently falls back to localStorage, the same path the original PR took on 404. Consistent with AgentStore / JobStore; if a "chat history always on" mode is preferred we can pull DB init out of auth.InitDB into a shared helper. Assisted-by: Claude:claude-opus-4-7 [Read Edit Bash] Signed-off-by: TLoE419 <tloemizuchizu@gmail.com> * refactor(schema): drop now-unused ConversationsFile envelope ConversationsFile was the JSON wrapper for the old file-based chat history store — a {Conversations: [...], UpdatedAt: ...} pair serialised to disk under {baseDir}/{userID}/conversations.json. The previous commit replaced that store with a GORM-backed one that writes rows directly as ConversationRecord, leaving the envelope without a consumer. Assisted-by: Claude:claude-opus-4-7 [Read Edit Bash] Signed-off-by: TLoE419 <tloemizuchizu@gmail.com> * test(chathistory): switch tests to testutil.SetupTestDB Mirror the AgentStore test setup: spin up a PostgreSQL 16 container via testutil.SetupTestDB(), construct a Store on top of it, exercise CRUD round-trips. The previous file-based fixtures (TempDir, filepath.Join checks) are no longer meaningful now that the store talks GORM. Tests are skipped on macOS (testcontainers needs Docker) and run on CI, same as the other GORM-backed suites in the repo. Also reframed two cases: * "unsafe IDs" → "malformed IDs" — the DB-backed store doesn't care about path traversal, but idRegex still rejects whitespace / control characters so IDs stay safe in logs and HTTP responses. * "anonymous user" no longer pins a directory path (no directory exists anymore); instead it checks the round-trip plus the per-user isolation guarantee. Added a DeleteAll round-trip to cover the bulk-clear endpoint. Assisted-by: Claude:claude-opus-4-7 [Read Edit Bash] Signed-off-by: TLoE419 <tloemizuchizu@gmail.com> * test(chathistory): cover the conversation HTTP handlers The seven /api/conversations handlers had no tests. Inject the *chathistory.Store into the handlers instead of the whole Application and move the route table into RegisterChatHistoryRoutes, so a spec can drive the real registration and middleware chain (auth.Middleware plus the chat_history feature gate) against an in-memory SQLite auth database. The new specs cover 401 without credentials on every route, 403 when an admin revoked chat_history, the CRUD round trip, path id over body id, 400 on malformed ids and bodies, per-user isolation, bulk replace and delete-all, and that a nil store registers nothing (the UI then falls back to localStorage). The handlers' nil-store branches are gone: the routes are only registered with a store, so those branches could never run. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(chathistory): make deletions final with tombstones and versions A conversation deleted on one device came back as soon as a client with a stale copy wrote it again: Delete soft-deleted the row, and the next Save went through GORM's upsert, which cleared deleted_at and resurrected it for every device. The lookup that recovered CreatedAt also ran outside any transaction, so concurrent writers raced on it. The server is now authoritative over deletions: * Delete turns the row into a tombstone: content dropped, deleted_at set. It is idempotent, and deleting an id the server has not seen yet still records a tombstone, so a delete that overtakes the UI's in-flight debounced save of the same chat still wins. * Save runs in a transaction, refuses a tombstoned id with ErrDeleted (HTTP 410 Gone), and bumps the version with a single UPDATE ... SET version = version + 1 guarded by deleted_at IS NULL, so concurrent writers each count and a delete committed between the lookup and the write still wins. Two concurrent first writes of the same id retry once (ON CONFLICT DO NOTHING, then update). * Every row carries a server-assigned version (1 on the first save, +1 per write including the delete), returned to the client. A non-zero version tells the UI the server has seen a conversation, which is how it can tell a never-synced draft from a server-side deletion. * GET /api/conversations returns the tombstones under "deleted". * DeleteAll and ReplaceAll leave tombstones too. ReplaceAll no longer hard-deletes everything first; it upserts the payload, skips ids that were deleted (reported in "skipped"), and tombstones the rest. * createdAt/updatedAt are stored as the client sends them (only zero values are stamped), so clients compare updatedAt from their own clocks rather than against the server's. The store specs now run against PostgreSQL (testcontainers) and, with the auth tag, SQLite through auth.InitDB, since both back the auth database. The API specs cover a stale client reconnecting after a remote deletion. Swagger is regenerated with swag v1.16.6 (json.RawMessage fields get swaggertype), and the feature is documented, including that it needs user auth. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(react-ui): let the server decide which chats were deleted mergeRemoteAndLocal kept every chat that was only in the browser and let the server copy win for every shared id. A stale browser therefore uploaded chats deleted on another device as if they were new drafts, and a message sent while the list request was still in flight was replaced by the older server copy and then lost from localStorage too. The first load now reconciles with the server's tombstones and versions: * a chat the server lists under "deleted" is dropped locally; * a chat on both sides keeps the copy with the newer updatedAt, so an edit made during the load wins and is uploaded afterwards; * a chat only in the browser is uploaded only if it was never synced (no version). With a version the server has seen it and no longer has it, so it is dropped instead of resurrected (this also keeps one account's cached chats out of another account on a shared browser); * deletes made while the list is loading are queued and sent once it arrives. Saves record the version the server returns, and a 410 on save drops the chat locally instead of retrying it forever. Chats received from the server count as already sent, so adopting a server copy no longer echoes it straight back. The bulk localStorage migration is gone: it replaced the whole server set from a possibly stale browser, and the per-chat saves already upload never-synced drafts through the tombstone check. The Playwright spec drives each case against a stubbed /api/conversations that follows the server contract. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(chathistory): keep the LocalAI Assistant toggle on synced chats master's createNewChat sets localaiAssistant, but serializeChat did not send it and schema.Conversation had no field for it, so a chat with the Assistant mode on came back with it off after a reload from the server (and the local cache dropped it too). Add the field on both sides. The other fields createNewChat sets were already covered; a handler spec now round-trips all of them, and the UI spec checks the toggle on the way up and in the local cache. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(react-ui): scope chat cache per account, retry unconfirmed deletes Two gaps in the client half of server-side chat history, both found in review: The browser cache was one shared localStorage key. A draft that one account could not upload (a failed first save) stayed in it, and after that account signed out and another signed in, the reconcile kept the unversioned draft and the save pass uploaded it into the second account. The cache now lives under localai_chats_data:<user id>, Chat remounts its state when the account changes, and nothing is synced without a signed-in account. The shared key is left for browser-only use without auth and is not uploaded when signed in, because nothing tells which account it belongs to. Deletes were dropped locally and their DELETE errors swallowed; only deletes made while the first list was loading were queued, and not persisted. A delete made while the server was unreachable therefore came back with the next list. Deletes (single, delete all, and deletes on the Home page) are now kept as pending deletes in the account's cache until the server confirms them, override the server list in the reconcile, and are sent again on the next change, on the online event and on the next load. While a delete-all is pending no conversation is uploaded, so a chat started after it is not caught by it. The Playwright specs replay both review scenarios (account switch after a failed first save; delete and delete-all while offline) with counter-checks that the same account keeps and uploads its own draft and that a confirmed delete is neither re-sent nor resurrected. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(react-ui): persist cleared chats immediately Navigation cancels the debounced cache write and can restore unsynced chats after Clear all. Save the cleared list with its pending deletion before returning, and cover immediate navigation in the browser spec. Assisted-by: Codex:GPT-6 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * test(react-ui): make the chat history sync specs deterministic - "clear all persists before navigation" opened a single chat, and the menu offers no Clear all for one chat, so the spec timed out in every run. It now opens two unsynced chats and checks that neither survives. Reverting 03cd9c115's save in clearAllChats turns it red again. - "a delete the server confirmed is not sent again" reopened the page as soon as the server had the request, before the page had the answer; the unconfirmed delete was then rightly sent again (4 of 5 runs). It now waits until the cache holds no pending delete. Docs-Not-Needed: test-only change. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(chathistory): keep the chat's reasoning effort on the server reasoningEffort (#12700, #12716) is part of the chat the UI syncs since the master merge, but schema.Conversation had no field for it, so the server dropped it and a reload from the server reset the setting. Add the field. The handler spec that round-trips every chat field now includes reasoningEffort. A schema test reads serializeChat in useChat.js and fails for any key the UI sends that schema.Conversation cannot hold, so the next such drift shows up as a failing test rather than in review (it reports reasoningEffort on the previous head). Swagger and docs updated. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * test(chat): use Ginkgo for the schema drift guard The new drift guard uses testing methods forbidden by repository lint. Keep its field coverage and fail-fast parser checks in Ginkgo/Gomega. Assisted-by: nib:gpt-6.1-sol --------- Signed-off-by: TLoE419 <tloemizuchizu@gmail.com> Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> Co-authored-by: TLoE419 <tloemizuchizu@gmail.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> Co-authored-by: mudler-agent <mudler-bot@c3os.io> |
||
|
mudler-agentandEttore Di Giacinto
|
ae8092e1c6 |
feat(sound): expand Simple prompts with chat (#12772)
* feat(sound): expand simple prompts with chat models Allow callers to select an installed chat model for sound prompt expansion. Validate its JSON before audio generation and preserve explicit instrumental and language controls. Keep backend defaults when expansion is omitted, including native Python ACE-Step sampling. Assisted-by: Codex:gpt-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(sound): map expansion by loaded audio family Map neutral expanded fields where the resolved audio family is known. Keep unmarked requests unchanged and reject unsupported expansion before inference. Clear YuE2 and HeartMuLa transcripts to prevent lyric fallback. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(sound): choose a prompt expansion model Require an explicit Simple choice and remember it separately from chat. Clear unavailable choices instead of silently switching language models. Preserve the choice in history and restore it when editing a past result. Keep Backend default and Advanced requests compatible with existing APIs. Document family support and align the published optional request schema. Verify selection, request fields, persistence and history in the browser. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(sound): authorize both generation models Check the resolved audio model and explicit expansion model before loading either. Permission lookup failures stop generation. Assisted-by: Codex:gpt-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(sound): enforce JSON during prompt expansion Constrain expansion output with the existing JSON grammar before decoding. Require a llama-cpp chat model so unsupported backends cannot ignore it. Keep strict field validation and fail before generating audio. Assisted-by: Codex:gpt-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
f85087f992 |
feat(tts): stream Qwen text and add assistant preset (#12770)
* feat(tts): stream Qwen text through audio pipelines Feed cleaned LLM deltas into one persistent Qwen synthesis session. Return audio before input ends and preserve the live RPC through model cache, staging and in-flight wrappers. Stop generation on sink failures or cancellation without replaying speech through a fallback. Pin the incremental native fork and verify early audio, late text, finalization and cancellation with lifecycle and real-model checks. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(tts): use the organization Qwen fork Fetch the reviewed incremental synthesis commit from localai-org so backend builds use the organization-maintained fork. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add a Qwen streaming assistant preset Expose an English pipeline using existing VAD, ASR, LLM and named-voice TTS models. Enable incremental text and audio delivery, and document separate component installs because the gallery has no dependency installer. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): use Parakeet TDT v3 for the assistant Select the existing Parakeet TDT 0.6B v3 transcription entry and update its prerequisite install command. Keep the preset's English voice and incremental Qwen synthesis configuration. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(kokoros): reject unsupported backend RPCs Implement the MotionStream and SlotCache methods required by the shared protobuf service. Return Unimplemented so Kokoros compiles without advertising support for either operation. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(tts): mark cleanup and logging results ignored Explicitly ignore connection shutdown, test server shutdown and Ginkgo diagnostic writer errors. These operations do not replace the stream's inference or sink errors, and errcheck requires their return values to be acknowledged. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(tts): stream buffered clauses in realtime pipelines Start clause synthesis while LLM text is still arriving when both streaming flags are enabled. Keep explicit clause opt-out and native Qwen sessions, bound automatic queued work, and choose streaming support on resolved failover targets. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(tts): bound synthetic voice clone generation The sine-wave reference checks speaker conditioning and WAV output, but does not provide a meaningful speech EOS target. Limit that smoke request to 128 frames and verify the request limit reaches native synthesis. Leave production sampling defaults unchanged. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(tts): require explicit clause buffering opt-in Keep whole-reply synthesis for nonnative engines when clause_chunking is unset or false. Use the bounded cancellable worker when explicitly enabled, with unary clauses when TTS streaming is disabled. Preserve native Qwen precedence for every clause setting. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add VITS and Piper streaming assistants Provide comparable full pipeline configurations with shared VAD, Parakeet transcription, and Qwen chat components. Enable clause buffering explicitly and document the separate prerequisite model installs and Talk selection. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
e715af1def |
feat(llama-cpp): add slot prompt-cache save and restore API (#12730)
* feat(llama-cpp): add slot prompt-cache save and restore API Add POST /v1/slots/save and POST /v1/slots/restore. They write one llama.cpp slot's prompt cache (KV state) to a file and load it back, so a client can warm a long system prompt in a few hundred milliseconds instead of re-running prefill. Refs #12685. The new SlotCache gRPC call posts the same slot save and restore tasks the llama-server /slots endpoint uses. It is off unless the model sets the new slot_save_path option, so existing models are not affected. The /slots monitoring endpoint is not needed here: the backend has no HTTP server. The file name comes from the caller, so it is checked twice, in Go and again in the backend. It must be one path component: no empty name, "/", "\", "..", NUL, control characters or absolute path. The resolved path, with symbolic links followed, must be a direct child of slot_save_path. The routes are admin only because they write files on the backend host. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): answer 501 when a backend lacks slot cache A backend that does not implement the SlotCache rpc returns gRPC UNIMPLEMENTED. The API passed that text through, so the client saw "unknown method SlotCache". Map the status to a typed error in core/backend and return 501 with a fixed message instead. Every other error path is unchanged. Make the SlotCache comments in backend.proto backend-neutral: slot is the backend's state slot, filename must stay inside the save directory, and an empty model_identity means the backend does not check it. Only comments change, no field or name. Add tests that the identity sent is the configured model, and for the error mapping. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): issue slot cache IDs on the server The slot save and restore API let the client pick the file name. Restore could load any file in slot_save_path, and the symlink check ran before llama.cpp opened the file. The backend now issues the name. A save returns a random 128-bit ID (32 lowercase hex characters, read from /dev/urandom) and never overwrites an existing file. Files live in a 0700 directory per model, named from the SHA-256 of the model identity, with a manifest.json that records each save. The manifest is written atomically and a corrupt one stops both save and restore. A restore takes the ID, finds it in the manifest, refuses an entry of another model, builds the path from the entry and checks with lstat that it is a plain file of the recorded size. The directory mode is the main defence against another local user, since llama.cpp still opens the path itself. Each model keeps at most 32 saves and the oldest is deleted first. A file over 8 GiB is deleted after the save, because llama.cpp does not report the size before it writes. Responses carry no host path. The HTTP bodies change: save takes {model, slot}, restore takes {model, slot, id}, and an unknown ID returns 404. In the proto, SlotCacheRequest.filename and SlotCacheResponse.filename become id. SlotCache no longer calls checkModelIdentity, which reads a field this request does not have. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): copy slot_cache files in prepare.sh grpc-server.cpp includes slot_cache.h, but prepare.sh copies the helper headers into llama.cpp/tools/grpc-server/ one by one and skipped it. The llama-cpp, turboquant and bonsai builds share this step, so they all failed with "slot_cache.h: No such file or directory". Copy slot_cache.h and slot_cache_test.cpp, which CMakeLists.txt builds when LLAMA_GRPC_BUILD_TESTS is on. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(deps): bump LocalAGI to pick up the scheduler start/stop fix LocalAGI's scheduler could hit a nil pointer in run when Start and Stop raced. That made the core/services/agentpool tests panic at random and turned tests-linux red. Move to mudler/LocalAGI@7aa0c8fea0, the merge of mudler/LocalAGI#519 ("make Start and Stop safe to call concurrently"). The new module version also adds a few indirect requirements to go.mod and go.sum. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> |
||
|
mudler-agentandEttore Di Giacinto
|
9ad78381d8 |
[ui] Serve canvas HTML previews with their own CSP (#12715)
fix(ui): serve canvas HTML previews from a route with their own CSP External scripts in canvas previews (for example three.js from unpkg) were blocked. The preview used an iframe with srcdoc, which inherits the global Content-Security-Policy, and a meta tag can only tighten it. The UI now posts the HTML to POST /api/canvas/preview, gets a random id, and loads GET /api/canvas/preview/:id in the iframe. That response sets its own CSP that allows scripts from unpkg, jsdelivr, cdnjs, the Tailwind CDN and jQuery. The global CSP is unchanged. The iframe keeps sandbox="allow-scripts" without allow-same-origin. Previews are kept in memory for 10 minutes and are never written to disk. The GET route is public because the id is the capability and a frame navigation cannot send API key headers. Assisted-by: Claude:claude-haiku-5-5 [go test] [npm] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
e28d5009d9 |
stablediffusion-ggml: run MiniMax-H3 video with references and add its gallery entries (#12668)
* feat(stablediffusion-ggml): run MiniMax-H3 video with reference conditioning Bump stable-diffusion.cpp to 2988060 and follow its tiling rename (tile_size_x/y -> tile_size_w/h, rel_size_x/y -> rel_size_w/h). GenerateVideo now passes params["ref_image"] and the request audio to the MiniMax-H3 Ref2VA inputs (ref_images, ref_audios). The audio must be a WAV file. params["ref_video"] is refused, since the proto cannot carry a frame sequence yet. stable-diffusion.cpp cannot tell the FL2VA and REF2VA DiTs apart, and a reference input on FL2VA renders a wrong video without an error. The backend now reads the video_partition option, or the model file name, and fails early on that mix. Keyframes plus references are refused too. Add gallery entries for the FL2VA and REF2VA Q4_K_M sets. They do not enable any bf16 dequantisation. Assisted-by: Claude:claude-sonnet-5-5 go vet gofmt * feat(stablediffusion-ggml): take several reference images, videos and audio clips Phase 1 passed one reference image and one reference audio clip to stable-diffusion.cpp and refused ref_video. Add repeated reference fields to GenerateVideoRequest (ref_images, ref_videos, ref_audios) so the MiniMax-H3 Ref2VA path can use as many references as upstream accepts. A reference video is a directory of frames with an fps and an optional paired audio clip. The /video endpoint stages the new inputs through the same path as start_image, end_image and audio. Frames of one reference video are written to a private directory under zero-padded names so the backend can read them in order. The input count and frame count per request are bounded. The backend merges the new fields with params.ref_image, params.ref_video and audio, which keep working. The FL2VA partition refuses any reference input, REF2VA accepts all of them, and keyframes stay exclusive with references. Reference audio is still WAV only. Assisted-by: Claude:claude-sonnet-5-5 [Bash] * feat(gallery): offer MiniMax-H3 by partition, DiT layout and format Name each MiniMax-H3 entry after its partition (fl2va or ref2va), DiT layout (unpruned or pruned) and format. Add the vllm.cpp builds that its model page lists: pruned Q8_0 (renders), pruned Q6_K, Q5_0, Q4_K, Q3_K and Q2_K (load only), and the pruned FL2VA NVFP4 file (loads only). The earlier names minimax-h3-fl2va-q4 and minimax-h3-ref2va-q4 stay as aliases. The two stable-diffusion.cpp entries become minimax-h3-fl2va-unpruned-q4-ggml and minimax-h3-ref2va-pruned-q4-ggml, and the latter now lists video input. Fix two bad sources. The stable-diffusion.cpp entries took the VAEs from a path that does not exist in the GGUF repository; they now use the Comfy-Org copies. The vllm.cpp entries paired the fp16 video VAE weights with a config from a different directory; they now use the official weights and config from FL2VA/video_vae/source/. Drop video_dequant_bf16 from every vllm.cpp entry. Expanding a quantised DiT to bf16 is the suspected cause of a host out-of-memory error, so the quantised weights stay resident. The model page records its renders with the expansion on, and each description says so. Assisted-by: Claude:claude-sonnet-5-5 [Bash] * feat(gallery): add pruned FL2VA Q4 MiniMax-H3 for sd.cpp Add minimax-h3-fl2va-pruned-q4-ggml, the 11.4 GB pruned FL2VA counterpart of the existing pruned REF2VA entry. Assisted-by: Claude Code:claude-sonnet-5-5 [rc] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
008f9fdf81 |
fix(realtime): end a client response cleanly when its commit fails
A client response.create waits for the in-flight commit before the LLM starts. Two exits left the client without a usable outcome: - When the wait was cancelled (response.cancel, a newer response or teardown) the response returned with no event at all. - When the commit produced no user item (transcription error, empty transcript, voice gate rejection) the LLM ran on a history that lacked the turn. Send response.created and response.done with status cancelled for the first case. For the second, do not run the LLM: send response.done with status failed and an input_not_committed error. The commit still reports its own cause. A response.create with no commit in flight is unchanged. Add specs for STT error, empty transcript, cancel and supersede during the wait. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3a9a67aeb9 |
refactor(realtime): gate client responses on the commit tail
A client response.create waited on a separate mutex-guarded channel that only the last client commit recorded, so it ignored in-flight VAD commits. The commit chain already ends in session.commitTail, whose done channel closes after every earlier commit has finished. Read the tail under commitOrderMu when the response is issued and wait on it. This drops the extra mutex, the clientCommitDone field and the slotSink callback of issueCommit, and the response is now ordered against both commit producers. Fix the commitSlot.done comment: it closes when the commit function returns, not when the user item is appended. Add a spec for commit, commit, response ordering. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-bot
|
2aa8821876 |
test(realtime): sample pending response events
Poll the event count during the held transcription instead of checking\na captured value. Document the client commit and response ordering.
Assisted-by: Codex:gpt-6
Signed-off-by: nexxtmobile.de <kai@nexxtmobile.de>
(cherry picked from commit
|
||
|
kai
|
c27848c8c7 |
fix(realtime): wait for the client commit's STT before the response LLM starts
With client-driven turns (turn_detection null, e.g. a client VAD / speaker
gate) the client sends input_audio_buffer.commit and response.create
back-to-back. The response.create triggered the LLM immediately while the
commit's transcription was still in flight, so the LLM (fast) won the race
and answered an empty transcript; the response on the real transcript was
then discarded ("skipping response: turn context cancelled during
transcription").
Record the in-flight client commit's slot done channel (closed once the
commit has transcribed and appended its user item) and make the client
response.wait for it before the LLM starts, so the LLM only runs after the
STT is complete.
- issueCommit gains a slotSink callback (slot claim + issue stay under one
lock); the VAD commit path passes nil.
- The client commit / response.create handler cores are extracted to
issueClientCommit / issueClientResponse so the real path is unit-testable.
- Regression specs drive the real issue path with a held transcription: the
LLM must not start until the STT completes, and it must see the user's
transcript.
Signed-off-by: nexxtmobile.de <kai@nexxtmobile.de>
(cherry picked from commit
|
||
|
Ettore Di Giacinto
|
28c70be87e |
fix(realtime): keep sound detection for skipped turns
A VAD commit that is skipped because interrupt_response is false runs with an already cancelled response context. Sound detection used that context, so the model call failed and the turn never got its sound-detection event. Run it under the session context, like the transcription and the voice gate, so overlapping turns keep their sound events. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-bot
|
f22fd45e21 |
fix(realtime): honor interrupt_response
Keep active responses running when VAD interruption is disabled. Skip
an overlapping automatic response while preserving its transcript and
conversation history. Default interruption and explicit client actions
keep their existing behavior.
Add regression coverage for both VAD modes and document skipped turns.
Assisted-by: Codex:gpt-6
(cherry picked from commit
|
||
|
Ettore Di Giacinto
|
958add2336 |
fix(motion): cap concurrent motion sessions per user
A model serves one motion session at a time, so a single user could hold several models for as long as the session stays active. Limit each user to four sessions and free the slot on close. The one-session-per-model rule and the global limit are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 |
||
|
Ettore Di Giacinto
|
e7687e9fe9 |
fix(http): drop the global CSRF relaxation and CORS-derived socket origin
CSRFMiddlewareWithCORS changed CSRF handling for every route and, with a wildcard allowlist, disabled the Sec-Fetch-Site check. Restore the original CSRFMiddleware: clients that authenticate with validated header credentials, or with a single-use WebSocket ticket, are already exempt, so the motion routes need no relaxation. The motion WebSocket upgrade no longer reads the CORS allowlist to decide whether a cross-origin upgrade is acceptable. It accepts one only with a valid ticket or validated header credentials; everything else gets the same-origin default. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
3de91d48d9 |
feat(motion): add GEM-X backend and duplex streaming API
Add the GEM-X native backend with live pose and root displacement output, bounded duplex WebSocket streaming, session authorization, metrics and usage accounting. Share the embedded bidirectional RPC channel adapter across streaming backends.
Include backend build and gallery integration, API documentation, schemas and tests.
Assisted-by: Codex:gpt-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
(cherry picked from commit
|
||
|
Ettore Di Giacinto
|
29eba5b7f8 |
docs(models): describe what a queued manual import adds
A queued import goes through the gallery installer, so it persists recommended inference defaults and installs the backend when automatic backend installation is on, while the synchronous write stores the body as posted. Say so in the docs and the endpoint description, list the reference forms that are queued (URLs, hf:// and owner/repo, artifacts), and regenerate the swagger files. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Plamen K. Kosseff
|
addcec0c28 |
fix(models): resolver-gated queueing and editor polling for manual imports
Review round at |
||
|
Plamen K. Kosseff
|
a50eb4e68a |
docs(models): document the queued manual imports — swagger, specs, docs
- swagger: annotate POST /models/import (untyped body: documenting it
as config.ModelConfig would inline the whole type graph, ~5.5k
generated lines; the description spells out the dual response) and
regenerate the spec
- tests: pin the admission contract of the queued branch — job id +
status URL, job queryable before the worker runs, config not written
at admission, preload-matching filenames, checksum passthrough, and
the untouched synchronous path
- docs: the Writing YAML section explains when Create queues and what
the API answers in each case
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
(cherry picked from commit
|
||
|
Plamen K. Kosseff
|
28c4bb9277 |
feat(models): route manual imports with remote assets through the gallery job queue
Manual /models/import with downloadable assets ran synchronously; it now
enqueues like a gallery install and the import UI polls the job progress.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
(cherry picked from commit
|
||
|
localai-org-maint-bot
|
c5997fd71a |
fix(mcp): check token expiry after failed refresh
A slow failed refresh can cross the token's safety margin.
Check the current time before reusing the token and start retry
backoff when the failed fetch finishes. Apply backoff before the
first successful fetch too.
Add regression tests for these cases and document per-process caches.
Assisted-by: Codex:GPT-6
(cherry picked from commit
|
||
|
Stefan Walcz
|
beaeb05e52 |
feat(mcp): support OAuth2 client_credentials for remote MCP servers
Remote MCP servers could only be authenticated with a static bearer
token. Deployments that put their MCP servers behind an OIDC provider
(authentik, Keycloak, Entra ID, ...) issue short-lived service tokens
instead, so LocalAI had to be given a long-lived key or could not
connect at all.
A remote server entry now accepts an optional "oauth2" block
(token_url, client_id/client_id_env, client_secret/client_secret_env,
scopes, endpoint_params). The token is fetched with the
client_credentials grant by an http.RoundTripper installed on the MCP
client's HTTP client, so streamable HTTP and SSE transports both use it.
The token is cached and refreshed at half of its lifetime; if a refresh
fails the current token is kept until 30s before expiry, after which
requests fail with an error instead of going out unauthenticated.
"token" and "oauth2" are mutually exclusive and inconsistent blocks are
rejected when the model config is loaded. Static tokens behave as
before. MCPRemoteServer now implements fmt.Stringer so the existing
debug logs that print whole server entries no longer write the token
or client secret in clear text.
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
Assisted-by: Claude Code:claude-opus-5-5
(cherry picked from commit
|
||
|
mudler-agentandEttore Di Giacinto
|
0177e4988c |
feat(images): add ESRGAN upscaling and Studio workflow (#12620)
* feat(images): add upscale model capabilities * fix(config): keep SD GGML upscale opt-in * feat(stablediffusion): add lazy ESRGAN upscale bridge Keep native image ownership in C++ and serialize context creation, inference and teardown. Use upscale_scale in backend options as the explicit upscale-only marker because ModelOptions does not carry usecases. Assisted-by: Codex:gpt-5 * fix(stablediffusion): validate ESRGAN settings * fix(upscale): forward typed settings and contain resolved model paths Typed settings replace legacy options without duplicates. Resolve model and trusted root symlinks before caching the model path, rejecting missing files and escapes. Assisted-by: Codex:gpt-5 * fix(upscale): validate scale and clean failed HTTP outputs Require an explicit scale bounded to 1-16, check backend results, and keep destination cleanup owned by the allocating endpoint. Cover validation, result failures, cleanup, and successful URL responses. Assisted-by: Codex:gpt-5 * feat(studio): add inline image upscale workflow Assisted-by: OpenAI:gpt-5 * test(studio): serve upscale capability fixtures * fix(stablediffusion): default ESRGAN tiling to full frame * feat(gallery): add RealESRGAN anime upscaler * fix(stablediffusion): allow full-frame ESRGAN tiles * fix(routes): select explicit upscale models Give image upscaling dedicated usecase-filtered middleware and keep ordinary image models out of the endpoint. Mark and verify the legacy Diffusers x4 gallery model as an explicit upscaler. Assisted-by: OpenAI:gpt-5 [shell] * refactor(upscale): keep settings in backend options * test(config): keep upscale options backend-specific * fix: keep upscale routing metadata out of backend options --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
08871b1a6a |
feat(audio): add YuE2 music generation (#12617)
* feat(audio): forward sound generation params Pass string-valued sound generation parameters from the public request to the audio backend. Typed fields retain precedence over colliding generic parameters, and source audio remains unavailable in the public schema. Assisted-by: nib:gpt-5.6-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add YuE2 music model Assisted-by: nib:gpt-5.6-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(audio): document YuE2 sound generation Assisted-by: nib:gpt-5.6-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(audio): reserve source input parameter Assisted-by: nib:gpt-5.6-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
24a9feecbb |
feat(api): add OpenAI Decisions endpoint (#12603)
* feat: add OpenAI Decisions adapter with validated ordered answers Share SystemOne execution and preserve ordered, typed Decisions answers. Reject null probabilities and derive scores from the emitted distribution. Cover conversion and handler contracts using a valid generated PNG fixture. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix: preserve Decisions choice types and empty score labels Keep canonical JSON values in model-visible choice criteria and represent probability label presence explicitly. Add conversion and serialization regressions. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat: wire the OpenAI Decisions route Expose the adapter through the shared Decisions feature gate and usage middleware. Test real route registration with a fake backend, including unauthenticated access, user permissions, and exact usage accounting. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: document OpenAI-compatible decisions and discovery Describe the shared local execution contract, typed questions, image limits, and backend accounting. Publish the Decisions route in Swagger and retain the existing SystemOne discovery tag. Use documented loose object placeholders for JSON unions that Swagger 2 cannot express. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
6343a2dc6f |
feat(voice-detect): report the encoder family so audio-registered voices are fingerprinted (#12499)
* feat(voice-detect): report the encoder family so audio-registered voices are fingerprinted
A voice registered from audio through the voice-detect backend had no
encoder fingerprint, so the parakeet-cpp backend could not tell whether
it was comparable with the loaded speaker model and could only fall back
to the file-name rule.
The voice-detect backend now binds the three new libvoicedetect accessors
with a symbol probe (an older library still loads and reports nothing),
copies the borrowed strings at once and never frees them. It also hashes
the model file once at load. VoiceEmbedResponse gains two optional
fields, encoder_family and encoder_weights ("sha256:<hex>", empty when
the model is not a plain file).
/v1/voice/register stores them as encoder_family and a new
encoder_weights field in the registry entry; model keeps the encoder
name, so the 1:N identify filter by name is unchanged for old entries.
A voice with a family is sent to the backend whatever its file name, and
the backend decides by family. /v1/voice/identify compares the family
when both the stored voice and the probe have one. Old entries load
without the fields and stay unfingerprinted.
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(voice-detect): say how a voice with weights but no family is treated
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(voice-detect): mark the model file open as intended for gosec
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
0102751b31 |
fix(mcp): connect to legacy SSE servers (#12304)
* fix(mcp): connect to legacy SSE servers Remote model MCP connections only use Streamable HTTP, so legacy SSE servers fail initialization. Retry with SSE when the initial POST returns 400, 404, or 405. Share one discovery timeout across attempts and cancel failed connections without truncating successful sessions. Preserve HTTP policy and reject foreign SSE message endpoints before attaching credentials. Add SDK integration tests and document automatic transport selection. Assisted-by: Codex:gpt-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(mcp): avoid narrowing HTTP status codes Store the initialization status in a 64-bit atomic value to avoid the integer overflow conversion reported by gosec. Assisted-by: Codex:GPT-6 gosec Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Plamen K. Kosseff
|
c03acb0647 |
feat(models): report per-model disk usage in API and WebUI (#12551)
New admin endpoint GET /api/models/storage reports disk usage of the installed models: each model's files and sizes, files shared between models counted once, and configured files that are missing from disk. The models WebUI page shows a usage summary, each model's file table, and the full file list with per-file status. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com> |
||
|
mudler-agent
|
cd604b5c5b |
fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)
Fence model load jobs by generation, lease them on the database clock, bound the work on the worker with operations and a process-group watchdog, and add one stop path with a load-cancel API. See the pull request for the design, the rolling upgrade notes and the test evidence. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
c525ad16e8 |
fix(audio): record transcription usage and traces (#12546)
* fix(audio): record transcription usage and traces Count successful transcription requests by model, including streams. Keep token counts at zero because transcription exposes no token usage. Capture multipart API trace metadata without reading uploaded audio. Do not record failed transcription or client writes as successful usage. Assisted-by: Codex:gpt-6-astra * fix(audio): preserve aliases in streaming usage Streaming transcription records the resolved target as its usage model. Pass the requested name so JSON and SSE requests share the alias bucket. Assisted-by: Codex:gpt-6-astra * test(http): check multipart reader close errors Assert successful reader cleanup to satisfy the errcheck CI gate. Assisted-by: Codex:gpt-6 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
6b794651a4 |
feat(diarization): return sound events with include_sounds (#12544)
* feat(diarization): return sound events with include_sounds
A client that wants text, speakers, voice prints and sound events had to
make a diarization call and a separate sound call. Add an include_sounds
request field to /v1/audio/diarization that adds a sounds array of closed
events {start, end, label, confidence}, in seconds.
The parakeet-cpp backend runs a tagger-only scene stream over the clip,
the same stream and thresholds the live path uses, so a clip gives the
same events offline and live. A model with no sound_model companion, or a
backend that does not report sound events, fails with 501 and the stable
code include_sounds_unsupported instead of an empty list. The proto
carries sounds_included so an empty list still means "nothing heard".
The localai-proxy backend forwards the field. Swagger, docs and the
e2e mock backend are updated.
Assisted-by: Claude:claude-sonnet-5-5 [protoc swag go]
* feat(gallery): add parakeet-cpp-multilingual-diarization-speakers-sounds
Same as parakeet-cpp-multilingual-diarization-speakers (TDT 0.6B v3,
Nemotron-3-Diarization, WeSpeaker) plus a CED-Tiny sound_model, so one
model name serves /v1/audio/diarization with include_text,
include_speaker_profiles and include_sounds. It declares the
sound_classification usecase like the realtime scene entries.
Assisted-by: Claude:claude-sonnet-5-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
fa25493c9e |
feat(config): let parakeet-cpp declare the speaker_recognition usecase
parakeet-cpp now implements VoiceEmbed and VoiceVerify, so list them in its backend capabilities together with the speaker_recognition usecase. The usecase is not guessed, because most parakeet models have no speaker encoder: a bundle declares it in known_usecases, which is enough for /v1/voice/* and for voice_recognition.model. Add a realtime pipeline test where one bundle model names the vad, transcription, sound_detection and voice_recognition stages and the gate resolves the speaker through the backend's VoiceEmbed. Assisted-by: Claude Code:claude-sonnet-5-5 |
||
|
Ettore Di Giacinto
|
1d117292b1 |
feat(voice): name registered speakers through a bundle speaker component
Naming read only speaker_model:, so a parakeet-cpp bundle that sets speaker_component:voice never received the registered voices, in live transcription or in diarization. Resolve speaker_component: when speaker_model: is not set. A component is not an encoder file, so no file-name tag is derived for it: voices with a hash or family identity and untagged voices reach the backend, and a voice with only a file-name tag matches through the new optional speaker_tag: alias. speaker_model: still wins, including when it points at the bundle file. The bundle gallery entries do not declare speaker_recognition: that usecase means the VoiceEmbed and VoiceVerify RPCs, which the backend does not serve, and it selects the default model for /v1/voice/*. The pinning test now says so. Assisted-by: Claude Code:claude-sonnet-5-5 go golangci-lint |
||
|
mudler-agentandEttore Di Giacinto
|
79a7631cc5 |
feat(parakeet-cpp): encoder fingerprint for speaker naming, VAD trim and word filter options, pin bump (#12491)
* chore(parakeet-cpp): bump parakeet.cpp to 2de154c Brings in the speaker registry encoder fingerprint, the VAD segment trim and the opt-in word filter, a fix for a per-call thread count that stayed set on the process-wide backend after a Silero VAD pass, and bundle components loaded from memory. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * feat(parakeet-cpp): encoder fingerprint for speaker naming, vad_trim and guard_* options Speaker naming. A registered voice now carries the encoder that made it: the embedding family (voicedetect:<arch>:<name>:<dim>) and the sha256 of the weights. The backend reports the family of the loaded speaker model in Status, voice enrollment from speaker_profiles stores it as encoder_family (old entries load without it), and the registry sent to parakeet.cpp is built with parakeet_capi_speaker_registry_add_embedding_fp. The library then refuses a registry of another encoder family and the error names both families; another quantization of the same family only warns. A voice with only a weights hash gets the loaded family when the hashes are equal. Voices without a fingerprint (registered from audio: libvoicedetect cannot report one) keep the file-name rule and are used with a warning. The library cannot mix them with fingerprinted voices in one registry, so a request that has any uses the old registry for all. speaker_strict:true drops them instead. A library without the symbols behaves as before. Transcription. vad_trim (seconds, 0 keeps the whole cuts) goes through the VAD options JSON, so it reaches /v1/vad and the segmenter. The guard_* options guard_min_local_conf, guard_local_radius and guard_drop_punct_only turn on the word filter through parakeet_capi_transcribe_path_json_with, or through the segmenter with vad:true. They are off by default, bad values fail the load, and a library without the symbol fails it with a clear message. The dropped word count is logged at debug level. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Stefan Walcz
|
5f5feea15a |
fix(agentpool): keep the conversation in the agent web chat (#12410)
* feat(agents): keep the web chat history between messages The agent chat endpoint ran every message as a fresh job (ag.Ask(WithText(message))), so a follow-up such as "add two days to item 3 and recalculate" never saw the answer it referred to. Agents then rebuilt their reply from scratch instead of changing it. Use the agent's own conversation tracker, as the Telegram and Slack connectors already do: send the earlier turns with the new message and record successful answers. Failed, cancelled or empty runs are not recorded, and the tracker drops a conversation after the agent's last_message_duration of inactivity. The distributed (NATS) chat path is unchanged. Assisted-by: Claude:claude-opus-5-5 ginkgo Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(agents): keep each web chat conversation's history separate Review on this PR: the first version kept the history in the agent's conversation tracker under one fixed key per agent. The web UI keeps several conversations per agent (New Chat, switching, Clear) and did not send which one a message belongs to, so the hidden history mixed them: a new chat received the previous chat's turns, and Clear only cleared the screen. The client now sends the earlier turns of the conversation it is showing as `history` with POST /api/agents/:name/chat, and the server keeps no web chat history of its own. Each conversation only ever sees its own turns; New Chat and Clear start without history. The server uses only user and assistant turns with text, bounded to the most recent 40 turns and 64,000 characters. `history` is optional, so existing callers keep the previous behaviour (no history); the distributed (NATS) path does not forward it yet. Tests: two conversations of one agent stay apart, an empty history (New Chat or Clear) starts fresh, system/tool/empty turns are dropped, the bounds keep the most recent turns; the UI helper that builds the history from the visible messages has node --test coverage. Docs: features/agents.md describes `history` and the web UI behaviour. Assisted-by: Claude:claude-opus-5-5 ginkgo Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> --------- Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> |
||
|
nexxtmobile.deandnexxtmobile.de
|
4d681b7f6d |
fix(realtime): detach VAD-commit transcription from barge-in cancellation (#12446)
* fix(realtime): keep VAD-commit transcription alive across barge-in, cancel it at teardown, order commits
Barge-in (new speech onset) cancels the turn's SourceVAD response context
(realtime_turncoord.go: respSink.cancel(SourceVAD)). The VAD commit body
runs under that same context, so an in-flight Whisper STT call was aborted
with 'context canceled' whenever the caller kept talking while the first
chunk was transcribing. The user's turn was lost: no transcript, no
LLM/TTS response.
v1 of this fix ran the transcription with context.WithoutCancel(ctx). The
review correctly pointed out two correctness gaps:
1. Teardown lost its cancellation. WithoutCancel detaches from every
cancellation, so a transcription in flight at session close outlived
the session and blocked respSink.shutdown (which joins the response
goroutines) until the backend finished the job.
2. Out-of-order commits. Consecutive commits run in parallel goroutines,
so a fast second transcription could append its user item before a
slow first one: the conversation became [second, first] and the second
response saw only [second].
Changes (core/http/endpoints/openai/):
- Session gains a session-lifetime context (sessionCtx), cancelled by
conncoord's Teardown BEFORE respSink.shutdown joins the response
goroutines. The transcription (and the voice-gate resolution) run under
it: they survive barge-in (which cancels only the per-response context)
but are cancelled with the session.
- Commit slots order the user-item appends in speech order:
Session.nextCommitSlot() is claimed at commit issue time (VAD CommitTurn
/ client commit), a commit's item append waits on the previous slot's
done (aborts on the session context), and every exit closes the slot so
a failed or torn-down commit never blocks the next. Transcriptions stay
parallel; only the appends are ordered.
- If the turn's response context was cancelled while the (detached)
transcription ran — barge-in, superseded by a newer commit — the user
item still commits (appendUserItem, split out of generateResponse) so
the LLM context keeps the full user input, but no response is generated
for the superseded turn; the newer speech triggers its own response on
the complete history.
- Regression tests (realtime_commit_order_test.go) cover both review
schedules — teardown during an in-flight transcription, and
held-first/finished-second out-of-order completion — plus the
barge-in-during-transcription item survival, driving the real commit
path with a transcription double that honours context cancellation.
- docs/design/realtime-state-machines.md: implementation-status entry for
the committed-turn pipeline (transcription lifetime + commit order).
Fixes #12445
Validated: builds, go vet clean, all openai specs + respcoord/turncoord/
conncoord suites pass under -race (incl. the 3 new regression specs).
Production A/B (call-center voice agent, SIP, silero-vad +
whisper-large-turbo + LLM + TTS, server_vad ~600 ms) on LocalAI v4.11.0:
unpatched — 'transcription_failed: context canceled', first part of the
utterance lost, agent answers only the remainder; patched — full
transcript committed, agent answers the complete utterance, barge-in
still cancels the in-flight assistant TTS response as intended, and
teardown cancels the in-flight transcription instead of waiting for the
backend.
Signed-off-by: nexxtmobile.de <kai@nexxtmobile.de>
* fix(realtime): release commit slots in order on every exit; share slot+issue boundary
Follow-up to the review of
|
||
|
mudler-agentandEttore Di Giacinto
|
99043b442c |
feat(router): route with native decision models (#12449)
* fix(schema): preserve SystemOne image inputs Assisted-by: OpenAI * test(schema): follow Ginkgo conventions for decision inputs Assisted-by: OpenAI * feat(llama-cpp): dispatch native decisions through Score Upgrade the stock dependency and reconcile Score/TTS patches. Reuse native decision parsing, tasks, formatting and response-reader cleanup; preserve ordinary scoring admission and guard older dependencies. Assisted-by: OpenAI * refactor(systemone): share request and model validation Assisted-by: OpenAI:gpt-5 * fix(systemone): preserve HTTP wire-byte validation limit Keep structural validation separate from the serialized internal request bound so HTML escaping cannot reject valid HTTP payloads. Assisted-by: OpenAI:gpt-5 * feat(systemone): bound images and account native decisions Preserve public wire limits independently from router serialization. Reject unsupported NER images, map native request/capability errors, and stamp explicit usage once. Advertise decisions for stock llama-cpp. Assisted-by: OpenAI * fix(systemone): record usage on registered native route Exercise real registration and billing with a mock native backend. Reject empty native responses, malformed image URLs, trailing JSON, and wire overflow including whitespace. Assisted-by: OpenAI * feat(router): add lazy native decision transport Bind named models through internal ModelSystemOne calls with shared validation and bounded abandoned operations. Remove request and echoed-error contents from decision traces. Assisted-by: OpenAI:gpt-5 * feat(router): classify overlapping policies with native decisions Ask independent noul questions, validate probabilities and preserve first-superset routing. Wire the central factory with config-sensitive invalidation and cancellation-safe resolution. Document native framing and bounded operation limits. Assisted-by: OpenAI:gpt-5 * feat(gallery): add pinned Julia-1 native decision model Add a separate text-only llama-cpp Q8 entry with pinned Apache-2.0 source provenance and checksum. Installed using the gallery installer and exercised choice, score and noul on CPU. Assisted-by: OpenAI * test(router): verify native decisions through central factory Add an opt-in real-model Ginkgo integration covering the native Go loader and C++ transport, token usage, independent overlapping labels, and candidate selection. Document owned-server execution and the intentionally non-quality threshold. Assisted-by: Codex:gpt-5 * fix(llama-cpp): align upstream pin and preserve decision signatures Advance to bed0a856 without losing the automated upstream bump. Detect full-request fill_task support at compile time and forward every question for Nimble framing while retaining the earlier native signature. Preserve reconciled SCORE/TTS patches; add standalone compatibility coverage. Assisted-by: Codex:gpt-5 * feat(gallery): add native decision family defaults Pin Laya, Kev-4B, lev, OpenJev and Nimble artifacts. Verify Laya/Kev/lev gallery installs and CPU contracts on both native pins; clearly mark OpenJev/Nimble runtime validation pending and their noncommercial licenses. Assisted-by: OpenAI * docs(decisions): clarify integrated Nimble prerequisite Record the exact combined backend pin while retaining pending OpenJev and Nimble installation/runtime validation status. Assisted-by: Codex:gpt-5 * fix(gallery): indent native decision model sequences Match repository yamllint indentation for Laya, Kev, lev and OpenJev list fields. Parsed gallery data is unchanged; reproduce CI gallery lint failure before the whitespace-only fix and pass the same command afterward. Assisted-by: Codex:gpt-5 * docs(decisions): record OpenJev and Nimble CPU validation Record gallery installation, checksum/metadata verification and multiquestion native smoke results on bed0a856. Retain noncommercial and text-only limitations without accuracy or deterministic-output claims. Assisted-by: OpenAI * fix(ui): expose native Decisions router classifiers Select classifier models using metadata-driven capability routing, retain tuned thresholds, and validate native decision selections before saving. Cover both native backends and create/save/reopen in the real React editor. Assisted-by: Codex:gpt-5 * fix(router): exclude aliases from native decision discovery Check the originally named config before advertising native Decisions eligibility. Retain target capability inheritance for ordinary generation aliases. Exercise the actual capabilities endpoint with native models on both backends, aliases, and disabled models. Assisted-by: Codex:gpt-5 * feat(systemone): share bounded multimodal input validation Preserve text wire limits while admitting bounded PNG/JPEG decision input. Share collection and header validation across internal and public callers and keep the native runner response budget independent. Assisted-by: OpenAI:API-assistant * fix(systemone): bound admission lifetimes and validate complete images Retain shared admission leases through actual work completion, including abandoned internal operations. Decode bounded image pixels, cap public native responses before usage stamping, and preserve oversized malformed text status precedence. Assisted-by: OpenAI:API-assistant * fix(router): classify images before media fetching Preserve ordered structured probes for native decisions. Defer OpenAI media preparation until routing selects the served model, so rejected decision URLs cannot trigger downloads before shared validation. Guard direct image collection with context-aware shared admission. Keep text classifiers and embedding caches from discarding image input. Retain fail-closed classifier configuration and runtime fallback policy. Add middleware, typed-content, admission, cancellation and cache tests. Assisted-by: OpenAI:API-assistant * fix(router): bound extraction before serialization Check probe budgets before copying text or marshaling message state. Count JSON escaping so oversized internal inputs fail before allocation. Preserve typed Anthropic blocks through selected-model conversion and fallback. Keep retry coverage in Ginkgo without global test registration. Assisted-by: OpenAI * fix(router): bound supported probe serialization Arbitrary structs can bypass the probe budget through pointer marshalers, string tags, and promoted fields. Accept concrete chat schema types and plain JSON values instead of emulating arbitrary struct serialization. Budget escaped direct prompts before marshaling so raw length cannot hide serialized expansion. Preserve runtime fallback and reject oversized input before invoking the decision runner. Add Ginkgo allocation, boundary, and marshaler invocation regressions. Six-package tests, three-package race tests, and full-T2 delta lint pass. Assisted-by: OpenAI:GPT-5 golangci-lint * feat(decisions): enable bounded OpenJev images Validate native decision images before permissive media parsing and pixel allocation. Require both decision image support and a vision projector; missing or audio-only projectors cannot silently become text decisions. Pin the OpenJev Q8 projector and document its license and disk footprint. Add native safety tests, canonical limit parity, gallery and load-option checks, and a reproducible CPU direct-RPC contrasting-image smoke. Assisted-by: OpenAI:GPT-5 * fix(decisions): reject incomplete image streams stb accepts corrupt PNG Adler checksums and truncated JPEG scans. Use bounded zlib validation and strict libjpeg decoding before parsing. Keep dimension and aggregate pixel checks ahead of decoder allocations. Wire decoder dependencies into native builds and runtime packaging. Add regressions for appended EOI and embedded marker bypasses. Assisted-by: OpenAI:GPT-5 * fix(ci): gate native decision image validation Run the decoder security tests outside the stdlib-only native suite. Fetch vendor headers at the backend pin and provision decoder dependencies. Gate Go limit parity and production CMake wiring without model downloads. Assisted-by: OpenAI:GPT-5 * test(decisions): cover multimodal public API paths Exercise shared image contracts through the registered HTTP routes and external mock backend. Add opt-in cached gallery installation and real OpenJev image decisions through SystemOne and both routing APIs. Assisted-by: Codex:gpt-5 * test(decisions): assert isolation and cache bypass Observe external RPC calls and compare complete classifier history. Winner-only and cache-miss checks could hide dropped history or cache use. Give real inference its own application and model directory so shared backend mappings and loaded processes cannot affect mixed suite order. Assisted-by: OpenAI:ChatGPT * test(decisions): isolate fixture globals Disable optional global services in the isolated HTTP fixture and register cleanup before setup assertions. Verify meter provider identity survives fixture creation and destruction. Snapshot observed usage before assertions so failures cannot retain the mutex. Require a successful usage stamp before checking error responses. Assisted-by: Codex:gpt-5 golangci-lint * fix(application): honor optional telemetry controls Skip failover gauge registration when metrics are disabled. Register against the application meter rather than looking up the global provider. Allow embedders to retain the bounded routing log without billing stats. Keep the existing default when stats are disabled. The isolated HTTP fixture uses this option without losing its native router assertions. Assisted-by: Codex:gpt-5 golangci-lint --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
e4fa051ee3 |
refactor(distributed): put the NATS-only paths behind interfaces (#12395)
* feat(messaging): add shared subject rules Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(messaging): cover BroadcastRoots, ControlRoots and SubjectRoot Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add Broadcaster and enforce subject rules in every carrier Broadcaster is the fan-out half of MessagingClient. The NATS client and the in-memory FakeBus now refuse a subject outside the served roots and any wildcard other than a whole single token, and FakeBus shares MatchSubject instead of its own copy. FakeBus Unsubscribe now removes its own subscription instead of the first one with the same subject. A shared conformance suite in messagingtest runs against both carriers. The distributed e2e specs that used invented test.* subjects, and the one that subscribed with a > filter, now use subjects from subjects.go. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: depend on Broadcaster where only publish and subscribe are used Narrowed to messaging.Broadcaster: nodes/staging_progress.go, nodes/install_progress_publisher.go, galleryop/operation.go, galleryop/service.go, agentpool/user_services.go, agentpool/agent_jobs.go, openresponses/store.go, openresponses/sync.go, syncstate/syncstate.go, finetune/service.go, quantization/service.go and failover/distsync/distsync.go. SubscribeJSON now takes a Broadcaster because it only calls Subscribe, which lets the narrowed consumers use it. Stayed wide: worker/supervisor.go, because its client field also serves the SubscribeReply handlers in worker/lifecycle.go. The request/reply, queue and wiring files (nodes/unloader.go, nodes/file_stager_s3.go, jobs/dispatcher.go, agents/dispatcher.go, agents/events.go, worker/file_staging.go, cli/agent_worker.go, http/app.go) are unchanged by design. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): name the no-route condition and confine the carrier error Consumers matched nats.ErrNoResponders, which names an absence, to demote a node. They now match ErrNoRoute, the control path maps the carrier's failure onto it, and timeouts and worker refusals are pinned as not being no-route. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(nodes): state which FileStager implementations return ErrNoRoute Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): build backend clients through one node-aware seam Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe the distributed transport seams Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct comments that overclaim after the seams refactor Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(agent-worker): refuse an unserved LOCALAI_AGENT_SUBJECT at startup The messaging client now refuses a subject whose root no carrier serves. An agent worker started with a custom LOCALAI_AGENT_SUBJECT such as tenant-a.agent.execute used to start and then wait on a subject the frontend never publishes to. After the subject rules landed it exited at subscribe time with an error that did not name the setting. Behaviour change: the worker now checks LOCALAI_AGENT_SUBJECT before it registers or connects, and exits with an error that names the variable and says to use a served subject under the agent root, for example agent.execute. The served roots are not widened: a custom root was never delivered by the frontend, and a wider set would reopen the drift the subject rules exist to close. The flag help and the agent worker docs state the constraint. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(nodes): pin the reactions to ErrNoRoute Three callers react to ErrNoRoute and had no spec: the reconciler's upgrade drain falls back to the legacy forced install, the reconciler marks the node unhealthy when a pending op has no route, and the backend-op fan-out marks the node unhealthy. Each spec drives the real caller with a scripted no-responders reply and reads the result from the registry or the recorded requests. A fourth spec pins the other side: a pending op that times out leaves the node healthy and only counts the attempt, so mapping timeouts onto ErrNoRoute would fail here. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(messaging): pin client subject checks and fail the carrier suite in CI Add specs that call Publish, Request, Subscribe, QueueSubscribe, SubscribeReply and QueueSubscribeReply on a client with no connection. Each call must return ErrUnservedSubject for bogus.thing and ErrUnsupportedWildcard for jobs.>. This proves that the subject check runs before the connection is used, and needs no server. The NATS conformance suite is the only check that runs the subject rules against a real carrier. Before this change it skipped without output when Docker was missing. Now it fails when CI is set, so a Linux runner without Docker cannot hide it. It still skips on local runs and on macOS CI, which has no Docker. Add SubjectNodeBackendInstallProgress to the list of constructors that must build served subjects, and ask contributors to extend the list. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: state what ErrNoRoute may change, and group the distributed guides The seams note said MarkUnhealthy was the only state change allowed on ErrNoRoute. A pending backend op still records the failed attempt, counts toward the reconciler's retry limit and is dead-lettered after the maximum attempts. The note now says that MarkUnhealthy is the only change to the node's own state, and that the per-op accounting is not a verdict about the node. The note also documents that the NATS conformance run fails under CI when Docker is missing. The distributed-seams row moves next to the distributed-state row in the topics table. The liveness ping spec header now says no route is a reason to skip the worker, not proof that the worker is gone. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): give the backend client factory the node id Mechanical: the method gains a nodeID parameter and the eight test fakes are updated. No behaviour change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): drop the optional node-aware factory The node id is now in the main method, so the optional interface and its helper had no behaviour of their own. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): dial backend probes through the client factory Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): dial workers' file servers through a per-node dialer Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(http): proxy backend logs through the per-node worker dialer The admin backend-logs proxy (list, lines and the WebSocket stream) now reaches a worker through the same per-node dialer as the HTTP file stager, so every frontend-to-worker dial goes through one seam. The shared direct dialer keeps alive for 15s where the proxy used 30s. Harmless for requests bounded at 15s. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(http): keep the backend-logs proxy independent of the admin connection The proxy request had no context before the dialer change and is bounded only by its 15s timeout. Keep it that way. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: move the worker control payloads to workerctl Mechanical move of the request and reply structs, the install progress event and the file payloads out of messaging. The verbs no longer belong to one carrier. No alias is left behind. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): serve the lifecycle verbs through a controlServer The worker registers one handler per verb and a NATS server maps each verb to its subject. Registration errors now name the verb. node.stop is served with SubscribeReply, which is identical on the wire because the handler never replies. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): report install progress through the control sink Install and upgrade now emit download progress through the sink the control server hands them. The debounce and the terminal flush stay in the handler path, built over that sink by the new nodes.NewDebouncedInstallProgressSink, which replaces NewDebouncedInstallProgressPublisher. The subject and payload on the wire are unchanged. The supervisor no longer holds the bus, and installFn and upgradeFn let specs drive both verbs without a gallery. The malformed-request log lines are restored for install, upgrade, backend.delete, model.unload, model.stop and model.delete, with the reply bytes unchanged. The signal adapter is renamed noReply, which also lets worker.go import os/signal without an alias again. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): serve the file-staging verbs through a controlServer An empty list-dir answer is now {} rather than {"files":null}, because the typed reply omits an empty Files slice. The frontend decodes both to a nil slice in nodes/file_stager_s3.go. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add WorkQueue and the NATS producer This is the producer side of the competing-consumer seam. The work kinds map one to one to today's subjects and queue groups: task to jobs.new and mcp-ci to jobs.mcp-ci.new (both in group workers), agent-run to agent.execute (group agent-workers). Enqueue publishes the payload as Publish does today, with one JSON marshal. FakeBus now records queue groups and keeps reply handlers so later specs can pin and drive them. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add the NATS WorkConsumer An in-flight limit of one runs the handler inline on the delivery goroutine, as the MCP CI consumer does today. Any other limit spawns per delivery, as the agent consumer does. Queue groups are unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: publish queued work through WorkQueue The job dispatcher, the agent pool and the agent scheduler enqueue through messaging.WorkQueue; the NATS implementation publishes to the same subjects as before. DistributedServices builds the queue next to the NATS client and hands it to the dispatcher and the agent pool, whose distributed mode switch now reads a non-nil WorkQueue. The unused AgentPoolService.SetNATSClient is removed. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: consume queued work through WorkConsumer The agent dispatcher and the MCP CI consumer register through messaging.WorkConsumer. The NATS implementation keeps the inline one-at-a-time model for MCP CI and the per-delivery model for agent runs. handleMCPCIJob reports on the events publisher the carrier hands it instead of a captured client. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: delete the consumers nothing in production reached jobs.new has a producer and no production consumer, and the agent dispatcher's Dispatch was only called from tests. Publishing jobs.new is unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(mcp): send MCP requests to agent workers through AgentControl Timeouts still honour only the deadline, not cancellation, exactly as today. The NATS no-responders error maps to ErrNoRoute and a timeout does not. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(agent-worker): serve MCP requests and backend.stop through agentRPCServer The agent worker's MCP tool and discovery reply subscriptions and its backend stop listener move behind an unexported agentRPCServer interface, served on NATS by nodes.NATSAgentRPCServer. The handlers become typed mcp.ToolHandler and mcp.DiscoveryHandler values that answer every failure with a reply carrying Error. Queue group (agent-workers), inline execution on the delivery goroutine, the background handler context, the unmarshal error reply texts and the reply-less backend stop subscription are unchanged. The backend stop handler takes the decoded backend name, so it can still close that backend's MCP sessions. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(messaging): remove helpers that only tests used BroadcastRoots, ControlRoots and SubjectRoot had no production caller. The roots spec now asserts every served root through ValidateSubject instead. MatchSubject moves back into the test support package, the only place that used it, with its table. NATSAgentRPCServer drops the subscription list it stored and never read, and NewNATSAgentRPCServer gets a doc comment. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(mcp): round trip the agent RPC server over a real NATS server One spec sends a tool request and a discovery request through NATSAgentControl to NATSAgentRPCServer and checks that the handlers see the decoded requests and the replies come back. It also puts an undecodable body on the tool subject and checks the server answers with an unmarshal error instead of leaving the requester to time out. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe the distributed transport seams The developer note now lists the final seams: fan-out, queues, both halves of the control verbs and of agent RPC, and the dial. It records the open items a second carrier has to handle. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test: pin the in-flight limit each queue consumer asks for The work queue specs pin what Consume does for a given limit, but nothing pinned which limit each production consumer passes. Changing the agent worker's MCP CI limit from 1 to 0 would have let MCP CI jobs run concurrently on each worker with every test green. Move the MCP CI Consume call into startMCPCIConsumer with the same wiring and pin that it asks for (WorkMCPCI, 1). Pin that NATSDispatcher.Start asks for (WorkAgentRun, maxConcurrent) for several limits. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: remove helpers the branch left without a caller SubjectJobCancelWildcard lost its last subscriber when the frontend stopped listening on jobs.*.cancel; the NATS permissions and conformance suite spell the subject out, so nothing reads the constant. decodeBackendStopRequest returned a stopAll flag that production dropped and only a test read. decodeBackendStop is now the single decoder with the same semantics: an empty body is stop-all, an empty Backend is stop-all, malformed JSON is an error. stopBackends still derives stop-all from Backend, so no reply changes. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(messaging): keep an explicitly empty agent queue a plain subscription Before the work queue seam the agent worker passed LOCALAI_AGENT_QUEUE straight to QueueSubscribe, so an explicitly empty value made a plain subscription and every agent worker ran every agent run. WithAgentRunRoute replaced an empty queue with agent-workers, which silently changed that. Keep the queue as given once the option is applied. An empty subject still falls back to agent.execute, since it never had a meaning of its own. The flag default stays agent-workers, so only an explicitly empty value reaches this. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct comments and record the PR B notes Fix the recordingFactory comment (it also records the parallel flag), document that a negative maxInFlight is unbounded and that Unsubscribe from a handler deadlocks, and say a permanently undecodable payload returns nil. Record controlHandler's undecodable return as a kept exception, and add the second carrier notes to the developer note: the reconciler has no ClientFactory option, the logs proxy honours HTTP_PROXY, verbs one carrier serves need an opt-out, terminal replies come from the result event, and agent runs publish through the NATS-bound EventBridge, which is not an additive change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
9eb5a9e61d |
feat(audio): remember speakers from diarization (#12414)
* feat(schema): validate portable speaker profiles Add the versioned profile schema for explicit speaker enrollment. Validate compatibility against separately supplied loaded-encoder metadata. Reject unusable speakers, invalid vectors, and inconsistent clean spans. This slice does not change HTTP routes, backend integration, or the UI. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(parakeet): export profiles with transcripts Export opt-in speaker profiles and trusted encoder metadata. Replay registrations by ID so duplicate display names keep independent vectors. Use one profile-capable diarization for slots, names, and clean spans. Assign timestamped ASR words to those slots without a second diarization. Preserve legacy opt-out and no-ASR behavior, and propagate failures. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(audio): enroll portable speaker profiles Gate profile exports with voice-recognition permission and validate registration against metadata from the loaded encoder. Preserve audio enrollment and independent registrations with duplicate display names. Exclude diarization and registration exchanges before API trace capture so persisted traces cannot retain profile vectors or JSON audio. Defer candidate dimensions to trusted loaded metadata. Sort candidates by registration ID so incompatible profiles cannot suppress legacy voices through registry iteration order. Keep portable identity checks closed when trusted metadata is unavailable. Test persisted traces, explicit slot zero, and selection through offline and live transport. Document privacy and the ephemeral registry lifecycle. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): remember speakers from diarization Add a Studio page for diarization and opt-in speaker profiles. Preview clean intervals from the original recording before explicit registration. Join profiles by raw speaker labels, preserve duplicate names, and relabel turns only after a successful save. Discard stale results when the model or recording changes. Share registration metadata with voice management without storing vectors or recordings from this flow. Document permissions and the global, ephemeral registry. Cover enrollment, permissions, previews, and asynchronous races with mocked Playwright tests. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: clarify HTTP speaker enrollment support Replace the stale enrollment limitation with the current HTTP workflow. Distinguish native transport from explicit registration and link its docs. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(parakeet): pin merged speaker profile support Use the merged commit from mudler/parakeet.cpp#80. Its tree matches the previously accepted native pin. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add diarization enrollment setup example Connect the existing gallery modes to the speaker enrollment workflow. Show installation, private profile export, explicit raw-slot registration, and later recognition without another export. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(blog): explain diarization speaker profiles Put the diarization walkthrough on the LocalAI website in the feature PR. Cover the three gallery modes, explicit enrollment, and privacy limits. Link setup instructions and keep availability conditional on feature support. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(blog): focus diarization on everyday use Explain what users can do with recordings before the setup steps. Replace the technical walkthrough with a short Studio guide and link readers to the existing reference for model names and developer use. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(blog): lead with speaker capabilities Present speaker recognition through everyday uses and a short UI flow. Keep technical reference details in the existing documentation. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(diarization): satisfy Go lint checks Avoid copying protobuf message state when extending backend status, check the multipart reader close result, and document the focused testing.T lint exemptions. Assisted-by: nib:gpt-5.6-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
2ae6cae70d |
feat(parakeet-cpp): name speakers from the shared voice registry (#12382)
* feat(voice): list registered voices and record which encoder made them The voice registry could register, identify and forget but not list, and it did not remember which speaker encoder produced an embedding. Add Metadata.Model and Registry.List, answered from the index the store registry already keeps for Forget. Needed so a backend can be given the registered voices that match its own speaker encoder. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * feat(voice): store the encoder model with a registered voice Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * feat(voice): pick the registered voices that match a speaker model Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * feat(proto): carry known voices and speaker names on diarize and live messages Assisted-by: Claude:claude-haiku-4-5 [Claude Code] * feat(diarization): name speakers from the voice registry When a diarization model has a speaker_model option, the endpoint sends the registered voices made by that encoder to the backend. The backend's name and name_score come back as extra fields next to the normalized SPEAKER_NN speaker, and the speakers summary carries the first name seen for each speaker. RTTM output and results without names are unchanged. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * feat(live): pass registered voices to a live session and surface speaker names Live sessions now send the registered voices that match the model's speaker_model to the backend, and each speaker segment carries the name the backend matched. The realtime segment event gains an optional speaker_name field. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * feat(parakeet-cpp): load a speaker model and build per-request voice registries Adds the speaker bindings (ABI v9 and v10, probed separately), the speaker_model, speaker_threshold and speaker_margin options, and a per-request registry builder over the known voices. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * feat(parakeet-cpp): name the speakers in Diarize from the known voices Diarize builds a per-request speaker registry from the known voices when a speaker model is loaded, calls the named C functions, and puts each slot's registered name and score on the segments. The registry is freed on every path. A library without ABI 10 reports Unimplemented instead of dropping the names. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * feat(parakeet-cpp): name speakers in the live scene stream The live scene stream now begins with a known-voice registry when a speaker model is loaded and the live config carries voices, and each closed speaker segment takes its slot's current name from the feed's names map. A segment that closes before its slot is identified has an empty name. The registry is freed after the stream, on every path. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * feat(gallery): speaker naming entries and docs for parakeet-cpp Add three gallery entries that load the WeSpeaker ResNet34 speaker model next to the diarization or realtime scene models, and document speaker names in the voice recognition, diarization, audio to text and realtime pages. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * fix(parakeet-cpp): skip an unusable registered voice instead of failing the request A registered voice with the wrong embedding size, or one the C side refused, failed the whole diarization request, so one legacy voice broke the model for every user. Skip such voices with a warning that does not carry the voice name, and take the plain path when none is left. Also map an exact 0 speaker threshold or margin to a tiny positive value, since the C side reads 0 as "use the default", and fix a stale comment about which contexts Free() walks. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * fix(diarization): warn once per model about voices from another encoder; document the privacy limit The different-encoder warning fired on every request. Log it once per feature and speaker model, then at debug level. Document that the global voice registry lets any caller of a speaker_model model learn matching names, and that skipped wrong-sized voices are logged. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * chore(parakeet-cpp): bump parakeet.cpp to 8c8cec0 (C-API v10) and check speaker naming against the real library The pin moves from 623a968 to 8c8cec0, which brings in everything merged in parakeet.cpp since: the voice identification change (C-API v9, #78) and raw-embedding enroll plus diarize-only speaker naming (C-API v10, #79). New real-library specs (gated on PARAKEET_BACKEND_TEST_SPEAKER_MODEL, _DIAR_MODEL, _WAV and, for the live path, _STREAM_MODEL) name the two speakers of two_speakers.wav from a committed pair of WeSpeaker embeddings, with the voices passed in reversed order. They also check that the float32 threshold reaches C through purego. The shared test loader now registers the v9/v10 and scene symbols as main.go does. The rebase onto origin/master had no conflicts. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
5613572f38 |
feat(systemone): validate requests and align the docs with Ollama's contract
All three routes now validate the request before it reaches a model: body size (413 over 64 KiB), state, question count, blank ids, option and level counts, and noul criteria keys. Forwarded decision requests skipped this before, so a malformed question surfaced as a backend error. The docs claimed the wire shape matches Ollama's. Field names and question types do; confidence, error shape, keep_alive and state rendering differ, and the docs now say so. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
70ce62901f |
refactor: name the capability decisions instead of systemone
The usecase describes what a model can do, and the category is the Decisions API. SystemOne stays as the wire contract: the /v1/systemone routes, the Score RPC question_type and the swagger tag are unchanged. The usecase, flag, auth feature, UI label, gallery tags and docs page are now decisions. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
b3d65fd538 |
fix(systemone): route NER models to the NER path and refuse decision models on permute and separate
vllm_decide refuses NER architectures and the NER entry point refuses decision architectures, so each model kind 500ed on half of the routes. A token_classify model now goes to the NER path on /v1/systemone, and /permute and /separate return 400 for decision models. Docs and instructions state which kind serves which route. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
b4852d62d2 |
feat(systemone): register the decisions API on auth and instructions
Adds a default-on systemone route feature for the three /v1/systemone routes and an /api/instructions area for them. No MCP tool is added: the endpoints run inference and are not admin install/edit actions, and the route-map test still passes. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
e03cf8dac8 |
feat(systemone): refuse models that do not declare the usecase
A chat-only model now gets a 400 naming known_usecases: [systemone] instead of a backend error. Configs declaring no usecases and token_classify models stay allowed so existing laya and GLiNER setups keep working. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
2fa36e7147 |
feat(parakeet-cpp): speaker diarization, sound detection and live scene events (#12335)
* feat(parakeet-cpp): load diarization and CED models and companions
Repin PARAKEET_VERSION to parakeet.cpp PR #75's head, which adds
parakeet_capi_model_kind (ABI v8). Bind the new diarization, sound
event and combined scene stream C symbols through the same
purego.Dlsym probe pattern already used for the batched JSON entry
point, so the backend still loads against an older libparakeet.so.
Load now classifies the loaded GGUF by role (ASR, diarization or
sound) via parakeet_capi_model_kind and can load up to two companion
models from Options[] (asr_model:, diarization_model:, sound_model:,
paths resolved against opts.ModelPath), verifying each companion's
kind and freeing every context opened so far on any failure. Free
releases the primary and every companion. AudioTranscription now
names the loaded role when it is not ASR instead of a generic model
not loaded error. The dynamic batcher starts only when an ASR context
ends up loaded, primary or companion.
This is groundwork only: the Diarize and SoundDetection RPCs and the
live scene stream that actually use these new roles land in later
commits.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): reset role fields on a failed companion load
loadRoles' freeLoaded only released the C contexts it had opened; it
left ctxPtr/diarCtx/tagCtx and companions pointing at those now-freed
contexts, so a later Free() on the same instance would double-free.
Zero all four alongside the CppFree calls.
Also route AudioTranscriptionStream and AudioTranscriptionLive through
notASRError when ctxPtr is unset but a diarization or sound model is
loaded, matching AudioTranscription: both used to return the generic
model-not-loaded error instead of naming the loaded role.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(parakeet-cpp): add speaker diarization
Implement the Diarize RPC for the parakeet-cpp Go backend, wired to
Nemotron-3-Diarization through libparakeet.so's diarization C-API.
Plain diarization uses parakeet_capi_diarize_pcm; when include_text is
set and an ASR companion is loaded, parakeet_capi_transcribe_and_
diarize_json fills each segment's text instead. Speaker labels are the
decimal index, or "unknown" for -1 (no diarized speaker overlaps).
min_duration_off merges same-speaker segments across a short gap
before min_duration_on drops the segments still too short, then ids
are renumbered. num_speakers/min_speakers/max_speakers/clustering_
threshold have no Sortformer equivalent and are logged at debug
instead of rejected.
Verified against the real Nemotron-3-Diarization + parakeet-tdt_ctc-
110m checkpoints on the two_speakers.wav fixture: correct A-B-A-B
speaker segmentation and matching speaker-attributed transcripts.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(parakeet-cpp): add sound event detection
Wire the SoundDetection RPC to the CED tagger context (p.tagCtx)
loaded by Task 1's role classification. It runs the whole clip
through a one-shot parakeet_capi_sound_stream_* session (window
10s, hop 10s, top_k set to the tagger's class count so every
drained window carries a full score list), averages each class's
score across the drained windows, sorts descending, then applies
the request's threshold and top_k (0 keeps every class).
No tagCtx returns FailedPrecondition; a libparakeet.so missing the
sound_stream symbols returns Unimplemented. Every C call runs under
engineMu, and the stream is always freed, even when a feed or drain
call fails partway through.
Verified against a real ced-tiny-q8_0.gguf on the rooster.wav demo
clip: "Chicken, rooster" tops the list at score 0.91.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): cancel sound detection mid-feed, shrink the lock
SoundDetection now checks ctx before each 10 s feed slice (mirroring
driver.go's feedSlices) and returns Canceled if the caller gave up,
so a long clip can be interrupted instead of feeding to completion
regardless. The stream is still freed on every path, cancellation
included.
Also narrow engineMu to the C calls: the drained JSON document is
now decoded after the lock is released, splitting soundStreamScores
into a locked soundStreamDrain (opts, begin, feed, drain, free) and
an unlocked json.Unmarshal.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(parakeet-cpp): stream speaker and sound events during live transcription
Add two additive proto fields, LiveSpeakerSegment and LiveSoundEvent,
repeated on TranscriptLiveResponse. When a diarization or sound
companion model is loaded, AudioTranscriptionLive now runs a no-ASR
scene stream (parakeet_capi_scene_stream_begin) beside the ASR
streaming session, feeding it the same PCM slices and forwarding any
closed speaker or sound events alongside the matching ASR delta, or
on their own when a slice has no ASR output.
The scene stream is freed and reopened on a mid-stream Config reset,
flushed with is_last before the closing FinalResult, and degrades
gracefully (a warning, not an error) when begin or a later feed call
fails, so live transcription keeps working ASR-only. Existing live
behavior is unchanged when no companion is configured, and no scene
C call is made in that case.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): keep scene events off the ASR critical path in live
Emit each slice's ASR result right after the ASR feed, before the
scene feed for that slice runs, so a companion diarization/sound
model never adds scene compute latency in front of the delta or
<EOU> that drives realtime turn detection. Closed speakers/sounds go
out afterward as their own response, so a slice with both now
produces two responses, ASR first. The live feed log line now
reports ASR and scene wall time separately.
Re-check the diarization/sound contexts a scene stream was begun
with against the live contexts before every feed, under the same
lock: Free() can race between an ASR feed and the matching scene
feed and free the model the stream borrows. A mismatch now returns
without touching the C side. Freeing the stream itself stays
unconditional; the scene stream's destructor only releases its own
buffers and never touches the borrowed contexts.
Also recover a panicking stub inside the live test goroutine instead
of crashing the test binary, and reset the live decode-lag tracker on
a mid-stream config reset, matching what its own comment already
promised.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): surface live speaker and sound events
Carry the backend's closed speaker segments and sound events
(TranscriptLiveResponse fields 7/8) through LiveTranscriptionEvent
as LiveSpeakerSegment/LiveSoundEvent (nanoseconds mapped to
seconds), and forward them from the semantic_vad live path.
Each speaker segment emits
conversation.item.input_audio_transcription.segment with speaker,
start, end and empty text under the turn's item id. Each sound
event emits conversation.item.sound_detection with one tag
(label, score = peak, index) and the event's new optional
start/end seconds fields, omitted when unset so the existing
unary/windowed sound-detection path is unaffected.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): keep start/end on a zero-second transcription segment
ConversationItemInputAudioTranscriptionSegmentEvent.Start/End used
omitempty, so a speaker segment starting at 0.0s dropped its
"start" key. Nothing emitted this event before the live scene-event
path, so drop omitempty: the segment always carries real times.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(gallery): add parakeet-cpp diarization, CED and realtime scene models
Add gallery entries for the new parakeet-cpp capabilities: standalone
Nemotron-3-Diarization, the same paired with the Parakeet TDT+CTC
110M ASR model for speaker-attributed text, CED-Tiny and CED-Base
sound classifiers, and a realtime scene bundle combining the
streaming EOU ASR model with diarization and sound companions.
SHA256 taken from the Hub API; licenses from each model card
(openmdw-1.1 for Nemotron-3-Diarization, apache-2.0 for CED,
cc-by-4.0 for the Parakeet ASR models).
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: document parakeet-cpp diarization, sound detection and live scene events
Cover the new parakeet-cpp capabilities across the feature pages:
Nemotron-3-Diarization as a diarization backend (with and without
speaker text, the ignored speaker-count hints, the Sortformer
voice-like-sound quirk), CED as a sound classification backend, the
asr_model/diarization_model/sound_model/diarization_latency companion
options, and the realtime live speaker/sound events (event shapes,
the speech-turn-only limitation, and using this or
pipeline.sound_detection but not both).
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): correct the realtime-scene license and wording nits
parakeet-cpp-realtime-scene mistakenly copied cc-by-4.0 from the
existing realtime_eou_120m-v1 entry; the model card lists the NVIDIA
open model license instead. Switch to the gallery's usual spelling
for that license and keep the diarization/CED licenses called out in
the description.
Also: audio-diarization.md now says getting per-segment text needs
both an asr_model companion and include_text=true on the request, and
audio-to-text.md's option table reads "Use on" (a pairing the loader
does not enforce) instead of "Allowed on".
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): reject a companion role that duplicates the primary's
loadRoles let a companion option (asr_model:/diarization_model:/
sound_model:) assign into a role field the primary already occupied,
for example asr_model: on an already-ASR primary. The companion's
context silently overwrote ctxPtr/diarCtx/tagCtx, and Free() only
walks those three fields, so the original primary context was never
freed again.
Reject a companion whose role the primary already holds before its
GGUF is even loaded, freeing everything loadRoles opened so far, the
same way a wrong-kind companion is already rejected.
Also warn, rather than silently fall through, when
parakeet_capi_model_kind reports PARAKEET_MODEL_KIND_NONE for a
successfully loaded primary; the primary is still treated as ASR,
matching today's behavior.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): cap live scene sound score retention
sceneBegin started the live diarization/sound companion stream with
the C API's default sound options, whose top_k keeps 5 scores per
window forever until drained. The live scene path never drains sound
scores (only the offline SoundDetection RPC does, with its own fresh
stream), so this window queue on the C side grew for the whole
session's lifetime.
Set opts.Sound.TopK = 0 before starting the scene stream: this
disables score retention while leaving sound event detection (onset/
offset), which the live path actually consumes, unaffected.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): merge diarization segments per speaker, harden Diarize
mergeCloseSegments only compared neighbors in the single start-sorted
segment list, so two same-speaker segments never merged once another
speaker's turn fell between them (A, B, A): the short B segment broke
the adjacency the merge relied on. Group segments by speaker first,
merge within each speaker's own start-ordered run, then re-sort the
result by start so interleaved speakers come back out in timeline
order.
Also harden Diarize's entry points the same way streamFeedDoc/
sceneFeed already are: diarizeCall re-checks p.diarCtx (and, on the
include_text path, p.ctxPtr) under engineMu right before the C call,
so a Free() racing between Diarize's own checks and the lock can no
longer reach the C side with a freed context. When the include_text
call returns NULL, last_error is now read from both contexts and
whichever came back non-empty is reported, since either side of the
pairing can be the one that failed. A WAV decode failure is reported
as InvalidArgument instead of an unwrapped/untyped error.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): harden SoundDetection's engine checks
soundStreamDrain ran every C call under engineMu but never re-checked
p.tagCtx there, so a Free() racing between SoundDetection's own
tagCtx==0 check and this lock could still reach the C side with a
freed context. Re-check p.tagCtx under the lock and return
ModelNotLoaded when it was cleared, mirroring diarizeCall's own
re-check. A WAV decode failure is now reported as InvalidArgument
instead of an unwrapped/untyped error.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(parakeet-cpp): cover a mid-session scene feed failure
feedSlicesScene already degrades gracefully when a scene feed call
fails mid-session: it frees the broken stream and carries the ASR-only
session forward. Add a spec covering that path end to end: the scene
stream is freed exactly once, later audio slices still produce ASR
responses, and no speaker/sound events appear before or after the
failure.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: fix the parakeet-cpp companion role table and realtime scene docs
audio-to-text.md's companion option table read "Use on" with a note
that the loader did not enforce the pairing; it now rejects a
companion whose role duplicates the primary's, so restore the
"Allowed on" wording and describe the real enforcement.
openai-realtime.md's live speaker/sound section claimed a mid-stream
session.update resets the companion stream and that it flushes on
session close; neither happens, since the realtime core opens one
live stream (and so one scene stream) per speech turn and closes it
at that turn's commit, with no mid-stream Config in between. Document
that lifecycle instead, state precisely that start/end are seconds
from the start of the turn's own audio, and note that the diarization
model starts a fresh session every turn, so a speaker index is only
meaningful within one turn. The example sound tag ("Rooster", index
17) did not match any real CED label; index 17 in ced-tiny-q8_0.gguf
is "Baby laughter". Replaced with "Chicken, rooster" at its real
index, 99.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): use CED's real index for Chicken, rooster
The scene feed comment and the live test's canned document gave
"Chicken, rooster" index 365. In CED's AudioSet label list it is 99,
which is also what the realtime docs show.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): call the test event accessor
The scene-event tests range over a method instead of its returned slice.
Call the synchronized accessor so the OpenAI test package compiles.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(parakeet-cpp): pin parakeet.cpp master with sound events
mudler/parakeet.cpp#75 (sound events, scene stream, model kinds) and
#74 (the missing <algorithm> include that broke the image builds) are
on master now. Pin 6dea76a instead of the #75 PR head, and update the
header comment the bump bot reads.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(parakeet-cpp): pin parakeet.cpp with ced.cpp on main
parakeet.cpp #76 moved its ced.cpp submodule from the head of
localai-org/ced.cpp#3 (a branch-only commit) to ced.cpp main, where
#3 landed with an identical tree. Pin 623a968 so the image builds no
longer depend on that branch.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(transcription): carry speaker labels on words and streamed segments
A diarizing backend could label transcript segments, but two paths
dropped the label: TranscriptWord had no speaker field, so live
transcription words and word-level timestamps could not carry one, and
the stream=true transcript.text.done event left the speaker out of
its segments.
TranscriptWord gains an optional speaker (proto field 4, additive).
It flows through the live event and result mapping, the JSON word
output of the endpoint and the CLI, and transcript.text.done now
includes a segment's speaker when there is one. Empty labels are
omitted, so responses without diarization are unchanged.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
(cherry picked from commit
|
||
|
Ettore Di Giacinto
|
9c156656bd |
Merge PR #12285: feat(failover): serve a model name from a chain of local and remote targets
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
590512d9eb |
feat(distributed): report worker version and show models in node inspector (#12328)
* ci: bump Hugo from 0.146.3 to 0.166.0 The hugo-theme-relearn submodule was bumped to 9.1.x in #12096, which requires Hugo >= 0.165.0. The pinned 0.146.3 broke the docs site build with a template error in alias.html that could not evaluate the Locale field on langs.Language. Bump HUGO_VERSION to 0.166.0 (latest stable) to satisfy the theme minimum and resolve the alias.html template error. Assisted-by: nib:claude-sonnet-4.5 [bash] [read] [edit] * feat(distributed): report worker version and show models in node inspector Workers now send their LocalAI build version and git commit at registration. The controller stores them on BackendNode and exposes them through the existing node list/detail API responses. The node inspector side pane now fetches and renders the list of loaded models (name, state, in-flight) instead of showing only a count, matching what the node detail page already displays. --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
f82efdb43b |
fix(models): fallback to application config default context size in /v1/models/capabilities (#12202) (#12216)
* fix(models): fallback to application config default context size (#12202) Honor appConfig.ContextSize in /v1/models/capabilities when model context_size is unset. * docs(models): explain context size fallback Describe the application default used by capability discovery and preserve the distinction between total context and per-request limits. Assisted-by: Codex:GPT-6 * fix(models): apply the default context size only when context_size is unset The request path applies the application default context size only when a model leaves context_size unset. An explicit 0 or -1 falls through to the backend fallback. The capabilities endpoint now does the same, so it reports the value the backend uses. Assisted-by: Claude:claude-opus-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |