mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
773cae7259cbd422f3fc368e5ecfd6ecebf899bf
341
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Stefan Walcz
|
b56b903c92 |
feat(config): let model aliases carry an opt-in profile block (system prompt, MCP, client tools) (#12661)
* feat(config): let model aliases carry an opt-in profile block An alias can now act as a profile of its target through an explicit `profile:` block carrying system_prompt, mcp (remote/stdio servers, plus the new mcp.auto) and the new client_tools. ResolveAlias applies only that block on top of a copy of the target, while Name, Model, Backend and the persisted revision stay the target's. The loader key is therefore unchanged and every profile runs on the target's backend process; today a persona needs a separate model config, which loads the weights a second time. The block is opt-in so existing aliases never change: top-level fields on an alias stay ignored, and an alias without a profile block resolves to exactly the target. A profile block on a non-alias config is rejected. A profile with its own MCP servers caches its sessions under the alias name (MCPSessionKey), because the cache slot keyed on the target's name is invalidated whenever its config hash changes and two server sets would tear each other's sessions down. mcp.auto enables all configured servers on /v1/chat/completions when the request does not name any in metadata.mcp_servers, which plain OpenAI clients cannot send. client_tools: drop discards client tools and tool_choice before the request is merged (chat completions, responses, Anthropic messages), so only the model's own MCP tools are offered. Both keys work on any model and inside an alias's profile block. Refs #12548 Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(openai): only auto-enable model MCP when MCP is enabled and permitted mcp.auto turns a plain /v1/chat/completions request into a server-side MCP request. It now applies the same controls as the dedicated MCP routes before any server is selected: the instance switch (LOCALAI_DISABLE_MCP) and, with auth on, the caller's "mcp" feature, checked with auth.HasFeatureAccess against the auth database that the chat route now passes to ChatEndpoint. A denied request is answered as a plain chat, without contacting a server or adding tools, rather than rejected: the client did not ask for MCP, it only reached a model that offers it. Endpoint specs drive ChatEndpoint against a counting MCP fixture server: MCP disabled and a user with chat but without mcp see no tools and no connection; MCP enabled and a user with mcp get the fixture tool. The user specs need the auth build tag (SQLite auth database). Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(mcp): resolve profile aliases in the MCP discovery endpoints GET /v1/mcp/servers/:model, /v1/mcp/prompts/:model and /v1/mcp/resources/:model looked the model up with GetModelConfig, which returns the alias stub. A profile alias carries its servers in its profile block, so the stub has none of its own and the endpoints reported an empty list for an alias whose chat requests do use tools. Resolve the alias the way the request middleware does before reading the MCP config, and open the sessions under MCPSessionKey so discovery shares the profile's session cache slot instead of tearing down the target's. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * feat(config): opt-in evidence check for profile aliases A profile with server-side MCP tools (profile.mcp, mcp.auto) runs its tools on the server and returns only the answer text. On the next turn the client sends the history back without those tool calls and results, and the model then tends to skip the tool and answer from nothing. In a measurement on a follow-up question, 0 of 12 turns called the tool and the answers carried invented numbers (e.g. "0" where the true value was 9); with the tool trail kept in the history the tool was called 12 of 12 times. A prompt instruction does not reliably prevent this, so the check sits at the output. New profile keys: profile: evidence_check: numbers: true # enables the check note: "..." # optional, defaults to an English note Rule, per request and stateless: when the check is on, no MCP tool call succeeded on the server during this request (a call that returned an error delivers no evidence and does not count), and the final answer contains a number (\b\d+([.,]\d+)?\b) that occurs in none of the request's input (the profile's system prompt and every message as received, before compression), the note is appended after a blank line. "3,5" and "3.5" count as the same number. In a streamed response the note is one extra content delta before the finish chunk. Choices that hand tool calls to the client are not checked. The setting is only read from a profile block, so plain models and aliases behave as before; a note without numbers: true is rejected at validation. Tests (Ginkgo): the number rule (invented number, number from the input, no numbers, comma/dot decimals, system prompt as evidence, tool ran), the overlay into the resolved config with plain alias, target and a top-level key as counter-checks, validation, and ChatEndpoint specs on a stubbed backend for both paths: note on an invented number, none for a number from the request, none for a profile without the check, none after a successful MCP tool call against an in-process MCP server, and the note again after a failed tool call (arguments that are not JSON). Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> --------- Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> |
||
|
mudler-agentandEttore Di Giacinto
|
dce91ba7aa |
fix(backends): prefer installed development builds (#12782)
* fix(ci): capture and surface ginkgo errors in run-coverage.sh The coverage script ran ginkgo per test root with `|| fail=1`, which swallowed the exit code. Combined with `make --output-sync=target`, which buffers all output and flushes at the end, the actual ginkgo error was invisible in interleaved CI logs. Every test suite passed but the script still exited 1, and the error message was buried in 22 000 lines of buffered output. Each ginkgo invocation now captures combined stdout+stderr to a per-root log file. On failure the last 80 lines are printed to stderr so the error is visible in the CI log. On success the log is replayed to stdout so test output remains visible. The build flags (--tags, --coverpkg) are quoted correctly so values containing spaces (e.g. --tags="debug auth") are preserved through eval. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): create output dir before writing ginkgo logs The previous commit added per-root log files but forgot to create the output directory before the first ginkgo invocation. On CI the coverage/ dir did not exist yet, so the shell redirect to cover-pkg.out.log failed with "No such file or directory" before ginkgo even ran. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(core/http): add canvas preview route to auth coverage allowlist Commit |
||
|
mudler-agentandEttore Di Giacinto
|
f85087f992 |
feat(tts): stream Qwen text and add assistant preset (#12770)
* feat(tts): stream Qwen text through audio pipelines Feed cleaned LLM deltas into one persistent Qwen synthesis session. Return audio before input ends and preserve the live RPC through model cache, staging and in-flight wrappers. Stop generation on sink failures or cancellation without replaying speech through a fallback. Pin the incremental native fork and verify early audio, late text, finalization and cancellation with lifecycle and real-model checks. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(tts): use the organization Qwen fork Fetch the reviewed incremental synthesis commit from localai-org so backend builds use the organization-maintained fork. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add a Qwen streaming assistant preset Expose an English pipeline using existing VAD, ASR, LLM and named-voice TTS models. Enable incremental text and audio delivery, and document separate component installs because the gallery has no dependency installer. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): use Parakeet TDT v3 for the assistant Select the existing Parakeet TDT 0.6B v3 transcription entry and update its prerequisite install command. Keep the preset's English voice and incremental Qwen synthesis configuration. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(kokoros): reject unsupported backend RPCs Implement the MotionStream and SlotCache methods required by the shared protobuf service. Return Unimplemented so Kokoros compiles without advertising support for either operation. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(tts): mark cleanup and logging results ignored Explicitly ignore connection shutdown, test server shutdown and Ginkgo diagnostic writer errors. These operations do not replace the stream's inference or sink errors, and errcheck requires their return values to be acknowledged. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(tts): stream buffered clauses in realtime pipelines Start clause synthesis while LLM text is still arriving when both streaming flags are enabled. Keep explicit clause opt-out and native Qwen sessions, bound automatic queued work, and choose streaming support on resolved failover targets. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(tts): bound synthetic voice clone generation The sine-wave reference checks speaker conditioning and WAV output, but does not provide a meaningful speech EOS target. Limit that smoke request to 128 frames and verify the request limit reaches native synthesis. Leave production sampling defaults unchanged. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(tts): require explicit clause buffering opt-in Keep whole-reply synthesis for nonnative engines when clause_chunking is unset or false. Use the bounded cancellable worker when explicitly enabled, with unary clauses when TTS streaming is disabled. Preserve native Qwen precedence for every clause setting. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add VITS and Piper streaming assistants Provide comparable full pipeline configurations with shared VAD, Parakeet transcription, and Qwen chat components. Enable clause buffering explicitly and document the separate prerequisite model installs and Talk selection. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Stefan Walcz
|
f4505dbb5d |
feat(pii): optionally block content the detectors cannot inspect (#12755)
The PII middleware scans the text of every message, but images, audio,
video and files pass to the backend unchecked. For an egress policy
("nothing sensitive leaves the box") that is a hole: a photographed
invoice or an attached PDF reaches a cloud-proxy model without any check.
Add pii.block_unscanned_media. While PII is enabled for a model and the
option is set, a request that carries such content is rejected with the
usual 400 pii_blocked response and an audit event with pattern
__unscanned_media__, before any detector runs. Adapters report the
kinds of unscanned content through a new optional Adapter.Unscanned:
OpenAI chat (non-text content blocks), Anthropic messages (image and
document blocks; tool blocks stay allowed) and Ollama chat (images).
Off by default, so existing configurations behave as before.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
|
||
|
github-actions[bot]andmudler
|
fdfb171c3e |
chore: bump inference defaults from unsloth (#12732)
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler-agentandEttore Di Giacinto
|
c7ce0c602c |
fix: stop listing the models directory on request paths and cache GGUF header reads (#12683)
* fix(http): stop listing the models directory on embeddings and bearer requests Every embeddings request read the whole models directory before the body was parsed, even when the body named its model. On a network-backed models directory this cost seconds per request. Embeddings now binds the body first and looks up a default model only when the body has none, like chat. The pre-bind default lookup and the gpt-4o constant default are gone from that route. The first-available default lookup skips the directory when the filter cannot accept an unconfigured file. Every usecase filter rejects files without a config, so the scan never added results for them. Match-all, nil and name filters keep the scan. The bearer token check is now an exact test: a configured name, or one stat of that name in the models directory. The value is a literal, not a regular expression, and a name that is not a local relative path never matches a file. Assisted-by: Claude Code:claude-haiku-5-5 * perf(config): cache GGUF header reads for model defaults Model defaults re-parsed the GGUF header on every SetDefaults call, which runs per request and at startup reload. Put a bounded LRU in front of the parser, keyed by absolute path and validated by size and modification time. The cached value is a snapshot of the fields the defaults use, not the parser output. A changed size or mtime is a miss. Missing files and parse errors are not cached, so the existing fallback is unchanged. LOCALAI_DISABLE_GUESSING, the empty-path guard, the alias early return, the panic recovery and the context-size default are all kept in front of the cache. Assisted-by: Claude:claude-haiku-5-5 * test(http): cover listing-free default lookup on every usecase route Assisted-by: Claude Code:claude-haiku-5-5 * feat(config): persist the GGUF header cache per instance The header cache kept parsed GGUF results only in memory, so every restart parsed each model file again before its defaults were known. Write the cache to a per-instance file in the data path and load it at startup. A restarted frontend with a fixed instance ID starts warm. The file is versioned and records the estimator inputs (gguf-parser version, candidate contexts, auto context size, offload count). A mismatch on any of them makes the file ignored. Each entry keeps path, size and mtime, so a changed model file is a miss on load. Writes run on one background saver, debounced, through a temp file and rename. The entry bound applies to the file as well. Without a data path the cache stays in memory only. Documented as a per-instance exception in the model configuration docs. Assisted-by: Claude:claude-haiku-5-5 * docs(embeddings): document model selection and the no-model error Assisted-by: Claude Code:claude-haiku-5-5 * refactor(config): split the header cache into a generic cache The GGUF header cache mixed two things: the mechanics (bounded LRU keyed by path and identity, per-instance file, debounced atomic save, tolerant load) and the GGUF fields. Move the mechanics into headerCache[S], driven by a headerCacheSpec[S] that names the file prefix, schema, fingerprint, parser and the snapshot encode/decode pair. The GGUF cache becomes one instance of it. File names, the identity check, LRU size, debounce, atomic write and the corruption handling are unchanged. The on-disk JSON keeps the "estimator" key so existing files still load. Assisted-by: Claude Code:claude-haiku-5-5 * feat(config): share one GGUF header cache file between frontends Every frontend used to write its own header cache file, named after its instance ID, so a header read by one frontend was never reused by another. The cache now uses one file, header-cache.json, in the data path, for all frontends and for standalone mode. The instance ID no longer changes the file name. Frontends share the file in three ways. A save first merges the file's entries in, so one frontend's write does not drop another's entries. A miss reloads the file when its modification time has changed, and a hit never reads it. Writes go to a temp file that is renamed into place, so a reader never sees a partial file. Two frontends saving at the same moment can still drop each other's last entries. That costs one re-parse, never a wrong value, because every entry is checked against its model file before use. The shared file is valid only when every frontend mounts the same data volume. The distributed-mode and model-configuration docs state this mode and its volume limit. Assisted-by: Claude:claude-haiku-5-5 * feat(config): give an instance ID its own header cache file Assisted-by: Claude Code:claude-haiku-5-5 * style(config): gofmt the distributed config struct Assisted-by: Claude Code:claude-haiku-5-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
4ebff15e76 |
chore(motion): tidy proto, imports and usecase guess; add weights licence
Remove a stray blank line in backend.proto and group the imports in pkg/motion/validation.go. Decide the motion usecase guess from the backend capability registry instead of a hardcoded backend name, so variant builds of the backend resolve too. Declare the weights licence in the gallery entry, as stated on the model card. Assisted-by: Claude Code:claude-sonnet-5-5 |
||
|
Ettore Di Giacinto
|
db929f4a08 |
test(config): assert every usecase flag is a distinct bit
FLAG_MOTION and FLAG_UPSCALE were both declared as 1 << 26 on separate branches, and a merge combined the two const blocks without a conflict. Add a spec that checks every flag from GetAllModelConfigUsecases() is a single bit that no other flag uses, so a collision fails the tests. FLAG_MOTION itself moves to 1 << 27. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
3de91d48d9 |
feat(motion): add GEM-X backend and duplex streaming API
Add the GEM-X native backend with live pose and root displacement output, bounded duplex WebSocket streaming, session authorization, metrics and usage accounting. Share the embedded bidirectional RPC channel adapter across streaming backends.
Include backend build and gallery integration, API documentation, schemas and tests.
Assisted-by: Codex:gpt-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
(cherry picked from commit
|
||
|
Ettore Di Giacinto
|
f420bfcc26 |
fix(pii): give extend_to_previous_word its own settings order
It shared order 215 with extend_to_next_word, so the settings UI had no defined order between the two. Move it to 216 and shift the protected terms fields. Pin that a number in front of a hit is never taken. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Stefan Walcz
|
e3341d21a0 |
feat(pii): extend a detection over the previous word (extend_to_previous_word)
The mirror of extend_to_next_word. A name model often tags the surname with
high confidence and scores the first name in front of it below min_score,
so "Gast Greta Waldmeister" became "Gast Greta [REDACTED:ner:LASTNAME]".
extend_to_previous_word lists entity groups whose accepted detections also
cover the word directly before them on the same line. The preceding word
must consist of letters (plus '-' and the apostrophe): a word with digits -
an amount, a date, a room number - is never taken, nor is a protected term,
and the search never crosses a line break.
Docs: operations/middleware.md and advanced/model-configuration.md.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
Stefan Walcz
|
e16b9d04cc |
feat(pii): protected terms that pass the NER filter unchanged
A name model reads many business names as surnames. Measured with
privacy-filter-multilingual on 24 synthetic German receipts and
invoices: the supplier name ("Hotel Seeblick", "Café Lindenhof", ...)
was masked as LASTNAME (0.78-0.98) or FIRSTNAME in 9 of 12 receipts
that carried one. The supplier is exactly what the upstream model needs
to categorise a receipt, and no score floor helps at those confidences.
pii_detection.protected_terms (inline) and
pii_detection.protected_terms_files (files inside the models path, one
term per line, '#' comments, re-read when size or mtime change) list
values that must pass unchanged. Each whole-word, case-insensitive
occurrence is replaced by a neutral placeholder before the detector
runs; every detection is mapped back onto the original text with the
placeholder parts cut out, so a protected term is never masked or
blocked, the detector never sees it as context, and event offsets keep
addressing the original text. A word extension (extend_to_next_word)
is dropped rather than run into a protected term.
Terms shorter than three characters are ignored. A missing or
unreadable file is logged and skipped: fewer protected terms means more
masking, never less. Paths are verified to stay inside the models path.
Opt-in, empty by default. Covered by Ginkgo specs in
core/services/routing/pii/protected_test.go (detector-visible text,
exact restoration, straddling detections, whole-word/case/line-wrap
matching, the segment path, file loading and reload), a YAML round-trip
spec, registry entries and the middleware/model-configuration docs.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
Ettore Di Giacinto
|
180822c546 |
fix(mcp): require https for the oauth2 token_url
The client secret is sent to token_url, so a plain http endpoint would expose it. Reject non-https URLs except for loopback hosts, which keeps local identity providers and tests working. Document the restriction, recommend client_secret_env over a literal secret, and note that rotating a *_env secret needs a restart because sessions and cached tokens are keyed by configuration, not by the environment value. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-bot
|
340b2537ae |
fix(mcp): redact credentials in JSON logs
JSON log handlers bypass String and serialize exported credentials.
Implement LogValue so remote server entries use the same redacted
representation in text and JSON logs. Extend the regression test
to cover both handlers.
Assisted-by: Codex:GPT-6
(cherry picked from commit
|
||
|
Stefan Walcz
|
beaeb05e52 |
feat(mcp): support OAuth2 client_credentials for remote MCP servers
Remote MCP servers could only be authenticated with a static bearer
token. Deployments that put their MCP servers behind an OIDC provider
(authentik, Keycloak, Entra ID, ...) issue short-lived service tokens
instead, so LocalAI had to be given a long-lived key or could not
connect at all.
A remote server entry now accepts an optional "oauth2" block
(token_url, client_id/client_id_env, client_secret/client_secret_env,
scopes, endpoint_params). The token is fetched with the
client_credentials grant by an http.RoundTripper installed on the MCP
client's HTTP client, so streamable HTTP and SSE transports both use it.
The token is cached and refreshed at half of its lifetime; if a refresh
fails the current token is kept until 30s before expiry, after which
requests fail with an error instead of going out unauthenticated.
"token" and "oauth2" are mutually exclusive and inconsistent blocks are
rejected when the model config is loaded. Static tokens behave as
before. MCPRemoteServer now implements fmt.Stringer so the existing
debug logs that print whole server entries no longer write the token
or client secret in clear text.
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
Assisted-by: Claude Code:claude-opus-5-5
(cherry picked from commit
|
||
|
Stefan Walcz
|
9b13408c94 |
feat(pii): extend a detection over the next word (extend_to_next_word)
Address models tend to tag a postal code with high confidence but miss
or under-score the town right after it. Measured with
privacy-filter-multilingual on 24 synthetic German invoices: the postal
code was masked every time, while the town after it stayed in the clear
in 6 of 18 cases (4 not detected at all, 2 tagged CITY at 0.43/0.47,
below the configured floor). Masking the code alone leaves the pair half
redacted: "[REDACTED:ner:ZIPCODE] Ludwigshafen".
pii_detection.extend_to_next_word lists entity groups whose accepted
detections also cover the word that directly follows them on the same
line, with the same action and group. Horizontal whitespace is skipped,
a line break ends the search, and nothing is extended when no word
follows, so a hit never grows over bare whitespace or onto the next
line (or across the segment separator on the multi-message path).
The option is opt-in and empty by default; existing detector configs
behave exactly as before. Covered by Ginkgo specs in
core/services/routing/pii/extend_test.go (fail without the change), a
YAML round-trip spec, a registry entry for the model editor, and the
middleware and model-configuration docs.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
mudler-agentandEttore Di Giacinto
|
8a03ce7f6c |
perf(config): skip GGUF guessing for pure aliases (#12629)
Aliases have no model file, so guessing attempts to parse the models directory. Skip that work while retaining the context fallback and normal alias validation. Use a private call-local parser dependency to verify operation counts without a mutable global hook. Cover concrete guessing, defaults, alias validation, and resolved target equivalence. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
0177e4988c |
feat(images): add ESRGAN upscaling and Studio workflow (#12620)
* feat(images): add upscale model capabilities * fix(config): keep SD GGML upscale opt-in * feat(stablediffusion): add lazy ESRGAN upscale bridge Keep native image ownership in C++ and serialize context creation, inference and teardown. Use upscale_scale in backend options as the explicit upscale-only marker because ModelOptions does not carry usecases. Assisted-by: Codex:gpt-5 * fix(stablediffusion): validate ESRGAN settings * fix(upscale): forward typed settings and contain resolved model paths Typed settings replace legacy options without duplicates. Resolve model and trusted root symlinks before caching the model path, rejecting missing files and escapes. Assisted-by: Codex:gpt-5 * fix(upscale): validate scale and clean failed HTTP outputs Require an explicit scale bounded to 1-16, check backend results, and keep destination cleanup owned by the allocating endpoint. Cover validation, result failures, cleanup, and successful URL responses. Assisted-by: Codex:gpt-5 * feat(studio): add inline image upscale workflow Assisted-by: OpenAI:gpt-5 * test(studio): serve upscale capability fixtures * fix(stablediffusion): default ESRGAN tiling to full frame * feat(gallery): add RealESRGAN anime upscaler * fix(stablediffusion): allow full-frame ESRGAN tiles * fix(routes): select explicit upscale models Give image upscaling dedicated usecase-filtered middleware and keep ordinary image models out of the endpoint. Mark and verify the legacy Diffusers x4 gallery model as an explicit upscaler. Assisted-by: OpenAI:gpt-5 [shell] * refactor(upscale): keep settings in backend options * test(config): keep upscale options backend-specific * fix: keep upscale routing metadata out of backend options --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
0e0e91f95c |
feat(diagnostics): add opt-in profiling (#12610)
* feat(diagnostics): add private opt-in profiler Keep profiling on a dedicated loopback listener rather than the public API. Validate settings and bind synchronously so startup reports configuration and socket failures. Own sampling and shutdown for the run process. Cover private handlers, blocked cmdline access, sampling cleanup, and serving failures with focused Ginkgo tests. Assisted-by: nib:gpt-6-astra * feat(diagnostics): add sanitized phase timing Carry generated request and reload identities through private contexts. Keep timing disabled without allocations and emit only validated fields at Info level when enabled. Cover identity isolation, nested durations, inheritance, and sanitized logging with focused race-tested Ginkgo specs. Assisted-by: nib:gpt-6-astra * fix(diagnostics): retain request observations Reuse an existing request observation across recorders to avoid splitting correlation. Keep the original context, identity, and sink without new allocations. Preserve nil-recorder behavior and fresh reload identities. Verify the regression fails before the fix and passes with the full Diagnostics race suite afterward. Assisted-by: nib:gpt-6-astra * feat(cli): own opt-in diagnostics lifecycle Expose profiling and phase timing flags without enabling them for debug. Bind the private profiler before application initialization and propagate serving failures through the run lifecycle. Bound signal cleanup and preserve preload-only backend startup. Cover flag parsing, preload isolation, cancellation, and once-only shutdown. Document initialization and shutdown cancellation limitations. Assisted-by: nib:gpt-6-astra * feat(config): measure filter and reload phases Separate request lock waits from filtering and held work. Aggregate reload filesystem and parsing durations without logging inside the config lock. Preserve caller option execution under the lock after metadata collection. Extract only the internal diagnostics option before waiting for the lock. Keep direct callers uninstrumented unless they supply diagnostics options. Verify focused Diagnostics examples with the race detector, including option ordering, gated contention, errors, panic propagation, and parity. Assisted-by: nib:gpt-6-astra * fix(config): separate early reload observation Opaque callbacks cannot select timing before their execution point. Carry the startup recorder explicitly through loader construction. Apply late options once under the lock and honor composed last writes. Document early and late phase boundaries and observation identities. Report malformed gallery syntax as an internal parse failure without changing strict or non-strict loader returns. Assisted-by: nib:gpt-6-astra * fix(config): preserve per-model option callbacks The resolved snapshot suppresses existing callbacks during model defaults, even when diagnostics are disabled. Pass original options through the existing defaults pathway without adding diagnostics-driven replay. Test stateful options with two models in enabled and disabled modes. Preserve three calls and thread defaults of two and three. Assisted-by: nib:gpt-6-astra * feat(model): observe discovery phases Carry request diagnostics through config and loose-file discovery to separate directory enumeration from in-memory filtering and fallback. Keep existing APIs as background-context wrappers and preserve discovery policies, ordering, duplicates, errors, and filesystem operation counts. Add focused parity, privacy, phase, and disabled-allocation tests. The focused diagnostics race gate passes for model and galleryop. Assisted-by: nib:gpt-6-astra * feat(http): observe extraction phases Generate diagnostic identity before request discovery, independently of client correlation headers. End each extraction segment before inference so downstream latency does not inflate discovery measurements. Carry observations through discovery helpers and rebuilt request contexts without changing cancellation or model selection. Cover disabled parity, error outcomes, privacy, concurrent identity isolation and route contexts. Assisted-by: nib:gpt-6-astra * fix(http): retain diagnostics across failover Failover restores its entry request before retrying. Initialize the observation before that capture so direct extraction retains one ID. Keep extraction spans per attempt and end them before downstream work. Exercise two real targets without default middleware, including replay, shared identity, phase boundaries, and timing-off response parity. Assisted-by: nib:gpt-6-astra * feat(model): observe initialization and routing Mark loader entry before cache and admission checks. Measure only the ModelRouter callback, without attributing transport or worker latency. Test inherited identity and sanitized events with fake backends. Exercise the actual Echo router to keep profiles private across debug and timing options. Cover capture shutdown, forced close, and lifecycle failures. Assisted-by: nib:gpt-6-astra * docs(diagnostics): guide bounded investigations Document private profiling, sampling ownership, and phase boundaries so operators can interpret captures without assuming request attribution. Add Ginkgo drift checks for flags, defaults, phases, links, and capture warnings. These checks do not certify the operator prose. The focused documentation race test passes after four expected failures. The consolidated Diagnostics race gate passes across all eight packages. This gate does not validate the full suite, auth, or production UI. Assisted-by: nib:gpt-6-astra * refactor(cli): restore inline run lifecycle Remove diagnostic startup hooks so application startup stays readable in Run. Keep the profiler as an optional sidecar with bounded cleanup and an error watcher, without a general lifecycle runner. Replace hook simulations with real CLI startup checks and component capture tests. Document that application cleanup can exceed the private listener budget. The eight-package Diagnostics race gate passes. A cleanup mutation fails the real CLI listener-release regression as expected. Assisted-by: nib:gpt-6-astra * fix(cli): retain concurrent startup errors Cancellation must not hide a public API bind failure. Join the public serve result, cancellation cause, and HTTP shutdown errors inline. Ignore ErrServerClosed only after intentional shutdown. Exercise real startup with an occupied API port and a failed private listener. Isolate the Linux socket fixture in a subprocess and wait for cancellation acknowledgment without timing sleeps. Assisted-by: nib:gpt-6-astra * refactor(http): make request context primary Move context preparation into the request extractor. Diagnostics remain an optional addition, not the entry point for request processing. Preserve context initialization before failover captures the request. Assisted-by: nib:gpt-6-astra --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
895d50385f |
fix(distributed): converge model configs across frontends (#12558)
* fix(galleryop): announce model changes before the preload After a gallery install or delete, the replica that ran it replaced its config loader, then preloaded every installed model, and only then published the models invalidation. The preload does remote lookups and checksums for each model, so on a large models directory peers learned about the change minutes after the originator listed it. When the preload failed or the operation was cancelled, the event was never sent. Publish the invalidation, and apply the delete lifecycle, as soon as the loader holds the new set. The preload still runs afterwards with its own error handling. Its failure is reported on the operation, but it no longer rolls back a deletion that peers have already applied. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): resync model configs from the models directory Frontends refresh their model configs only when a models invalidation arrives on NATS. NATS keeps no history, so a frontend that is disconnected when the message is published never applies the change. It keeps serving the old config, for example an alias that points at the previous model, until some later change happens to touch it. Each frontend now reruns the peer reconcile against the shared models directory after every NATS reconnect, and every --model-config-resync-interval (default 30s) when a config file changed. The pass names no model, so only models whose file changed get a revision transition, and an unchanged directory costs one read of the config files. The reconcile replaced the whole loader with a parse of the models directory, which dropped models loaded with --config-file and published a deletion revision for them. Configs defined outside the directory are now kept, both there and after a gallery install. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nodes): stop stale frontends from retargeting alias rules A scheduling rule keyed by an alias derives its target from the alias mapping of the frontend that reads it. Every frontend keeps its own copy of the model configs, so after an alias is repointed a frontend that has not reloaded it still resolves the old target. Two frontends then rewrote the rule's stored target_model against each other on alternate reconciler ticks, and the outdated one scaled up the model the alias used to point at. The registry already records the accepted config revision of each model. A frontend now derives a rule's target from its own alias mapping only when its config revision for the rule's name matches that record. Otherwise it keeps the stored target_model: it neither writes the column nor reconciles replicas of the old target. The check reads the database only for a rule whose stored and derived targets differ. With no accepted revision on record, the old behaviour stays. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
fa25493c9e |
feat(config): let parakeet-cpp declare the speaker_recognition usecase
parakeet-cpp now implements VoiceEmbed and VoiceVerify, so list them in its backend capabilities together with the speaker_recognition usecase. The usecase is not guessed, because most parakeet models have no speaker encoder: a bundle declares it in known_usecases, which is enough for /v1/voice/* and for voice_recognition.model. Add a realtime pipeline test where one bundle model names the vad, transcription, sound_detection and voice_recognition stages and the gate resolves the speaker through the backend's VoiceEmbed. Assisted-by: Claude Code:claude-sonnet-5-5 |
||
|
mudler-agentandEttore Di Giacinto
|
99043b442c |
feat(router): route with native decision models (#12449)
* fix(schema): preserve SystemOne image inputs Assisted-by: OpenAI * test(schema): follow Ginkgo conventions for decision inputs Assisted-by: OpenAI * feat(llama-cpp): dispatch native decisions through Score Upgrade the stock dependency and reconcile Score/TTS patches. Reuse native decision parsing, tasks, formatting and response-reader cleanup; preserve ordinary scoring admission and guard older dependencies. Assisted-by: OpenAI * refactor(systemone): share request and model validation Assisted-by: OpenAI:gpt-5 * fix(systemone): preserve HTTP wire-byte validation limit Keep structural validation separate from the serialized internal request bound so HTML escaping cannot reject valid HTTP payloads. Assisted-by: OpenAI:gpt-5 * feat(systemone): bound images and account native decisions Preserve public wire limits independently from router serialization. Reject unsupported NER images, map native request/capability errors, and stamp explicit usage once. Advertise decisions for stock llama-cpp. Assisted-by: OpenAI * fix(systemone): record usage on registered native route Exercise real registration and billing with a mock native backend. Reject empty native responses, malformed image URLs, trailing JSON, and wire overflow including whitespace. Assisted-by: OpenAI * feat(router): add lazy native decision transport Bind named models through internal ModelSystemOne calls with shared validation and bounded abandoned operations. Remove request and echoed-error contents from decision traces. Assisted-by: OpenAI:gpt-5 * feat(router): classify overlapping policies with native decisions Ask independent noul questions, validate probabilities and preserve first-superset routing. Wire the central factory with config-sensitive invalidation and cancellation-safe resolution. Document native framing and bounded operation limits. Assisted-by: OpenAI:gpt-5 * feat(gallery): add pinned Julia-1 native decision model Add a separate text-only llama-cpp Q8 entry with pinned Apache-2.0 source provenance and checksum. Installed using the gallery installer and exercised choice, score and noul on CPU. Assisted-by: OpenAI * test(router): verify native decisions through central factory Add an opt-in real-model Ginkgo integration covering the native Go loader and C++ transport, token usage, independent overlapping labels, and candidate selection. Document owned-server execution and the intentionally non-quality threshold. Assisted-by: Codex:gpt-5 * fix(llama-cpp): align upstream pin and preserve decision signatures Advance to bed0a856 without losing the automated upstream bump. Detect full-request fill_task support at compile time and forward every question for Nimble framing while retaining the earlier native signature. Preserve reconciled SCORE/TTS patches; add standalone compatibility coverage. Assisted-by: Codex:gpt-5 * feat(gallery): add native decision family defaults Pin Laya, Kev-4B, lev, OpenJev and Nimble artifacts. Verify Laya/Kev/lev gallery installs and CPU contracts on both native pins; clearly mark OpenJev/Nimble runtime validation pending and their noncommercial licenses. Assisted-by: OpenAI * docs(decisions): clarify integrated Nimble prerequisite Record the exact combined backend pin while retaining pending OpenJev and Nimble installation/runtime validation status. Assisted-by: Codex:gpt-5 * fix(gallery): indent native decision model sequences Match repository yamllint indentation for Laya, Kev, lev and OpenJev list fields. Parsed gallery data is unchanged; reproduce CI gallery lint failure before the whitespace-only fix and pass the same command afterward. Assisted-by: Codex:gpt-5 * docs(decisions): record OpenJev and Nimble CPU validation Record gallery installation, checksum/metadata verification and multiquestion native smoke results on bed0a856. Retain noncommercial and text-only limitations without accuracy or deterministic-output claims. Assisted-by: OpenAI * fix(ui): expose native Decisions router classifiers Select classifier models using metadata-driven capability routing, retain tuned thresholds, and validate native decision selections before saving. Cover both native backends and create/save/reopen in the real React editor. Assisted-by: Codex:gpt-5 * fix(router): exclude aliases from native decision discovery Check the originally named config before advertising native Decisions eligibility. Retain target capability inheritance for ordinary generation aliases. Exercise the actual capabilities endpoint with native models on both backends, aliases, and disabled models. Assisted-by: Codex:gpt-5 * feat(systemone): share bounded multimodal input validation Preserve text wire limits while admitting bounded PNG/JPEG decision input. Share collection and header validation across internal and public callers and keep the native runner response budget independent. Assisted-by: OpenAI:API-assistant * fix(systemone): bound admission lifetimes and validate complete images Retain shared admission leases through actual work completion, including abandoned internal operations. Decode bounded image pixels, cap public native responses before usage stamping, and preserve oversized malformed text status precedence. Assisted-by: OpenAI:API-assistant * fix(router): classify images before media fetching Preserve ordered structured probes for native decisions. Defer OpenAI media preparation until routing selects the served model, so rejected decision URLs cannot trigger downloads before shared validation. Guard direct image collection with context-aware shared admission. Keep text classifiers and embedding caches from discarding image input. Retain fail-closed classifier configuration and runtime fallback policy. Add middleware, typed-content, admission, cancellation and cache tests. Assisted-by: OpenAI:API-assistant * fix(router): bound extraction before serialization Check probe budgets before copying text or marshaling message state. Count JSON escaping so oversized internal inputs fail before allocation. Preserve typed Anthropic blocks through selected-model conversion and fallback. Keep retry coverage in Ginkgo without global test registration. Assisted-by: OpenAI * fix(router): bound supported probe serialization Arbitrary structs can bypass the probe budget through pointer marshalers, string tags, and promoted fields. Accept concrete chat schema types and plain JSON values instead of emulating arbitrary struct serialization. Budget escaped direct prompts before marshaling so raw length cannot hide serialized expansion. Preserve runtime fallback and reject oversized input before invoking the decision runner. Add Ginkgo allocation, boundary, and marshaler invocation regressions. Six-package tests, three-package race tests, and full-T2 delta lint pass. Assisted-by: OpenAI:GPT-5 golangci-lint * feat(decisions): enable bounded OpenJev images Validate native decision images before permissive media parsing and pixel allocation. Require both decision image support and a vision projector; missing or audio-only projectors cannot silently become text decisions. Pin the OpenJev Q8 projector and document its license and disk footprint. Add native safety tests, canonical limit parity, gallery and load-option checks, and a reproducible CPU direct-RPC contrasting-image smoke. Assisted-by: OpenAI:GPT-5 * fix(decisions): reject incomplete image streams stb accepts corrupt PNG Adler checksums and truncated JPEG scans. Use bounded zlib validation and strict libjpeg decoding before parsing. Keep dimension and aggregate pixel checks ahead of decoder allocations. Wire decoder dependencies into native builds and runtime packaging. Add regressions for appended EOI and embedded marker bypasses. Assisted-by: OpenAI:GPT-5 * fix(ci): gate native decision image validation Run the decoder security tests outside the stdlib-only native suite. Fetch vendor headers at the backend pin and provision decoder dependencies. Gate Go limit parity and production CMake wiring without model downloads. Assisted-by: OpenAI:GPT-5 * test(decisions): cover multimodal public API paths Exercise shared image contracts through the registered HTTP routes and external mock backend. Add opt-in cached gallery installation and real OpenJev image decisions through SystemOne and both routing APIs. Assisted-by: Codex:gpt-5 * test(decisions): assert isolation and cache bypass Observe external RPC calls and compare complete classifier history. Winner-only and cache-miss checks could hide dropped history or cache use. Give real inference its own application and model directory so shared backend mappings and loaded processes cannot affect mixed suite order. Assisted-by: OpenAI:ChatGPT * test(decisions): isolate fixture globals Disable optional global services in the isolated HTTP fixture and register cleanup before setup assertions. Verify meter provider identity survives fixture creation and destruction. Snapshot observed usage before assertions so failures cannot retain the mutex. Require a successful usage stamp before checking error responses. Assisted-by: Codex:gpt-5 golangci-lint * fix(application): honor optional telemetry controls Skip failover gauge registration when metrics are disabled. Register against the application meter rather than looking up the global provider. Allow embedders to retain the bounded routing log without billing stats. Keep the existing default when stats are disabled. The isolated HTTP fixture uses this option without losing its native router assertions. Assisted-by: Codex:gpt-5 golangci-lint --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
e4fa051ee3 |
refactor(distributed): put the NATS-only paths behind interfaces (#12395)
* feat(messaging): add shared subject rules Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(messaging): cover BroadcastRoots, ControlRoots and SubjectRoot Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add Broadcaster and enforce subject rules in every carrier Broadcaster is the fan-out half of MessagingClient. The NATS client and the in-memory FakeBus now refuse a subject outside the served roots and any wildcard other than a whole single token, and FakeBus shares MatchSubject instead of its own copy. FakeBus Unsubscribe now removes its own subscription instead of the first one with the same subject. A shared conformance suite in messagingtest runs against both carriers. The distributed e2e specs that used invented test.* subjects, and the one that subscribed with a > filter, now use subjects from subjects.go. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: depend on Broadcaster where only publish and subscribe are used Narrowed to messaging.Broadcaster: nodes/staging_progress.go, nodes/install_progress_publisher.go, galleryop/operation.go, galleryop/service.go, agentpool/user_services.go, agentpool/agent_jobs.go, openresponses/store.go, openresponses/sync.go, syncstate/syncstate.go, finetune/service.go, quantization/service.go and failover/distsync/distsync.go. SubscribeJSON now takes a Broadcaster because it only calls Subscribe, which lets the narrowed consumers use it. Stayed wide: worker/supervisor.go, because its client field also serves the SubscribeReply handlers in worker/lifecycle.go. The request/reply, queue and wiring files (nodes/unloader.go, nodes/file_stager_s3.go, jobs/dispatcher.go, agents/dispatcher.go, agents/events.go, worker/file_staging.go, cli/agent_worker.go, http/app.go) are unchanged by design. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): name the no-route condition and confine the carrier error Consumers matched nats.ErrNoResponders, which names an absence, to demote a node. They now match ErrNoRoute, the control path maps the carrier's failure onto it, and timeouts and worker refusals are pinned as not being no-route. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(nodes): state which FileStager implementations return ErrNoRoute Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): build backend clients through one node-aware seam Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe the distributed transport seams Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct comments that overclaim after the seams refactor Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(agent-worker): refuse an unserved LOCALAI_AGENT_SUBJECT at startup The messaging client now refuses a subject whose root no carrier serves. An agent worker started with a custom LOCALAI_AGENT_SUBJECT such as tenant-a.agent.execute used to start and then wait on a subject the frontend never publishes to. After the subject rules landed it exited at subscribe time with an error that did not name the setting. Behaviour change: the worker now checks LOCALAI_AGENT_SUBJECT before it registers or connects, and exits with an error that names the variable and says to use a served subject under the agent root, for example agent.execute. The served roots are not widened: a custom root was never delivered by the frontend, and a wider set would reopen the drift the subject rules exist to close. The flag help and the agent worker docs state the constraint. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(nodes): pin the reactions to ErrNoRoute Three callers react to ErrNoRoute and had no spec: the reconciler's upgrade drain falls back to the legacy forced install, the reconciler marks the node unhealthy when a pending op has no route, and the backend-op fan-out marks the node unhealthy. Each spec drives the real caller with a scripted no-responders reply and reads the result from the registry or the recorded requests. A fourth spec pins the other side: a pending op that times out leaves the node healthy and only counts the attempt, so mapping timeouts onto ErrNoRoute would fail here. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(messaging): pin client subject checks and fail the carrier suite in CI Add specs that call Publish, Request, Subscribe, QueueSubscribe, SubscribeReply and QueueSubscribeReply on a client with no connection. Each call must return ErrUnservedSubject for bogus.thing and ErrUnsupportedWildcard for jobs.>. This proves that the subject check runs before the connection is used, and needs no server. The NATS conformance suite is the only check that runs the subject rules against a real carrier. Before this change it skipped without output when Docker was missing. Now it fails when CI is set, so a Linux runner without Docker cannot hide it. It still skips on local runs and on macOS CI, which has no Docker. Add SubjectNodeBackendInstallProgress to the list of constructors that must build served subjects, and ask contributors to extend the list. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: state what ErrNoRoute may change, and group the distributed guides The seams note said MarkUnhealthy was the only state change allowed on ErrNoRoute. A pending backend op still records the failed attempt, counts toward the reconciler's retry limit and is dead-lettered after the maximum attempts. The note now says that MarkUnhealthy is the only change to the node's own state, and that the per-op accounting is not a verdict about the node. The note also documents that the NATS conformance run fails under CI when Docker is missing. The distributed-seams row moves next to the distributed-state row in the topics table. The liveness ping spec header now says no route is a reason to skip the worker, not proof that the worker is gone. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): give the backend client factory the node id Mechanical: the method gains a nodeID parameter and the eight test fakes are updated. No behaviour change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): drop the optional node-aware factory The node id is now in the main method, so the optional interface and its helper had no behaviour of their own. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): dial backend probes through the client factory Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): dial workers' file servers through a per-node dialer Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(http): proxy backend logs through the per-node worker dialer The admin backend-logs proxy (list, lines and the WebSocket stream) now reaches a worker through the same per-node dialer as the HTTP file stager, so every frontend-to-worker dial goes through one seam. The shared direct dialer keeps alive for 15s where the proxy used 30s. Harmless for requests bounded at 15s. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(http): keep the backend-logs proxy independent of the admin connection The proxy request had no context before the dialer change and is bounded only by its 15s timeout. Keep it that way. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: move the worker control payloads to workerctl Mechanical move of the request and reply structs, the install progress event and the file payloads out of messaging. The verbs no longer belong to one carrier. No alias is left behind. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): serve the lifecycle verbs through a controlServer The worker registers one handler per verb and a NATS server maps each verb to its subject. Registration errors now name the verb. node.stop is served with SubscribeReply, which is identical on the wire because the handler never replies. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): report install progress through the control sink Install and upgrade now emit download progress through the sink the control server hands them. The debounce and the terminal flush stay in the handler path, built over that sink by the new nodes.NewDebouncedInstallProgressSink, which replaces NewDebouncedInstallProgressPublisher. The subject and payload on the wire are unchanged. The supervisor no longer holds the bus, and installFn and upgradeFn let specs drive both verbs without a gallery. The malformed-request log lines are restored for install, upgrade, backend.delete, model.unload, model.stop and model.delete, with the reply bytes unchanged. The signal adapter is renamed noReply, which also lets worker.go import os/signal without an alias again. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): serve the file-staging verbs through a controlServer An empty list-dir answer is now {} rather than {"files":null}, because the typed reply omits an empty Files slice. The frontend decodes both to a nil slice in nodes/file_stager_s3.go. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add WorkQueue and the NATS producer This is the producer side of the competing-consumer seam. The work kinds map one to one to today's subjects and queue groups: task to jobs.new and mcp-ci to jobs.mcp-ci.new (both in group workers), agent-run to agent.execute (group agent-workers). Enqueue publishes the payload as Publish does today, with one JSON marshal. FakeBus now records queue groups and keeps reply handlers so later specs can pin and drive them. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add the NATS WorkConsumer An in-flight limit of one runs the handler inline on the delivery goroutine, as the MCP CI consumer does today. Any other limit spawns per delivery, as the agent consumer does. Queue groups are unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: publish queued work through WorkQueue The job dispatcher, the agent pool and the agent scheduler enqueue through messaging.WorkQueue; the NATS implementation publishes to the same subjects as before. DistributedServices builds the queue next to the NATS client and hands it to the dispatcher and the agent pool, whose distributed mode switch now reads a non-nil WorkQueue. The unused AgentPoolService.SetNATSClient is removed. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: consume queued work through WorkConsumer The agent dispatcher and the MCP CI consumer register through messaging.WorkConsumer. The NATS implementation keeps the inline one-at-a-time model for MCP CI and the per-delivery model for agent runs. handleMCPCIJob reports on the events publisher the carrier hands it instead of a captured client. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: delete the consumers nothing in production reached jobs.new has a producer and no production consumer, and the agent dispatcher's Dispatch was only called from tests. Publishing jobs.new is unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(mcp): send MCP requests to agent workers through AgentControl Timeouts still honour only the deadline, not cancellation, exactly as today. The NATS no-responders error maps to ErrNoRoute and a timeout does not. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(agent-worker): serve MCP requests and backend.stop through agentRPCServer The agent worker's MCP tool and discovery reply subscriptions and its backend stop listener move behind an unexported agentRPCServer interface, served on NATS by nodes.NATSAgentRPCServer. The handlers become typed mcp.ToolHandler and mcp.DiscoveryHandler values that answer every failure with a reply carrying Error. Queue group (agent-workers), inline execution on the delivery goroutine, the background handler context, the unmarshal error reply texts and the reply-less backend stop subscription are unchanged. The backend stop handler takes the decoded backend name, so it can still close that backend's MCP sessions. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(messaging): remove helpers that only tests used BroadcastRoots, ControlRoots and SubjectRoot had no production caller. The roots spec now asserts every served root through ValidateSubject instead. MatchSubject moves back into the test support package, the only place that used it, with its table. NATSAgentRPCServer drops the subscription list it stored and never read, and NewNATSAgentRPCServer gets a doc comment. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(mcp): round trip the agent RPC server over a real NATS server One spec sends a tool request and a discovery request through NATSAgentControl to NATSAgentRPCServer and checks that the handlers see the decoded requests and the replies come back. It also puts an undecodable body on the tool subject and checks the server answers with an unmarshal error instead of leaving the requester to time out. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe the distributed transport seams The developer note now lists the final seams: fan-out, queues, both halves of the control verbs and of agent RPC, and the dial. It records the open items a second carrier has to handle. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test: pin the in-flight limit each queue consumer asks for The work queue specs pin what Consume does for a given limit, but nothing pinned which limit each production consumer passes. Changing the agent worker's MCP CI limit from 1 to 0 would have let MCP CI jobs run concurrently on each worker with every test green. Move the MCP CI Consume call into startMCPCIConsumer with the same wiring and pin that it asks for (WorkMCPCI, 1). Pin that NATSDispatcher.Start asks for (WorkAgentRun, maxConcurrent) for several limits. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: remove helpers the branch left without a caller SubjectJobCancelWildcard lost its last subscriber when the frontend stopped listening on jobs.*.cancel; the NATS permissions and conformance suite spell the subject out, so nothing reads the constant. decodeBackendStopRequest returned a stopAll flag that production dropped and only a test read. decodeBackendStop is now the single decoder with the same semantics: an empty body is stop-all, an empty Backend is stop-all, malformed JSON is an error. stopBackends still derives stop-all from Backend, so no reply changes. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(messaging): keep an explicitly empty agent queue a plain subscription Before the work queue seam the agent worker passed LOCALAI_AGENT_QUEUE straight to QueueSubscribe, so an explicitly empty value made a plain subscription and every agent worker ran every agent run. WithAgentRunRoute replaced an empty queue with agent-workers, which silently changed that. Keep the queue as given once the option is applied. An empty subject still falls back to agent.execute, since it never had a meaning of its own. The flag default stays agent-workers, so only an explicitly empty value reaches this. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct comments and record the PR B notes Fix the recordingFactory comment (it also records the parallel flag), document that a negative maxInFlight is unbounded and that Unsubscribe from a handler deadlocks, and say a permanently undecodable payload returns nil. Record controlHandler's undecodable return as a kept exception, and add the second carrier notes to the developer note: the reconciler has no ClientFactory option, the logs proxy honours HTTP_PROXY, verbs one carrier serves need an opt-out, terminal replies come from the result event, and agent runs publish through the NATS-bound EventBridge, which is not an additive change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
70ce62901f |
refactor: name the capability decisions instead of systemone
The usecase describes what a model can do, and the category is the Decisions API. SystemOne stays as the wire contract: the /v1/systemone routes, the Score RPC question_type and the swagger tag are unchanged. The usecase, flag, auth feature, UI label, gallery tags and docs page are now decisions. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
84c83a70dd |
feat(config): add systemone usecase for decision models
Explicit-only, reserving usecase like score and token_classify: a declared list is authoritative and the heuristic never guesses it. vllm-cpp now lists systemone and vision as possible usecases. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
2fa36e7147 |
feat(parakeet-cpp): speaker diarization, sound detection and live scene events (#12335)
* feat(parakeet-cpp): load diarization and CED models and companions
Repin PARAKEET_VERSION to parakeet.cpp PR #75's head, which adds
parakeet_capi_model_kind (ABI v8). Bind the new diarization, sound
event and combined scene stream C symbols through the same
purego.Dlsym probe pattern already used for the batched JSON entry
point, so the backend still loads against an older libparakeet.so.
Load now classifies the loaded GGUF by role (ASR, diarization or
sound) via parakeet_capi_model_kind and can load up to two companion
models from Options[] (asr_model:, diarization_model:, sound_model:,
paths resolved against opts.ModelPath), verifying each companion's
kind and freeing every context opened so far on any failure. Free
releases the primary and every companion. AudioTranscription now
names the loaded role when it is not ASR instead of a generic model
not loaded error. The dynamic batcher starts only when an ASR context
ends up loaded, primary or companion.
This is groundwork only: the Diarize and SoundDetection RPCs and the
live scene stream that actually use these new roles land in later
commits.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): reset role fields on a failed companion load
loadRoles' freeLoaded only released the C contexts it had opened; it
left ctxPtr/diarCtx/tagCtx and companions pointing at those now-freed
contexts, so a later Free() on the same instance would double-free.
Zero all four alongside the CppFree calls.
Also route AudioTranscriptionStream and AudioTranscriptionLive through
notASRError when ctxPtr is unset but a diarization or sound model is
loaded, matching AudioTranscription: both used to return the generic
model-not-loaded error instead of naming the loaded role.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(parakeet-cpp): add speaker diarization
Implement the Diarize RPC for the parakeet-cpp Go backend, wired to
Nemotron-3-Diarization through libparakeet.so's diarization C-API.
Plain diarization uses parakeet_capi_diarize_pcm; when include_text is
set and an ASR companion is loaded, parakeet_capi_transcribe_and_
diarize_json fills each segment's text instead. Speaker labels are the
decimal index, or "unknown" for -1 (no diarized speaker overlaps).
min_duration_off merges same-speaker segments across a short gap
before min_duration_on drops the segments still too short, then ids
are renumbered. num_speakers/min_speakers/max_speakers/clustering_
threshold have no Sortformer equivalent and are logged at debug
instead of rejected.
Verified against the real Nemotron-3-Diarization + parakeet-tdt_ctc-
110m checkpoints on the two_speakers.wav fixture: correct A-B-A-B
speaker segmentation and matching speaker-attributed transcripts.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(parakeet-cpp): add sound event detection
Wire the SoundDetection RPC to the CED tagger context (p.tagCtx)
loaded by Task 1's role classification. It runs the whole clip
through a one-shot parakeet_capi_sound_stream_* session (window
10s, hop 10s, top_k set to the tagger's class count so every
drained window carries a full score list), averages each class's
score across the drained windows, sorts descending, then applies
the request's threshold and top_k (0 keeps every class).
No tagCtx returns FailedPrecondition; a libparakeet.so missing the
sound_stream symbols returns Unimplemented. Every C call runs under
engineMu, and the stream is always freed, even when a feed or drain
call fails partway through.
Verified against a real ced-tiny-q8_0.gguf on the rooster.wav demo
clip: "Chicken, rooster" tops the list at score 0.91.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): cancel sound detection mid-feed, shrink the lock
SoundDetection now checks ctx before each 10 s feed slice (mirroring
driver.go's feedSlices) and returns Canceled if the caller gave up,
so a long clip can be interrupted instead of feeding to completion
regardless. The stream is still freed on every path, cancellation
included.
Also narrow engineMu to the C calls: the drained JSON document is
now decoded after the lock is released, splitting soundStreamScores
into a locked soundStreamDrain (opts, begin, feed, drain, free) and
an unlocked json.Unmarshal.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(parakeet-cpp): stream speaker and sound events during live transcription
Add two additive proto fields, LiveSpeakerSegment and LiveSoundEvent,
repeated on TranscriptLiveResponse. When a diarization or sound
companion model is loaded, AudioTranscriptionLive now runs a no-ASR
scene stream (parakeet_capi_scene_stream_begin) beside the ASR
streaming session, feeding it the same PCM slices and forwarding any
closed speaker or sound events alongside the matching ASR delta, or
on their own when a slice has no ASR output.
The scene stream is freed and reopened on a mid-stream Config reset,
flushed with is_last before the closing FinalResult, and degrades
gracefully (a warning, not an error) when begin or a later feed call
fails, so live transcription keeps working ASR-only. Existing live
behavior is unchanged when no companion is configured, and no scene
C call is made in that case.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): keep scene events off the ASR critical path in live
Emit each slice's ASR result right after the ASR feed, before the
scene feed for that slice runs, so a companion diarization/sound
model never adds scene compute latency in front of the delta or
<EOU> that drives realtime turn detection. Closed speakers/sounds go
out afterward as their own response, so a slice with both now
produces two responses, ASR first. The live feed log line now
reports ASR and scene wall time separately.
Re-check the diarization/sound contexts a scene stream was begun
with against the live contexts before every feed, under the same
lock: Free() can race between an ASR feed and the matching scene
feed and free the model the stream borrows. A mismatch now returns
without touching the C side. Freeing the stream itself stays
unconditional; the scene stream's destructor only releases its own
buffers and never touches the borrowed contexts.
Also recover a panicking stub inside the live test goroutine instead
of crashing the test binary, and reset the live decode-lag tracker on
a mid-stream config reset, matching what its own comment already
promised.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): surface live speaker and sound events
Carry the backend's closed speaker segments and sound events
(TranscriptLiveResponse fields 7/8) through LiveTranscriptionEvent
as LiveSpeakerSegment/LiveSoundEvent (nanoseconds mapped to
seconds), and forward them from the semantic_vad live path.
Each speaker segment emits
conversation.item.input_audio_transcription.segment with speaker,
start, end and empty text under the turn's item id. Each sound
event emits conversation.item.sound_detection with one tag
(label, score = peak, index) and the event's new optional
start/end seconds fields, omitted when unset so the existing
unary/windowed sound-detection path is unaffected.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): keep start/end on a zero-second transcription segment
ConversationItemInputAudioTranscriptionSegmentEvent.Start/End used
omitempty, so a speaker segment starting at 0.0s dropped its
"start" key. Nothing emitted this event before the live scene-event
path, so drop omitempty: the segment always carries real times.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(gallery): add parakeet-cpp diarization, CED and realtime scene models
Add gallery entries for the new parakeet-cpp capabilities: standalone
Nemotron-3-Diarization, the same paired with the Parakeet TDT+CTC
110M ASR model for speaker-attributed text, CED-Tiny and CED-Base
sound classifiers, and a realtime scene bundle combining the
streaming EOU ASR model with diarization and sound companions.
SHA256 taken from the Hub API; licenses from each model card
(openmdw-1.1 for Nemotron-3-Diarization, apache-2.0 for CED,
cc-by-4.0 for the Parakeet ASR models).
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: document parakeet-cpp diarization, sound detection and live scene events
Cover the new parakeet-cpp capabilities across the feature pages:
Nemotron-3-Diarization as a diarization backend (with and without
speaker text, the ignored speaker-count hints, the Sortformer
voice-like-sound quirk), CED as a sound classification backend, the
asr_model/diarization_model/sound_model/diarization_latency companion
options, and the realtime live speaker/sound events (event shapes,
the speech-turn-only limitation, and using this or
pipeline.sound_detection but not both).
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): correct the realtime-scene license and wording nits
parakeet-cpp-realtime-scene mistakenly copied cc-by-4.0 from the
existing realtime_eou_120m-v1 entry; the model card lists the NVIDIA
open model license instead. Switch to the gallery's usual spelling
for that license and keep the diarization/CED licenses called out in
the description.
Also: audio-diarization.md now says getting per-segment text needs
both an asr_model companion and include_text=true on the request, and
audio-to-text.md's option table reads "Use on" (a pairing the loader
does not enforce) instead of "Allowed on".
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): reject a companion role that duplicates the primary's
loadRoles let a companion option (asr_model:/diarization_model:/
sound_model:) assign into a role field the primary already occupied,
for example asr_model: on an already-ASR primary. The companion's
context silently overwrote ctxPtr/diarCtx/tagCtx, and Free() only
walks those three fields, so the original primary context was never
freed again.
Reject a companion whose role the primary already holds before its
GGUF is even loaded, freeing everything loadRoles opened so far, the
same way a wrong-kind companion is already rejected.
Also warn, rather than silently fall through, when
parakeet_capi_model_kind reports PARAKEET_MODEL_KIND_NONE for a
successfully loaded primary; the primary is still treated as ASR,
matching today's behavior.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): cap live scene sound score retention
sceneBegin started the live diarization/sound companion stream with
the C API's default sound options, whose top_k keeps 5 scores per
window forever until drained. The live scene path never drains sound
scores (only the offline SoundDetection RPC does, with its own fresh
stream), so this window queue on the C side grew for the whole
session's lifetime.
Set opts.Sound.TopK = 0 before starting the scene stream: this
disables score retention while leaving sound event detection (onset/
offset), which the live path actually consumes, unaffected.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): merge diarization segments per speaker, harden Diarize
mergeCloseSegments only compared neighbors in the single start-sorted
segment list, so two same-speaker segments never merged once another
speaker's turn fell between them (A, B, A): the short B segment broke
the adjacency the merge relied on. Group segments by speaker first,
merge within each speaker's own start-ordered run, then re-sort the
result by start so interleaved speakers come back out in timeline
order.
Also harden Diarize's entry points the same way streamFeedDoc/
sceneFeed already are: diarizeCall re-checks p.diarCtx (and, on the
include_text path, p.ctxPtr) under engineMu right before the C call,
so a Free() racing between Diarize's own checks and the lock can no
longer reach the C side with a freed context. When the include_text
call returns NULL, last_error is now read from both contexts and
whichever came back non-empty is reported, since either side of the
pairing can be the one that failed. A WAV decode failure is reported
as InvalidArgument instead of an unwrapped/untyped error.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): harden SoundDetection's engine checks
soundStreamDrain ran every C call under engineMu but never re-checked
p.tagCtx there, so a Free() racing between SoundDetection's own
tagCtx==0 check and this lock could still reach the C side with a
freed context. Re-check p.tagCtx under the lock and return
ModelNotLoaded when it was cleared, mirroring diarizeCall's own
re-check. A WAV decode failure is now reported as InvalidArgument
instead of an unwrapped/untyped error.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(parakeet-cpp): cover a mid-session scene feed failure
feedSlicesScene already degrades gracefully when a scene feed call
fails mid-session: it frees the broken stream and carries the ASR-only
session forward. Add a spec covering that path end to end: the scene
stream is freed exactly once, later audio slices still produce ASR
responses, and no speaker/sound events appear before or after the
failure.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: fix the parakeet-cpp companion role table and realtime scene docs
audio-to-text.md's companion option table read "Use on" with a note
that the loader did not enforce the pairing; it now rejects a
companion whose role duplicates the primary's, so restore the
"Allowed on" wording and describe the real enforcement.
openai-realtime.md's live speaker/sound section claimed a mid-stream
session.update resets the companion stream and that it flushes on
session close; neither happens, since the realtime core opens one
live stream (and so one scene stream) per speech turn and closes it
at that turn's commit, with no mid-stream Config in between. Document
that lifecycle instead, state precisely that start/end are seconds
from the start of the turn's own audio, and note that the diarization
model starts a fresh session every turn, so a speaker index is only
meaningful within one turn. The example sound tag ("Rooster", index
17) did not match any real CED label; index 17 in ced-tiny-q8_0.gguf
is "Baby laughter". Replaced with "Chicken, rooster" at its real
index, 99.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): use CED's real index for Chicken, rooster
The scene feed comment and the live test's canned document gave
"Chicken, rooster" index 365. In CED's AudioSet label list it is 99,
which is also what the realtime docs show.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): call the test event accessor
The scene-event tests range over a method instead of its returned slice.
Call the synchronized accessor so the OpenAI test package compiles.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(parakeet-cpp): pin parakeet.cpp master with sound events
mudler/parakeet.cpp#75 (sound events, scene stream, model kinds) and
#74 (the missing <algorithm> include that broke the image builds) are
on master now. Pin 6dea76a instead of the #75 PR head, and update the
header comment the bump bot reads.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(parakeet-cpp): pin parakeet.cpp with ced.cpp on main
parakeet.cpp #76 moved its ced.cpp submodule from the head of
localai-org/ced.cpp#3 (a branch-only commit) to ced.cpp main, where
#3 landed with an identical tree. Pin 623a968 so the image builds no
longer depend on that branch.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(transcription): carry speaker labels on words and streamed segments
A diarizing backend could label transcript segments, but two paths
dropped the label: TranscriptWord had no speaker field, so live
transcription words and word-level timestamps could not carry one, and
the stream=true transcript.text.done event left the speaker out of
its segments.
TranscriptWord gains an optional speaker (proto field 4, additive).
It flows through the live event and result mapping, the JSON word
output of the endpoint and the CLI, and transcript.text.done now
includes a segment's speaker when there is one. Empty labels are
omitted, so responses without diarization are unchanged.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
(cherry picked from commit
|
||
|
Ettore Di Giacinto
|
dae9a431e8 |
Merge remote-tracking branch 'origin/master' into feat/failover-chains
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5-5 [Claude Code] |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
490b952d06 |
feat(gallery): publish signed OCI fallbacks (#12182)
* feat(gallery): publish signed OCI fallbacks Publish both official gallery indexes with their local base configs so an outage of the HTTP and GitHub sources can fall back to Quay. Keep artifact signing policies separate from backend image policies, and expose each moving gallery tag only after its digest is signed. Assisted-by: Codex:gpt-6 * fix(gallery): confine packaged files to selected roots Use directory-scoped file access to reject symlink escapes during gallery packaging. Create private bundle files for the publishing runner. Assisted-by: Codex:GPT-6 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
27ceda46b0 |
refactor: name the realtime pipeline stages with constants
Review asked for constants instead of the stage literals ("vad",
"transcription", "llm", "tts", "sound_detection") passed to resolveStage,
stageCall and isChainStage, so the uses can be cross-checked. Add
PipelineStage* constants next to the Pipeline type in core/config: the
names match its yaml keys, and core/backend (preload roles) and the openai
realtime endpoint both need them.
Use them in realtime_model.go (stage routing and preload roles),
realtime.go (the voice_recognition preload role) and core/backend
preload.go. model_failover events take their stage from the stageChains
keys, so they now carry the constants too. The failover tests use the
constants for inputs and keep literal wire values in their event
assertions.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
8db6c15fe0 |
refactor: name the proxy backends with constants
Review asked for constants instead of the "cloud-proxy" and "localai-proxy" literals. Add CloudProxyBackend and LocalAIProxyBackend next to the other backend-name constants in pkg/model (WhisperBackend, TransformersBackend, ...), which core/config already imports, and use them in every production check: the proxy options builder, IsRemoteProxy, IsCloudProxyBackendPassthrough, the PII defaults, the localai-proxy backend hook and loader warning, and the PII middleware metadata in the routes and the in-process MCP client. The proxy options builder now calls IsRemoteProxy() instead of repeating the two-backend check, so the set of proxy backends is defined once. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
bdd4a73d05 |
feat(ui): edit failover chain targets with a dedicated field
Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
6df1767133 |
fix(localai-proxy): keep rerank, stream errors and chat intact through the proxy
Rerank no longer sends top_n 0, which the upstream rejects. A mid-stream upstream error frame now fails the call instead of ending it as a short success. Temperature 0 is forwarded. An upstream 429 becomes ResourceExhausted, which failover skips like Unimplemented. A localai-proxy config sends its own name upstream when upstream_model is unset, and a chat proxy defaults to the tokenizer template so chat reaches the upstream as messages. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
509cca35e0 |
feat(grpc): serve Rerank from Go backends and skip Unimplemented targets
Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-bot
|
e62854c340 |
fix(failover): pass credential lookup from CLI
Pass the API key environment lookup through ApplicationConfig to satisfy core configuration lint. Keep credential resolution dynamic and exclude the callback from serialization. Handle the five close results reported by errcheck. Assisted-by: Codex:gpt-6 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
f7f12f8203 |
fix(failover): warn about warm on a remote target, align the spec
The spec promised a load-time warning when a chain marks a remote target warm, where the flag does nothing; the loader now logs it. The remote-backend test moves into ModelConfig.IsRemoteProxy so the loader and the failover manager agree on what is remote. The spec now says what ships: a load blocked by pinned warm targets proceeds over the limit after eviction retries, without an error that names them. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
b0a4c132ca |
fix(failover): treat a set-but-empty api_key_env as unset in ResolveAPIKey
Matches cloud-proxy's resolveAPIKey exactly: os.Getenv + empty check rather than os.LookupEnv, so a variable that's set but empty errors instead of probing unauthenticated. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a7dc8bf42c |
feat(failover): probe remote targets over HTTP and local ones over gRPC
Remote liveness uses /v1/models, which every OpenAI-compatible upstream serves. Recovery sends one minimal request for the target's usecase. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
f7c0bb5f4b |
feat(config): validate failover chain targets across configs
Reject chains whose targets are missing or are chains, at load and on create or edit, and warn when the targets share no usecase. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
63221ee779 |
feat(config): add failover chain block to model configs
A chain is a model config with an ordered list of target models, probe, trip and recovery settings. Like an alias it has no backend. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
42c58a5838 |
feat(vllm-cpp): unify decision pipeline through Score RPC with vllm_decide ABI v29 (#12247)
* feat(vllm-cpp): unify decision pipeline through Score RPC with vllm_decide ABI v29 Replace the model-specific SystemOne gRPC approach with a generic Score RPC extension. The pre-existing Score RPC (previously unused by any backend) now carries question_type and response_json fields: - question_type="systemone" routes kev/laya decision-pipeline requests through the unified vllm_decide C ABI (v29), returning the full response JSON in response_json. - question_type empty routes cua-s1-forms candidate scoring through the same vllm_decide ABI, returning CandidateScore probabilities. The vllm-cpp backend's Score() method calls vllm_decide and dispatches by architecture internally. The /v1/systemone HTTP endpoint checks whether the model's backend supports Score; if so, it forwards the raw request JSON and returns the backend response as-is. Other backends fall through to the existing NER-based path. This mirrors the vllm.cpp C ABI refactor (PR #3301) that replaced vllm_systemone + vllm_score with a single vllm_decide function. The purego bindings bump abiVersion from 27 to 29 and resolve vllm_decide and vllm_decide_free symbols. Also fixes validModelPath to accept cua-s1-forms.json and rl_agent_config.json alongside config.json, matching the engine's model_loader.cpp config-filename ordering. AI-Assisted: true Assisted-by: Maki:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: ⬆️ Update mudler/vllm.cpp to e28ec46c6 (fix macOS -Werror build) Bumps vllm.cpp to e28ec46c6 which fixes a -Wnull-conversion error in qwen3_5.cpp:12483 that broke the macOS Metal CI build under -Werror. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
be0671c635 |
feat(gallery): optionally pin the signing certificate's source repository (#12235)
* feat(cosignverify): optionally pin the certificate's source repository Assisted-by: Claude:claude-opus-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): source_repository in the verification policy Assisted-by: Claude:claude-opus-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(gallery): when source_repository is checked; test the issuer Assisted-by: Claude:claude-opus-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
21c5495a99 |
feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify (#12140)
* feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify Wire the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner, ABI v27) so LocalAI can serve zero-shot named entity recognition through the existing TokenClassify gRPC method. backend.go: TokenClassify method on *VllmCpp calls vllm_gliner_ner with the text and labels, copies the C-owned entity array into protobuf TokenClassifyEntity messages, and frees the result. govllmcpp.go: cNerEntity and cNerResult Go POD mirrors matching the C structs; vllmGlinerNer and vllmNerResultFree purego bindings; abiVersion bumped 26 -> 27. options.go: ner_labels, ner_threshold, ner_max_width parsed from engine_args. pkg/grpc: ClassifyModel interface and TokenClassify server handler (follows the Embedding locking pattern). core/config: vllm-cpp backend declares MethodTokenClassify and UsecaseTokenClassify. docs/content/features/vllm-cpp.md: NER section documenting the engine_args keys and the host-forward contract. Assisted-by: MAKI:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): correct NER pointer lint directive Use the govet directive for the C-owned NER array, matching the other purego pointer conversions. The array remains valid until its deferred free; the misspelled directive caused CI to flag this conversion. Assisted-by: Codex:gpt-6 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vllm-cpp): add kev-compatible SystemOne API endpoints Add POST /v1/systemone, /v1/systemone/permute, and /v1/systemone/separate to LocalAI, mirroring the kev project's structured-extraction API. Each endpoint runs zero-shot NER over the rendered state text and builds kev-compatible answers for three question types: noul (binary entity presence), choice (pick one option), and score (pick one level). The TokenClassifyRequest proto gains a `repeated string labels` field so each question can supply its own labels at inference time, and TokenClassifier gains TokenClassifyWithLabels for per-call label selection. The vllm-cpp backend uses request labels when non-empty, falling back to configured ner_labels then the built-in defaults. Helpers (renderState, softmax, choiceConfidence, scoreConfidence, r2) are ported from kev/api.py and mirrored in vllm.cpp's api_server.cpp so both servers produce the same answer shape. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): suppress gosec G404 on seeded permutation RNG The SystemOne permute endpoint uses math/rand with a caller-supplied seed for reproducible option permutations, matching kev's random.seed. gosec flags this as G404 (weak RNG). Add #nosec with a comment naming the intent: this is reproducibility, not cryptography. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(vllm-cpp): bump vllm.cpp pin to GLiNER2.5 merge commit Advance VLLM_CPP_VERSION from f3cd97e to 5058268d, the commit that landed GLiNER2.5 zero-shot NER support (PR #3224) in vllm.cpp. This brings the DeBERTa v2 encoder, GLiNER2 boundary head, C ABI NER functions, and server endpoints into the LocalAI vllm-cpp backend. The ABI version (27) and Go struct mirrors already match. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): use instruction text as NER label in SystemOne handler The SystemOne handler was passing question IDs as NER labels for noul questions and bare key names for choice questions, so the model never matched any entities. Port the label mapping from vllm.cpp's ParseSystemOneBody: - noul: use the rendered instructions field (with instr alias) as the NER label, not the question ID - choice: use optionText(name, desc) — "name: description" or "name" when the description is null/empty — not the bare key - score: already correct (rendered criteria text) - permute: shuffle indices and build parallel key/label arrays so the NER call uses the optionText labels while the response is keyed by the original option names Also add the instructions field to the SystemOneQuestion schema struct (accepted alongside the instr backward-compat alias). Verified end-to-end against the real GLiNER2.5 model: noul questions now find "Apple Inc. is" (organization, 0.999) and "Tim Cook is" (person, 0.852) where they previously returned zero entities. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
2facfc0d88 |
feat: Add kimodo.cpp and 3D animation API/UI (#12095)
* fix(vulkan): preserve host ICD discovery for packaged backends Add bundled Mesa manifests through VK_ADD_DRIVER_FILES instead of replacing the system driver list. Merge inherited and model-specific additive paths while preserving explicit operator overrides, with regression coverage. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add Kimodo CPU and Vulkan animation backend Introduce a distinct animation capability and model-described 3D operations, with a typed /3d/animate API, RPC transport, distributed media staging, permissions, and tracing. Add a persistent kimodo.cpp adapter, skeleton GLB export, CPU/Vulkan packages, model and backend galleries, importer support, CI builds, and documentation. Adapt Studio inputs to each model and provide real-time skeleton playback, seeking, and history. Cover backend validation, packaging, API behavior, importer inventories, distributed staging, and Studio workflows. Validate real-model CPU/Vulkan generation and deploy the integration to the local QA instance. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(kimodocpp): adopt monolithic encoders and resident inference Update upstream for resident weights, packed execution paths, and cached motion graphs. Default to all 32 text layers while retaining configurable streaming and legacy bundle support. Use monolithic Q8_0 encoders by default and offer all six published quantizations through the gallery and importer. Refresh pinned hashes, tests, and documentation; remove the obsolete thread patch and ensure cached source checkouts follow the upstream pin. Validated CPU and Vulkan generation, lower-bit streaming, gallery/importer suites, packaging, lint, and cold/warm Studio generation on localai-dev. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Adira Denis Muhando
|
869000ceb7 |
fix(distributed): exclude pinned models from cluster eviction and idle scale-down
pinned: true was only honoured by the per-node watchdog. Every distributed eviction path was pinned-blind: the router's LRU eviction (EvictLRU, evictLRUAndFreeNode) and the replica reconciler's idle scale-down would happily unload a pinned model — and since eviction is gated on in_flight = 0, a pinned model became eviction-eligible the instant each response completed. Under capacity pressure that surfaces as the backend being freed immediately after every request (#11101). Wire the model config loader into the router and reconciler through a new PinnedModelResolver seam (mirroring ConcurrencyConflictResolver): - EvictLRU passes the pinned set into FindLRUModel's query so the next-oldest unpinned model is selected instead of the attempt failing - evictLRUAndFreeNode filters pinned models inside its locked selection - scaleDownIdle skips pinned models entirely: trimming to the floor still means requests beyond the survivor's capacity pay a cold reload Deliberate teardown (admin unload, model delete, node drain) intentionally still applies to pinned models, as does dead-row reaping (state correction, not eviction). Regression specs verified to fail with the exclusion disabled. Addresses the cluster-side eviction gap in #11101 Assisted-by: Claude Code:claude-fable-5 [Claude Code] Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> |
||
|
Ettore Di Giacinto
|
d55474a149 |
feat(audio): list available TTS voices
Clients cannot discover the named voices that an installed TTS model accepts without consulting backend-specific documentation. Expose voice metadata through the audio API and let custom model configs declare their own catalog. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a15780858e |
feat(diffusers): add AudioLDM2 generation
Expose diffusers audio pipelines through the existing sound-generation RPC. AudioLDM2 can now return PCM WAV output from the model gallery without a separate backend. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
21df4a120a |
feat(mlx): add Apple Silicon video backend
Add a Darwin-only MLX-Video backend for LTX-2 and converted Wan checkpoints, expose it through the existing video API, and wire packaging, discovery, tests, docs, and an example. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3e4a44be9d |
fix(diffusers): forward original config for single files
Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
419375baae |
fix(config): register original diffusers config field
Expose original_config_file through the model configuration metadata registry so registry completeness checks and generated configuration surfaces stay in sync. Assisted-by: Codex:gpt-5 [systematic-debugging] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
9bd7d17ff6 |
[model-config] feat: add environment variables support for backends (#10721)
* feat: add environment variables support for backends in model configurations - Add field to model configuration to pass environment variables to backend processes - Update backend options and model configuration handling - Add documentation for environment variables configuration with examples including CUDA_VISIBLE_DEVICES Assisted-by: qwen-agentworld-35b-a3b Signed-off-by: nold <nold42@pm.me> * fix(test): Test environment variables configuration parsing from YAML Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: nold <Nold360@users.noreply.github.com> --------- Signed-off-by: nold <nold42@pm.me> Signed-off-by: nold <Nold360@users.noreply.github.com> Co-authored-by: nold <nold42@pm.me> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |