mirror of
https://github.com/mudler/LocalAI.git
synced 2026-11-04 11:33:22 -05:00
cfafec79cabc7da7e8a85908dffcd3e8747df70f
45
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 mudler-agentandEttore Di Giacinto
|
2cadf319a0 |
feat(realtime): LFM3-Audio speech-to-speech on llama.cpp, remove liquid-audio (#12958)
* chore(backends): remove the liquid-audio Python backend The backend targets liquid-audio, which loads LFM2 checkpoints only. Remove its directory, the gallery file and the LFM2.5-Audio model entries, the library and generated backend gallery entries, the backend matrix and CI job, the Makefile targets and the golden test fixtures that listed it. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(config): drop liquid-audio from capabilities, importers and UI With the liquid-audio backend gone, remove its capability entry, its importer, its artifact-gate entry and the realtime_audio heuristic that only matched that backend. realtime_audio is now set only through known_usecases. Update the Talk and fine-tuning pages, the docs and the swagger text that named the backend. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump llama.cpp to e486a4b5 Move the llama.cpp pin to e486a4b559617dc88bf79978996f6c670b6bbc9d, the master head of 2026-10-28. It contains the LFM3-Audio support (ggml-org/llama.cpp#31257) and the realtime server split (ggml-org/llama.cpp#31269). The score task patch is regenerated against the new pin. It applies with git apply and with patch -p1, with no offset and no fuzz. Its hunks in server-task.h and server-context.cpp were rebased by hand next to SERVER_TASK_TYPE_REALTIME and the control dispatch. The C++ backend was built on CPU for this change: the llama, mtmd and server-context libraries and the grpc-server binary link. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): compile server-realtime.cpp into grpc-server server-context.cpp includes server-realtime.h, and its realtime slot type is defined in server-realtime.cpp. Upstream compiles that file into the server library. The gRPC server is one translation unit, so it must include the file too, or the link fails. The include is guarded by __has_include. Forks that predate the realtime split do not ship the file, so they keep building as before. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): probe realtime and TTS voice support in prepare.sh prepare.sh writes llama_compat.h from probes on the checkout. Add two: LOCALAI_HAS_REALTIME (SERVER_TASK_TYPE_REALTIME in server-task.h) and LOCALAI_HAS_TTS_VOICE (set_voice in mtmd-helper.h). The probes let the gRPC server choose the realtime and voice code paths per checkout, so forks without them keep compiling. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): advertise speech recognition and speech-to-speech llama-cpp now registers the AudioTranscription and AudioToAudioStream RPCs, with the transcript and realtime_audio usecases. The default usecase stays chat, so a plain GGUF is still a chat model. The backend implements the RPCs in a follow-up change; until then the calls return UNIMPLEMENTED. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(config): add pipeline.any_to_any for speech-to-speech models A realtime_audio model that serves the whole speech loop itself sets pipeline.any_to_any. The flag is off by default, so chained pipelines and existing realtime_audio configs are unchanged. The realtime_audio heuristic stays explicit: a llama-cpp model is never a Talk-page model unless known_usecases says so. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(config): read the audio encoder and speech-to-speech projector flags Add HasAudioEncoderProjector for clip.has_audio_encoder and HasSpeechToSpeechProjector, which requires the LFM3 audio encoder family together with the gen-audio decoder. Qwen3-TTS sets both encoder flags too, so the flags alone cannot tell it apart from LFM3. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(importer): import LFM3-Audio projectors as speech-to-speech An mmproj that carries the LFM3 audio encoder and the gen-audio decoder now imports with known_usecases realtime_audio, chat, transcript and tts, pipeline.any_to_any: true, the parallel:2 option and the tokenizer template. Other gen-audio projectors keep the TTS-only result, Qwen3-TTS included. The probe runs against a local HTTP server in the tests. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): named-voice TTS and the LFM3 ASR prompt Models without a speaker reference (LFM3-Audio) take a named voice through their "tts" chat template instead of a reference file. The request is validated with needs_reference=false, the prompt is rendered with common_chat_get_tts_prompt(), and the voice is set on the gen-audio input. Qwen3-TTS Base keeps the reference-file rule. The branch is guarded by LOCALAI_HAS_TTS_VOICE, so forks without named voices keep the old behaviour. Transcription takes its prompt from common_chat_get_asr_prompt(). LFM3 gets a system turn that names the language and an empty user turn. Qwen3-ASR's preset is unchanged: fixed user text, no system turn, and a caller prompt is still appended after a newline. The preset call is guarded by a new LOCALAI_HAS_ASR_PRESET probe. The pure parts sit in tts_request_options.h and transcription_prompt.h, with standalone tests run by backend/cpp/run-unit-tests.sh. Add the test-extra-backend-llama-cpp-lfm3 target. It runs the tts, ttsstream and transcription specs against LFM3-Audio GGUF weights. It is a manual run and is not wired into CI. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): mock AudioToAudioStream and pending realtime S2S specs The mock backend serves AudioToAudioStream deterministically: session created, commit with RMS, a tool call on the first turn with tools, then transcript pieces and paced 24 kHz tone frames, with response.cancel ending a reply as cancelled. The realtime S2S specs cover the mock and real-weights variants. They skip until the /v1/realtime S2S path lands, so the mock suite stays green in CI. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): speech-to-speech AudioToAudioStream on the realtime slot Implement AudioToAudioStream on upstream's realtime slot, so a model that answers after the audio (LFM3-Audio) can hold a conversation over the existing duplex RPC. The proto schema does not change: tools travel in the config, tool calls leave as function_call_arguments.done and tool results arrive as a function_call_output item. realtime_session.h keeps what needs no model: the turn text derived from the chat template (with a tool turn), the conversation state, a filter that keeps <tool_call> blocks out of the transcript and drops audio produced inside them, and the translation of slot results into events. grpc-server.cpp wires it to the slot with control tasks and parses calls with the parser the chat template selects. The unit test uses a fake renderer. A ctest-only check renders the real LFM3 template and compares the slot text with the full render. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add the LFM3-Audio-2B entries Two llama-cpp entries for LiquidAI/LFM3-Audio-2B-GGUF, the Q4_K_M default (lfm3-audio-2b) and the Q8_0 build (lfm3-audio-2b-q8), both with the Q8_0 audio projector. Each declares pipeline.any_to_any and the realtime_audio, chat, transcript and tts usecases the importer writes for an LFM3 projector, with parallel:2 and the tokenizer template. The Q4 entry lists the Q8 one as a variant. File sizes and sha256 come from the Hugging Face API for the pinned repo revision. Speaker reference files and the VAD model are not part of the entries: VAD stays optional. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe LFM3-Audio speech-to-speech and its gallery entries Adds a speech-to-speech section to the Realtime API page (any_to_any configuration, VAD versus manual commit, tools, transcripts, barge-in, limits), a pipeline.any_to_any row, a llama.cpp speech-to-speech row in the compatibility table, and LFM3-Audio mentions on the TTS and ASR pages. The feature is marked experimental: the behaviour follows the backend session contract and has not been run with real weights. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): run any-to-any models on AudioToAudioStream A realtime_audio model with pipeline.any_to_any now opens one duplex stream to the backend and keeps it for the session, replacing the "any-to-any models not implemented" branch. The chained pipeline is unchanged. The endpoint forwards input audio (with a pre-roll from speech start when a VAD model is set, otherwise every chunk and a client commit), sends response.create, and turns the backend's events into the usual realtime events: audio and transcript deltas, tool calls through the existing function_call items, and a single response.done. A function call output is forwarded before the next response. Barge-in sends response.cancel ahead of the new speech and stops forwarding audio at once. The user's speech is transcribed on the side and reported on the user item; a failure there does not touch the response. Server VAD is on only when pipeline.vad is set. Classifier mode, compaction, streaming.llm, voice gating, sound detection and semantic_vad are rejected for these models. The mock backend now matches the llama.cpp contract (session.updated, the cancel error when idle) and its VAD reports the loud stretches of the audio, so the S2S e2e specs run a VAD turn. The specs are enabled. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): add the real-weights realtime S2S variant and the gRPC s2s spec The realtime S2S specs now check reply audio on real models: duration, loudness, clipping, and keywords found in the model transcript or in a separate ASR of the output audio. run-realtime-lfm3.sh stages the llama-cpp and sherpa-onnx backends, the LFM3-Audio files and the sherpa VAD, ASR and TTS models, then runs the real variant. It is manual and never part of CI: `make test-realtime` still filters out real-models. The backend suite gains an s2s capability (spoken turn, tool call, tool result, spoken answer over AudioToAudioStream) and optional TTS duration and RMS bounds plus a transcript keyword check, all enabled for test-extra-backend-llama-cpp-lfm3. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): expect the idle-cancel error from the mock S2S The mock now answers a response.cancel with no reply running with response_cancel_not_active, as the llama.cpp backend does. The cancel spec still treated that cancel as a no-op and read the error as extra data after the stream ended. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(talk): show any-to-any models as one speech-to-speech model /api/pipeline-models lists a pipeline.any_to_any model with any_to_any set and its optional VAD. A realtime_audio model that is not any-to-any is listed only when it has the four pipeline slots, since the session takes the chained path for it. The Talk page shows a speech-to-speech model as one model: no four-slot view, the VAD slot only when a VAD model is set, and the female and male voices offered. Without a VAD model the server does not detect turns, so the page offers Done speaking, which sends input_audio_buffer.commit and then response.create. Text-only replies stream through response.output_text.delta and .done. self_contained stays in the response for clients built before any_to_any. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): refuse a llama-cpp realtime model without pipeline.any_to_any llama.cpp has no VAD, transcription or TTS RPCs, so filling the empty pipeline slots with the model name failed later with UNIMPLEMENTED. Fail at session start and name the missing setting instead. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): hold manual-commit S2S audio until the commit In manual-commit mode the endpoint streamed every appended chunk to the backend, so input_audio_buffer.clear only emptied the Go buffer while the model kept the audio, and idle audio filled its context between turns. Keep the audio in the input buffer and send it, behind pending text and tool-result items, when the client commits or asks for a response. Frames and controls now go through one ordered write queue, so no gRPC send happens under the audio buffer lock. A tool result created while a VAD turn is open is refused with a clear error. When the stream ends outside a response the session reports it and closes, and a bare EOF from Send reports the status that Recv holds. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): drop the S2S readiness gate and stale comments The gate was always true. Some comments referred to work items that readers of the repository cannot see. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): add the liquid-audio migration note and the clear semantics Say that the Python liquid-audio backend is gone, which gallery entries replace it and that LFM2.5-Audio has no replacement. Describe what input_audio_buffer.clear drops for speech-to-speech sessions now that manual-commit audio is held back until the commit. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): do not promise turn detection when no VAD model is set The idle note said the server notices when you stop talking. A speech-to-speech model without a VAD model needs Done speaking, so the note now says that. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): make the mock S2S backend keep the llama-cpp session limits The mock accepted what the real backend refuses and hid two endpoint bugs. It now treats output_modalities from the config as a fixed session mode and picks the default per response from the tools set at that time. A fixed text plus audio mode with tools reads the tool call out loud. After the first commit or message, session.update of instructions, voice or tools is an error that does not belong to a response, committing an empty buffer is an error, and response.create while a reply runs is refused. Errors of a refused response.create carry response_scoped in their meta. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): leave the S2S output mode to the backend unless the client sets one The stream config sent text plus audio when the session had no tools yet. The llama-cpp backend keeps a mode from the config as a fixed session mode, and tools added later by session.update never cleared it. The model then read tool calls out loud. Send output_modalities only when the client chose them. The backend picks the default of each response from the tools it has when the response starts. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): mark the realtime errors that answer a response.create The error event now carries response_scoped: true when it ends the request for a response (a refused response.create, a failed commit made by it, or a stream failure). Other errors, such as a refused session.update or a commit of an empty buffer, are not about a response and one that runs keeps generating. The client needs the difference to know whether a response.done is still coming. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end an S2S response only on the errors that belong to it Every backend error ended the response in flight, although the backend also reports errors about other things (a refused session.update, a commit of an empty buffer) and goes on generating. The rest of the reply was lost and the next response.create met either a refusal or the leftover events of the old reply. End the response only on an error marked response_scoped, forward the others to the client and keep waiting for response.done. Drop the events buffered by an earlier response when a new one starts, and skip the commit of a turn that forwarded no audio. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): run the S2S input transcription detached from the commit slot The commit waited for the side transcription of the user audio while it held the commit slot, and response.create waits for that slot. The ASR latency was added before every reply, and on a backend with few slots a busy ASR task could stall the reply. The model already has the audio, so run the transcription detached under the session context and release the slot at once. The transcript is still reported on the user item. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): do not pin the place of the input transcript in the S2S order The transcript of the user audio is reported on the side and may land before or after the first response events. Check the order of the response events without it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): stop an S2S reader leak and a cancel that waits on the write queue The reader waited on a channel it closes itself when the events buffer was full, so it outlived the session after a turn ended on an error or timed out. It now also stops when the session closes. Stopping a response sent the cancel through the write queue and waited for it before the grace timer started. With backpressure on the stream the wait never ended and nothing read the events meanwhile. The timer now starts first and the cancel is only queued. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): let a chained pipeline run without a VAD model A pipeline that names no pipeline.vad failed to build, because the VAD slot was always loaded, and a new session still started with server_vad. The slot is now optional for chained pipelines as it already is for any-to-any models: the session starts with turn detection off and the client commits each turn. Turning server_vad on for such a session is refused, and a switch to a pipeline without a VAD drops turn detection. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add LFM3-Audio VAD and chained-pipeline entries Add lfm3-audio-2b-vad and -q8-vad (any-to-any with silero-vad-ggml as the turn detector), and lfm3-audio-2b-pipeline, -pipeline-vad and the -q8 forms (the model chained through its own transcription, chat and speech slots, with and without the VAD). The gallery has no dependencies between models, so the VAD entries state that silero-vad-ggml must be installed first. Drop the variants list from lfm3-audio-2b: it let the installer pick the larger Q8 build on any host with enough memory, so a plain install of the Q4 entry did not get Q4. All entries are now independent. Tests cover the entry shapes, the single-build install and their acceptance by the realtime endpoint. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): run the realtime specs for any-to-any and chained shapes The realtime specs covered one any-to-any model with a VAD. They now run for any-to-any and for a chained pipeline, each with and without a VAD, on the mock backend in CI. VAD-less shapes also check that turn detection starts off and that server_vad is refused. The manual real-weights script writes one config per shape that mirrors the gallery entries and runs the four shapes in turn (LFM3_MATRIX picks a subset). Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): document the LFM3-Audio VAD and chained-pipeline entries List the eight LFM3-Audio gallery entries, explain that silero-vad-ggml must be installed before the -vad entries, describe the chained pipeline on one model and the behaviour of a pipeline without a VAD, and describe the shape matrix of the e2e specs. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): wait for the detached input transcription in the S2S specs The input transcription runs in its own goroutine, so the transcription completed and failed events can arrive after the response ends. Poll for them instead of reading the event list right after the response. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): release the S2S stream before running server-side tool calls A response held the stream slot until it returned, and it ran the tool calls from inside that scope. A server-side tool starts a follow-up response on the same stream, which then waited for the slot its parent still held, so the client never got response.done. Run the tool calls once the slot is free; the event order is unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end an S2S session whose stream dies, and stop queuing after close A stream that died during a response left the session open: later requests failed one by one and VAD frames kept being forwarded. The response now ends the session through the transport, and enqueue drops requests once the session is closed so the write queue cannot grow. The writer is now the only caller of Send and of CloseSend. close() used to call CloseSend while the writer could be inside Send, which gRPC does not allow. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): close the open S2S turn when the VAD stops Switching turn detection to manual in the middle of a VAD turn left the turn forwarding, and the next client commit sent the whole buffer a second time. Stopping the VAD now commits the forwarded audio on the backend and drops it from the input buffer. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): keep a tool result pending while an S2S user turn is open The check that refuses a tool result during a user turn ran when the item arrived, but the item was forwarded later, and the result was marked as sent before the backend took it. If the user spoke in between, the backend refused it and it was never retried. The result is now claimed and queued under the audio lock only when no turn is open, and otherwise stays pending for the next commit or response. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): empty the input buffer for models without a VAD Without voice detection the microphone fills the server's input buffer for the whole session, so Done speaking committed idle silence and the echo of the previous reply. Clear the buffer when the session starts and when a reply ends. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(importer): keep an existing parallel option on LFM3-Audio imports The speech-to-speech import added parallel:2 even when the options already carried another parallel value. Use the same check as the hardware defaults, which also covers n_parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): list chained pipelines that have no VAD /api/pipeline-models required a VAD for a chained pipeline, so the VAD-less LFM3-Audio pipeline entries never showed up on the Talk page. Any pipeline with transcription, LLM and speech slots now qualifies. The Talk page treats every model without a VAD as manual-turn, and the session sheet says so for chained pipelines too. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump llama.cpp to 0ac1a7fd Move the llama.cpp pin to 0ac1a7fd35c127a18c0cbc6c57d12e256a6a385b, the master head of 2026-10-29. It contains tool calls in the realtime server (ggml-org/llama.cpp#31287). The score task patch is regenerated so that it applies with no offset and no fuzz. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): keep a null tool argument null A tagged parameter whose text was "null" came out as an empty string: unmarshalling JSON null into a string succeeds and leaves it empty, so the value was taken for a string. Check the first byte of the JSON value instead. A parameter with the text "null" now becomes JSON null, as in llama.cpp. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(functions): add an lfm3 XML format preset The LFM3 chat template writes a call as <tool_call>, <function=NAME>, one <parameter=KEY> block per argument, then the closing tags. The qwen3-coder format has the same shape and auto-detection already found it, but xml_format_preset: lfm3 was not a known name. Add the preset, with tests for one call, a call without arguments, two calls, string and non-string values (numbers, booleans, null, arrays, objects), a call between text, and unfinished input. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): parse tool calls with the chat template, as upstream Since ggml-org/llama.cpp#31287 the realtime server handles tools with the parser and grammar of the chat template. Do the same in the speech-to-speech stream instead of a hard-coded tag filter and a parse of the whole text at the end. - Each response gets a common_chat_session with the tools of the session and parallel calls. Its grammar and preserved tokens go into the sampling of the response, so the slot keeps the call tokens in the text and constrains the call. - The text of the response is fed to the session piece by piece (task_result_state::update_chat_msg). Content deltas become the transcript; tool-call deltas are collected into one function_call_arguments.done per call at the end. Audio after the start of a call is dropped. - A tool_choice of auto, none or required is accepted in the params of the config, in session.update and in response.create, with the response overriding the session. - The tool turn is derived from a conversation in which the assistant called a tool, like upstream, not from a plain reply. - response.done carries raw_text, everything the model wrote with the call markup, for a caller that parses it another way. If the parser throws, the rest of the response is content and the raw text is the way out. ToolCallFilter is no longer used and goes in the next commit. The unit tests cover the splitter contract (content, call deltas, the final piece, whole characters only). realtime_chat_check feeds LFM3-format text through the session in pieces of 1, 7 and all bytes and checks the call, the transcript and the grammar. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(realtime): drop the tool-call text filter The parser of the chat template now splits content from tool calls, so ToolCallFilter, which looked for the literal <tool_call> tags, has no caller. Remove it with its tests. Every case it covered (a call between text, split tags, several calls, an unfinished call, text after a call) is handled by the parser, and realtime_chat_check runs the LFM3 format through it in pieces of different sizes. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): fall back to the raw text for S2S tool calls For an any-to-any model, the calls the backend parsed still win. When there are none and the request had tools, parse the raw text that response.done now carries the way the chained pipeline parses the text of its LLM (CleanupLLMResult, ParseFunctionCall, ParseTextContent), so a call format the backend parser does not know can still be served with function.xml_format_preset or the other function settings of the model. The content around the calls replaces the streamed transcript only if the config captures it, as in the chained path. Also pass tool_choice to the backend: the session mode with the stream config and with session.update, and the mode of a response.create override with that event. A named function cannot be forced, so it is not sent. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): carry raw_text in the mock S2S response.done Match the llama-cpp session: with tools set, response.done has raw_text, the call markup of the LFM3 template for a tool turn and the transcript for any other reply. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): parse tool blocks that have text between them The iterative XML parser stopped at the first scope block that was preceded by text, so "<tool_call>A</tool_call> then <tool_call>B" gave one call. Skip text between blocks once a call was found; text before the first block still ends the parse as before. Blocks that follow each other, or only whitespace, already worked. Tests cover three calls back to back and with text between them, with the lfm3 preset and with auto-detection. They also pin the current reading of argument values: the parser has no tool schema, so a value such as 42 or true is read as JSON even when the schema types it as a string. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): read LFM3-Audio tool calls with the lfm3 preset Set function.xml_format_preset: lfm3 on the eight LFM3-Audio-2B entries and in the config the importer writes for a speech-to-speech projector, so the server parses the call format of the LFM3 chat template when the backend returns no call (and in the chained pipeline entries, which use the regular chat path). gallery/index.yaml is regenerated from the library. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): describe how S2S tool calls are read Replace the description of the hard-coded tag filter with the parser and grammar of the chat template, tool_choice, and the raw-text fallback in the server. Document function.xml_format_preset, and set it in the real-weights e2e model configs like the gallery entries do. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): keep earlier calls when a later block does not parse After a valid call, text that holds the opening tag again (a prose mention, a markdown example, a malformed second block) made the parser treat it as a new block, fail, and return no calls at all. Remember the state after each complete block and fall back to it when a later one fails: the calls found so far are kept, parsing stops there and the remaining text stays in the content. Text skipped between two valid blocks now goes to the content instead of being dropped, so StripToolCallMarkup keeps it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): do not drop audio after a tool call started In mixed mode the speech for text written before a call can still be pending when the call starts, so dropping every audio frame after the first call delta cut the speech while its transcript was sent. The llama.cpp server never drops audio when it reads a result; do the same. If a model speaks the call markup aloud, that is model behaviour to observe on real weights, not something to filter here. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): flush held-back text when a response is cancelled The parser got is_partial=true on a cancelled response, so text it held back (a possible start of a tag) never reached the transcript. The llama.cpp server parses a cancelled response with is_partial=false; do the same. A cancelled response still makes no tool call. The path where the parser throws already sends the text as content and needs no change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): stop the fake S2S stream from sending after Close The barge-in spec failed now and then with a data race and "send on closed channel": its script pushes audio from a goroutine, and the session closes the stream while that goroutine still runs. A push now does nothing once the stream is closed, and Close waits for a push in flight. The production close order is right: the writer closes the stream only after it stops sending. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): align the S2S tool-call fallback with the backend Four gaps in how the Go side reads the tool calls of an any-to-any reply: - With tool_choice none (the response value wins over the session value) the backend builds no tool parser, yet the raw-text fallback still turned a call the model wrote into a function_call item. Skip it. - Without a capture regex the fallback never cleaned the transcript, so the stored item and transcript.done kept the raw call markup. When the call is XML, use the text left after the markup is cut out; a reply that is only a call now has an empty transcript. - A named function as tool_choice mapped to nothing, so the backend kept the previous mode. Refuse it with an error in session.update and response.create. - A response-level tools list changed what the fallback looked for, but the backend only has the tools of the session. Use those in both. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): say how S2S tool_choice and response tools behave Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(e2e): use the real llama-cpp backend in the real realtime run The suite always registered the mock backend under the name llama-cpp, so a run with REALTIME_S2S_MODEL set never reached the llama-cpp backend found in REALTIME_BACKENDS_PATH. Register the mock only when that run is off. The real run converts the input audio with ffmpeg. Check for it in the suite and in run-realtime-lfm3.sh before any download or backend start, with a message that says what to install, and list it as a prerequisite in the script header and the Makefile target. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): cover the tools path of a client session.update to the S2S backend A tool call that never came back from an any-to-any model raised the question whether the tools of the session reach the backend at all. Add a spec that sends the session.update of the real-weights e2e (flat Responses API tool, no session type) and checks that the backend gets the tool in the chat-completions form. Log the number of tools sent when the stream opens and when a session.update carries them, so a run log shows it without reading the wire. The "No tools received" line of the llama-cpp backend does not show it: the server prints it for the input transcription request, which has no tools by design. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): interrupt the chained pipeline reply while it is in flight With unary stages, the default of the lfm3 pipeline gallery entries, the chained pipeline synthesizes the whole reply and sends it as one audio delta right before response.done. The barge-in spec waited for that delta and then rendered the interruption with the TTS, so the first response was already done when the speech started and ended completed instead of cancelled. Real runs on a GPU showed it with both quantizations. For a pipeline, start the interruption at response.created, which comes before the LLM and TTS stages. Render the interruption audio before the first utterance is sent, so nothing sits between the reply and the interruption. The any-to-any shape still waits for the first audio delta, which that backend streams while it generates. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): accept a number word and its homophone in the spoken-answer check The check that the reply audio says what the transcript says needs a shared word of 4+ letters. A short answer failed it: the model wrote "Four" and the ASR heard "for". Compare numbers in one spoken form, so digits, number words and their homophones match each other. A homophone only counts when the other text has the number itself, so "for" on both sides still shares nothing and the check does not pass on a stop word. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): ask LFM3-Audio for function calls when a realtime session has tools The default system prompt of a realtime session was "Respond with interleaved text and audio." whatever the tools. Under that prompt LFM3-Audio answers in words ("I'll check the weather ...") and never writes the call. The model card, and the llama.cpp realtime example client, use "Respond with function calls, and interleaved text and audio." when the session has tools, and the model then calls the tool. Pick the default by the tools of the session, and treat the plain default that the LocalAI server sends back as no choice: it follows the tools too, also when they arrive with a session.update before the first audio. A prompt the client wrote is kept as it is. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): transcribe the input of an any-to-any turn between responses The input transcript of a speech-to-speech turn ran at the same time as the reply, on the same model. llama.cpp serves both from one context: the realtime slot wants embeddings on, the chat slot of the transcript wants them off, and each flip makes it reserve its compute buffers again. In a GB10 run that was two reserves of about 175 ms per step, the reply fell to 2.6 tokens per second and the transcript to 380 ms per token, until the transcript was done. When pipeline.transcription is the model of the stream, run the transcript after the response, or before one that is not started yet, and keep responses out while it runs. A separate transcription model still runs in parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): send the input transcript of an any-to-any turn before its reply The previous change moved the same-model transcript after the response. A client that reads until response.done then never saw conversation.item.input_audio_transcription.completed: the transcript was made (the log shows it) but sent after the end of the window. Run it in line with the commit instead. A VAD turn transcribes and then responds. A client response.create already waits for the commit slot, so it starts once the transcript is out. The transcript event now comes before any output of the response of its turn, for a manual commit and for a VAD turn, and never overlaps the response on the llama.cpp context. A transcription model of its own still runs in parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): transcribe the input of an any-to-any model only when asked The stream answers from the audio, so the transcript of the input is for the client only. On the model of the stream it is costly: llama.cpp serves the transcript request and the realtime stream from one context and rebuilds its compute buffers on every step while both are active. On a GB10 that was 26 to 29 reserves of about 175 ms per turn, against 0 to 1 on the llama.cpp server, and the reply ran at a few tokens per second. An any-to-any model no longer fills pipeline.transcription with its own name. Without it the session does not transcribe the input; with another model it does, in parallel with the reply; with the model itself named it does, before the reply. The reply transcript always comes from the stream. The gallery entries set none, so they get the fast path. The mock configs and run-realtime-lfm3.sh name a transcription model, so the real-weights spec still covers the input transcript. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end a response as cancelled when a barge-in lands during the LLM call The chained pipeline cancels the response context on a barge-in. When that happens while the model is generating, the backend call fails with the context error, and the response ended as failed: an error event and no response.done. A client that waits for the terminal of the interrupted response then waits forever. On a GB10 the q4 model answered before the barge-in arrived, so only the slower q8 model showed it: the spec read events for 30 s and timed out. When the context is cancelled, a failed Predict call or a failed result is a cancel, as the TTS step and the streamed LLM path already treat it. The response ends with response.done{cancelled} and no error. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): wait for the input transcript that comes from a model of its own An any-to-any model now gets its input transcript from a separate ASR that runs next to the reply. The reply is often over first, so a spec that stops reading at response.done saw no transcript in the VAD shape. Keep reading, up to 30 s, until the transcript arrives. The contract is that it comes, not when. Add a guard spec for what the input transcription is handed: the whole turn, in a WAV that says the rate it was recorded at (the session rate for a client commit, 16 kHz for a VAD turn). Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): check the errors of Close in the realtime specs golangci-lint (errcheck) flagged the unchecked Close of the websocket connections, the response body and the fake stream. Use the `defer func() { _ = x.Close() }()` form the other e2e specs use. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): write the mock S2S specs with Ginkgo and drop the env read forbidigo only allows Ginkgo/Gomega specs and no os.Getenv outside the sanctioned places. Convert the mock S2S tests to Describe/It with a suite bootstrap (the package had none), with the same 11 cases. The mock no longer reads MOCK_S2S_FRAMES: only these specs set it, so the reply length is a package variable they change per spec and restore afterwards. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
3a02f29edc |
fix(proxy): propagate model env and accept direct keys (#12940)
* feat(proxy): accept direct upstream API keys Support direct credentials for models created in the UI. Keep env and file references as mutually exclusive alternatives. Use model env for frontend probes and omit credentials from structured config logs. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): isolate model environments in worker processes Forward complete model environment snapshots through backend installation. Restart worker children when overrides change or are removed, and include keyed snapshot identities in frontend install coalescing. Cover wire compatibility, routing, real worker children, and option copying; document rollout requirements and control-transport security. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(worker): serialize installs by model replica Different backend names can target the same process slot. Hold a slot lock through startup, readiness, and operation attachment so concurrent installs cannot reuse another configuration or an unready child. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(nodes): update endpoint stub for model env Keep the endpoint command stub aligned with the environment-aware install interface. Assert interface conformance so future changes fail at the stub declaration rather than at endpoint call sites. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Stefan Walcz
|
6d7e8171be |
feat(agents): per-model output cap for agent job completions
Agent jobs (/api/agent/jobs, local and on distributed agent workers) run
cogito against a LocalAI client that sends no max_tokens of its own, so
every completion gets the client's fixed default of 16384. max_iterations
bounds the number of turns and loop_detection repeated tool calls, but
neither stops one completion that keeps generating, and 16384 is the same
for a small tool-picking model and a reasoning model that thinks at length.
Add agent.max_generation_tokens to the model config. When set, both job
paths apply it to the client with SetMaxTokens; requests that carry their
own cap still win, so cogito's length retry can raise it. Unset keeps
today's behaviour.
The specs run each job path against an MCP fixture and a recording LLM
endpoint and check the max_tokens the completions carry; they fail when
either call site drops the cap.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
mudler-agentandEttore Di Giacinto
|
c60a2e47d6 |
feat(backends): timestral.cpp forecast backend and /v1/forecast API (#12885)
* feat(grpc): add a Forecast RPC for time-series models Add Forecast to the backend service with columnar request and response messages: flat row-major arrays, NaN for missing values, and one message per series so a caller can send series of different shapes in one call. The response keeps the request order. Wire the RPC through the server, client, embedded client, connection evicting client, in-flight tracking client and the base backend, which answers unimplemented. core/backend gains a Forecast helper that stamps the model identity and records a trace, like the other single-purpose RPCs. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(config): add the forecast usecase FLAG_FORECAST marks a model as a time-series forecaster. The timestral-cpp backend declares it as its default usecase, so a config with that backend needs no extra lines. The backend is also listed as a non-text backend, so a forecast model is never offered as a chat model. It shows up in the gallery filter and in the capability list. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(api): add POST /v1/forecast Take one or more numeric series and return quantile trajectories for the next prediction_length steps. Targets are [variates][steps] or a flat array for one variate; JSON null marks a missing value. Past and known covariates and categorical codes are checked for length before the backend sees them. Quantiles default to 0.1, 0.5 and 0.9 in the backend. The route is gated by a new default-on forecast API feature. The endpoint is listed in the swagger spec, the /api/instructions index, the UI model filter and the capability symbols. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(backends): add the timestral-cpp forecast backend Wrap the timestral.cpp C ABI with purego, the way the other ggml backends do: libtimestral is built from a pinned commit with ggml embedded, and the Go binary loads it at start. Forecast validates the counts against the array lengths, groups the series by shape, makes one library call per group and puts the results back in the order the caller sent them. Quantiles default to 0.1, 0.5 and 0.9, and a level the model does not have is refused with the list of valid ones. A mutex guards the handle because the library is not reentrant. Request size is bounded by two model options. The unit specs use a fake library. The integration specs and a gRPC spec run against the 5M F32 model that test.sh downloads and checks against a fixed SHA-256. Assisted-by: Claude:claude-sonnet-5-5 go-vet * build(backends): register timestral-cpp in the build and the library Add the CPU image to the backend matrix for amd64 and arm64, plus an Apple silicon entry that is not built or tested yet. The image has no GPU keys because the engine runs on the CPU only. Wire the make targets, the path filter and the dependency bump bot. The library entries were generated with backendcatalog. The images are not published yet, so the release entries are unpinned until a maintainer runs --repin. The gallery reader test now counts the names in the library instead of a fixed number, so adding a backend does not break it. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(gallery): add the Timestral forecast models Add Timestral 5M, 28M, 123M and 209M as F32 entries, each with its F16 build as a variant. Q8_0 and the K-quants are left out until the quality question is settled. Each entry runs on the timestral-cpp backend, on the CPU only. The weights use Mistral's Modified MIT licence, which bars use by companies above a monthly revenue threshold. The first paragraph after the summary says so, the licence file is installed next to the weights, and the revenue-restricted-license tag makes the entries filterable. The memory each build needs has not been measured, so no size is set. Assisted-by: Claude:claude-sonnet-5-5 * docs(forecast): document /v1/forecast and the Timestral models Add a feature page with the request and response shapes, the error cases, the model options and the licence note. List the engine in the README, the compatibility table and the backend categories. Assisted-by: Claude:claude-sonnet-5-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Stefan Walcz
|
b56b903c92 |
feat(config): let model aliases carry an opt-in profile block (system prompt, MCP, client tools) (#12661)
* feat(config): let model aliases carry an opt-in profile block An alias can now act as a profile of its target through an explicit `profile:` block carrying system_prompt, mcp (remote/stdio servers, plus the new mcp.auto) and the new client_tools. ResolveAlias applies only that block on top of a copy of the target, while Name, Model, Backend and the persisted revision stay the target's. The loader key is therefore unchanged and every profile runs on the target's backend process; today a persona needs a separate model config, which loads the weights a second time. The block is opt-in so existing aliases never change: top-level fields on an alias stay ignored, and an alias without a profile block resolves to exactly the target. A profile block on a non-alias config is rejected. A profile with its own MCP servers caches its sessions under the alias name (MCPSessionKey), because the cache slot keyed on the target's name is invalidated whenever its config hash changes and two server sets would tear each other's sessions down. mcp.auto enables all configured servers on /v1/chat/completions when the request does not name any in metadata.mcp_servers, which plain OpenAI clients cannot send. client_tools: drop discards client tools and tool_choice before the request is merged (chat completions, responses, Anthropic messages), so only the model's own MCP tools are offered. Both keys work on any model and inside an alias's profile block. Refs #12548 Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(openai): only auto-enable model MCP when MCP is enabled and permitted mcp.auto turns a plain /v1/chat/completions request into a server-side MCP request. It now applies the same controls as the dedicated MCP routes before any server is selected: the instance switch (LOCALAI_DISABLE_MCP) and, with auth on, the caller's "mcp" feature, checked with auth.HasFeatureAccess against the auth database that the chat route now passes to ChatEndpoint. A denied request is answered as a plain chat, without contacting a server or adding tools, rather than rejected: the client did not ask for MCP, it only reached a model that offers it. Endpoint specs drive ChatEndpoint against a counting MCP fixture server: MCP disabled and a user with chat but without mcp see no tools and no connection; MCP enabled and a user with mcp get the fixture tool. The user specs need the auth build tag (SQLite auth database). Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(mcp): resolve profile aliases in the MCP discovery endpoints GET /v1/mcp/servers/:model, /v1/mcp/prompts/:model and /v1/mcp/resources/:model looked the model up with GetModelConfig, which returns the alias stub. A profile alias carries its servers in its profile block, so the stub has none of its own and the endpoints reported an empty list for an alias whose chat requests do use tools. Resolve the alias the way the request middleware does before reading the MCP config, and open the sessions under MCPSessionKey so discovery shares the profile's session cache slot instead of tearing down the target's. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * feat(config): opt-in evidence check for profile aliases A profile with server-side MCP tools (profile.mcp, mcp.auto) runs its tools on the server and returns only the answer text. On the next turn the client sends the history back without those tool calls and results, and the model then tends to skip the tool and answer from nothing. In a measurement on a follow-up question, 0 of 12 turns called the tool and the answers carried invented numbers (e.g. "0" where the true value was 9); with the tool trail kept in the history the tool was called 12 of 12 times. A prompt instruction does not reliably prevent this, so the check sits at the output. New profile keys: profile: evidence_check: numbers: true # enables the check note: "..." # optional, defaults to an English note Rule, per request and stateless: when the check is on, no MCP tool call succeeded on the server during this request (a call that returned an error delivers no evidence and does not count), and the final answer contains a number (\b\d+([.,]\d+)?\b) that occurs in none of the request's input (the profile's system prompt and every message as received, before compression), the note is appended after a blank line. "3,5" and "3.5" count as the same number. In a streamed response the note is one extra content delta before the finish chunk. Choices that hand tool calls to the client are not checked. The setting is only read from a profile block, so plain models and aliases behave as before; a note without numbers: true is rejected at validation. Tests (Ginkgo): the number rule (invented number, number from the input, no numbers, comma/dot decimals, system prompt as evidence, tool ran), the overlay into the resolved config with plain alias, target and a top-level key as counter-checks, validation, and ChatEndpoint specs on a stubbed backend for both paths: note on an invented number, none for a number from the request, none for a profile without the check, none after a successful MCP tool call against an in-process MCP server, and the note again after a failed tool call (arguments that are not JSON). Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> --------- Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> |
||
|
Stefan Walcz
|
f4505dbb5d |
feat(pii): optionally block content the detectors cannot inspect (#12755)
The PII middleware scans the text of every message, but images, audio,
video and files pass to the backend unchecked. For an egress policy
("nothing sensitive leaves the box") that is a hole: a photographed
invoice or an attached PDF reaches a cloud-proxy model without any check.
Add pii.block_unscanned_media. While PII is enabled for a model and the
option is set, a request that carries such content is rejected with the
usual 400 pii_blocked response and an audit event with pattern
__unscanned_media__, before any detector runs. Adapters report the
kinds of unscanned content through a new optional Adapter.Unscanned:
OpenAI chat (non-text content blocks), Anthropic messages (image and
document blocks; tool blocks stay allowed) and Ollama chat (images).
Off by default, so existing configurations behave as before.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
|
||
|
Ettore Di Giacinto
|
f420bfcc26 |
fix(pii): give extend_to_previous_word its own settings order
It shared order 215 with extend_to_next_word, so the settings UI had no defined order between the two. Move it to 216 and shift the protected terms fields. Pin that a number in front of a hit is never taken. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Stefan Walcz
|
e3341d21a0 |
feat(pii): extend a detection over the previous word (extend_to_previous_word)
The mirror of extend_to_next_word. A name model often tags the surname with
high confidence and scores the first name in front of it below min_score,
so "Gast Greta Waldmeister" became "Gast Greta [REDACTED:ner:LASTNAME]".
extend_to_previous_word lists entity groups whose accepted detections also
cover the word directly before them on the same line. The preceding word
must consist of letters (plus '-' and the apostrophe): a word with digits -
an amount, a date, a room number - is never taken, nor is a protected term,
and the search never crosses a line break.
Docs: operations/middleware.md and advanced/model-configuration.md.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
Stefan Walcz
|
e16b9d04cc |
feat(pii): protected terms that pass the NER filter unchanged
A name model reads many business names as surnames. Measured with
privacy-filter-multilingual on 24 synthetic German receipts and
invoices: the supplier name ("Hotel Seeblick", "Café Lindenhof", ...)
was masked as LASTNAME (0.78-0.98) or FIRSTNAME in 9 of 12 receipts
that carried one. The supplier is exactly what the upstream model needs
to categorise a receipt, and no score floor helps at those confidences.
pii_detection.protected_terms (inline) and
pii_detection.protected_terms_files (files inside the models path, one
term per line, '#' comments, re-read when size or mtime change) list
values that must pass unchanged. Each whole-word, case-insensitive
occurrence is replaced by a neutral placeholder before the detector
runs; every detection is mapped back onto the original text with the
placeholder parts cut out, so a protected term is never masked or
blocked, the detector never sees it as context, and event offsets keep
addressing the original text. A word extension (extend_to_next_word)
is dropped rather than run into a protected term.
Terms shorter than three characters are ignored. A missing or
unreadable file is logged and skipped: fewer protected terms means more
masking, never less. Paths are verified to stay inside the models path.
Opt-in, empty by default. Covered by Ginkgo specs in
core/services/routing/pii/protected_test.go (detector-visible text,
exact restoration, straddling detections, whole-word/case/line-wrap
matching, the segment path, file loading and reload), a YAML round-trip
spec, registry entries and the middleware/model-configuration docs.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
Stefan Walcz
|
9b13408c94 |
feat(pii): extend a detection over the next word (extend_to_next_word)
Address models tend to tag a postal code with high confidence but miss
or under-score the town right after it. Measured with
privacy-filter-multilingual on 24 synthetic German invoices: the postal
code was masked every time, while the town after it stayed in the clear
in 6 of 18 cases (4 not detected at all, 2 tagged CITY at 0.43/0.47,
below the configured floor). Masking the code alone leaves the pair half
redacted: "[REDACTED:ner:ZIPCODE] Ludwigshafen".
pii_detection.extend_to_next_word lists entity groups whose accepted
detections also cover the word that directly follows them on the same
line, with the same action and group. Horizontal whitespace is skipped,
a line break ends the search, and nothing is extended when no word
follows, so a hit never grows over bare whitespace or onto the next
line (or across the segment separator on the multi-message path).
The option is opt-in and empty by default; existing detector configs
behave exactly as before. Covered by Ginkgo specs in
core/services/routing/pii/extend_test.go (fail without the change), a
YAML round-trip spec, a registry entry for the model editor, and the
middleware and model-configuration docs.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
mudler-agentandEttore Di Giacinto
|
0177e4988c |
feat(images): add ESRGAN upscaling and Studio workflow (#12620)
* feat(images): add upscale model capabilities * fix(config): keep SD GGML upscale opt-in * feat(stablediffusion): add lazy ESRGAN upscale bridge Keep native image ownership in C++ and serialize context creation, inference and teardown. Use upscale_scale in backend options as the explicit upscale-only marker because ModelOptions does not carry usecases. Assisted-by: Codex:gpt-5 * fix(stablediffusion): validate ESRGAN settings * fix(upscale): forward typed settings and contain resolved model paths Typed settings replace legacy options without duplicates. Resolve model and trusted root symlinks before caching the model path, rejecting missing files and escapes. Assisted-by: Codex:gpt-5 * fix(upscale): validate scale and clean failed HTTP outputs Require an explicit scale bounded to 1-16, check backend results, and keep destination cleanup owned by the allocating endpoint. Cover validation, result failures, cleanup, and successful URL responses. Assisted-by: Codex:gpt-5 * feat(studio): add inline image upscale workflow Assisted-by: OpenAI:gpt-5 * test(studio): serve upscale capability fixtures * fix(stablediffusion): default ESRGAN tiling to full frame * feat(gallery): add RealESRGAN anime upscaler * fix(stablediffusion): allow full-frame ESRGAN tiles * fix(routes): select explicit upscale models Give image upscaling dedicated usecase-filtered middleware and keep ordinary image models out of the endpoint. Mark and verify the legacy Diffusers x4 gallery model as an explicit upscaler. Assisted-by: OpenAI:gpt-5 [shell] * refactor(upscale): keep settings in backend options * test(config): keep upscale options backend-specific * fix: keep upscale routing metadata out of backend options --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
99043b442c |
feat(router): route with native decision models (#12449)
* fix(schema): preserve SystemOne image inputs Assisted-by: OpenAI * test(schema): follow Ginkgo conventions for decision inputs Assisted-by: OpenAI * feat(llama-cpp): dispatch native decisions through Score Upgrade the stock dependency and reconcile Score/TTS patches. Reuse native decision parsing, tasks, formatting and response-reader cleanup; preserve ordinary scoring admission and guard older dependencies. Assisted-by: OpenAI * refactor(systemone): share request and model validation Assisted-by: OpenAI:gpt-5 * fix(systemone): preserve HTTP wire-byte validation limit Keep structural validation separate from the serialized internal request bound so HTML escaping cannot reject valid HTTP payloads. Assisted-by: OpenAI:gpt-5 * feat(systemone): bound images and account native decisions Preserve public wire limits independently from router serialization. Reject unsupported NER images, map native request/capability errors, and stamp explicit usage once. Advertise decisions for stock llama-cpp. Assisted-by: OpenAI * fix(systemone): record usage on registered native route Exercise real registration and billing with a mock native backend. Reject empty native responses, malformed image URLs, trailing JSON, and wire overflow including whitespace. Assisted-by: OpenAI * feat(router): add lazy native decision transport Bind named models through internal ModelSystemOne calls with shared validation and bounded abandoned operations. Remove request and echoed-error contents from decision traces. Assisted-by: OpenAI:gpt-5 * feat(router): classify overlapping policies with native decisions Ask independent noul questions, validate probabilities and preserve first-superset routing. Wire the central factory with config-sensitive invalidation and cancellation-safe resolution. Document native framing and bounded operation limits. Assisted-by: OpenAI:gpt-5 * feat(gallery): add pinned Julia-1 native decision model Add a separate text-only llama-cpp Q8 entry with pinned Apache-2.0 source provenance and checksum. Installed using the gallery installer and exercised choice, score and noul on CPU. Assisted-by: OpenAI * test(router): verify native decisions through central factory Add an opt-in real-model Ginkgo integration covering the native Go loader and C++ transport, token usage, independent overlapping labels, and candidate selection. Document owned-server execution and the intentionally non-quality threshold. Assisted-by: Codex:gpt-5 * fix(llama-cpp): align upstream pin and preserve decision signatures Advance to bed0a856 without losing the automated upstream bump. Detect full-request fill_task support at compile time and forward every question for Nimble framing while retaining the earlier native signature. Preserve reconciled SCORE/TTS patches; add standalone compatibility coverage. Assisted-by: Codex:gpt-5 * feat(gallery): add native decision family defaults Pin Laya, Kev-4B, lev, OpenJev and Nimble artifacts. Verify Laya/Kev/lev gallery installs and CPU contracts on both native pins; clearly mark OpenJev/Nimble runtime validation pending and their noncommercial licenses. Assisted-by: OpenAI * docs(decisions): clarify integrated Nimble prerequisite Record the exact combined backend pin while retaining pending OpenJev and Nimble installation/runtime validation status. Assisted-by: Codex:gpt-5 * fix(gallery): indent native decision model sequences Match repository yamllint indentation for Laya, Kev, lev and OpenJev list fields. Parsed gallery data is unchanged; reproduce CI gallery lint failure before the whitespace-only fix and pass the same command afterward. Assisted-by: Codex:gpt-5 * docs(decisions): record OpenJev and Nimble CPU validation Record gallery installation, checksum/metadata verification and multiquestion native smoke results on bed0a856. Retain noncommercial and text-only limitations without accuracy or deterministic-output claims. Assisted-by: OpenAI * fix(ui): expose native Decisions router classifiers Select classifier models using metadata-driven capability routing, retain tuned thresholds, and validate native decision selections before saving. Cover both native backends and create/save/reopen in the real React editor. Assisted-by: Codex:gpt-5 * fix(router): exclude aliases from native decision discovery Check the originally named config before advertising native Decisions eligibility. Retain target capability inheritance for ordinary generation aliases. Exercise the actual capabilities endpoint with native models on both backends, aliases, and disabled models. Assisted-by: Codex:gpt-5 * feat(systemone): share bounded multimodal input validation Preserve text wire limits while admitting bounded PNG/JPEG decision input. Share collection and header validation across internal and public callers and keep the native runner response budget independent. Assisted-by: OpenAI:API-assistant * fix(systemone): bound admission lifetimes and validate complete images Retain shared admission leases through actual work completion, including abandoned internal operations. Decode bounded image pixels, cap public native responses before usage stamping, and preserve oversized malformed text status precedence. Assisted-by: OpenAI:API-assistant * fix(router): classify images before media fetching Preserve ordered structured probes for native decisions. Defer OpenAI media preparation until routing selects the served model, so rejected decision URLs cannot trigger downloads before shared validation. Guard direct image collection with context-aware shared admission. Keep text classifiers and embedding caches from discarding image input. Retain fail-closed classifier configuration and runtime fallback policy. Add middleware, typed-content, admission, cancellation and cache tests. Assisted-by: OpenAI:API-assistant * fix(router): bound extraction before serialization Check probe budgets before copying text or marshaling message state. Count JSON escaping so oversized internal inputs fail before allocation. Preserve typed Anthropic blocks through selected-model conversion and fallback. Keep retry coverage in Ginkgo without global test registration. Assisted-by: OpenAI * fix(router): bound supported probe serialization Arbitrary structs can bypass the probe budget through pointer marshalers, string tags, and promoted fields. Accept concrete chat schema types and plain JSON values instead of emulating arbitrary struct serialization. Budget escaped direct prompts before marshaling so raw length cannot hide serialized expansion. Preserve runtime fallback and reject oversized input before invoking the decision runner. Add Ginkgo allocation, boundary, and marshaler invocation regressions. Six-package tests, three-package race tests, and full-T2 delta lint pass. Assisted-by: OpenAI:GPT-5 golangci-lint * feat(decisions): enable bounded OpenJev images Validate native decision images before permissive media parsing and pixel allocation. Require both decision image support and a vision projector; missing or audio-only projectors cannot silently become text decisions. Pin the OpenJev Q8 projector and document its license and disk footprint. Add native safety tests, canonical limit parity, gallery and load-option checks, and a reproducible CPU direct-RPC contrasting-image smoke. Assisted-by: OpenAI:GPT-5 * fix(decisions): reject incomplete image streams stb accepts corrupt PNG Adler checksums and truncated JPEG scans. Use bounded zlib validation and strict libjpeg decoding before parsing. Keep dimension and aggregate pixel checks ahead of decoder allocations. Wire decoder dependencies into native builds and runtime packaging. Add regressions for appended EOI and embedded marker bypasses. Assisted-by: OpenAI:GPT-5 * fix(ci): gate native decision image validation Run the decoder security tests outside the stdlib-only native suite. Fetch vendor headers at the backend pin and provision decoder dependencies. Gate Go limit parity and production CMake wiring without model downloads. Assisted-by: OpenAI:GPT-5 * test(decisions): cover multimodal public API paths Exercise shared image contracts through the registered HTTP routes and external mock backend. Add opt-in cached gallery installation and real OpenJev image decisions through SystemOne and both routing APIs. Assisted-by: Codex:gpt-5 * test(decisions): assert isolation and cache bypass Observe external RPC calls and compare complete classifier history. Winner-only and cache-miss checks could hide dropped history or cache use. Give real inference its own application and model directory so shared backend mappings and loaded processes cannot affect mixed suite order. Assisted-by: OpenAI:ChatGPT * test(decisions): isolate fixture globals Disable optional global services in the isolated HTTP fixture and register cleanup before setup assertions. Verify meter provider identity survives fixture creation and destruction. Snapshot observed usage before assertions so failures cannot retain the mutex. Require a successful usage stamp before checking error responses. Assisted-by: Codex:gpt-5 golangci-lint * fix(application): honor optional telemetry controls Skip failover gauge registration when metrics are disabled. Register against the application meter rather than looking up the global provider. Allow embedders to retain the bounded routing log without billing stats. Keep the existing default when stats are disabled. The isolated HTTP fixture uses this option without losing its native router assertions. Assisted-by: Codex:gpt-5 golangci-lint --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
2fa36e7147 |
feat(parakeet-cpp): speaker diarization, sound detection and live scene events (#12335)
* feat(parakeet-cpp): load diarization and CED models and companions
Repin PARAKEET_VERSION to parakeet.cpp PR #75's head, which adds
parakeet_capi_model_kind (ABI v8). Bind the new diarization, sound
event and combined scene stream C symbols through the same
purego.Dlsym probe pattern already used for the batched JSON entry
point, so the backend still loads against an older libparakeet.so.
Load now classifies the loaded GGUF by role (ASR, diarization or
sound) via parakeet_capi_model_kind and can load up to two companion
models from Options[] (asr_model:, diarization_model:, sound_model:,
paths resolved against opts.ModelPath), verifying each companion's
kind and freeing every context opened so far on any failure. Free
releases the primary and every companion. AudioTranscription now
names the loaded role when it is not ASR instead of a generic model
not loaded error. The dynamic batcher starts only when an ASR context
ends up loaded, primary or companion.
This is groundwork only: the Diarize and SoundDetection RPCs and the
live scene stream that actually use these new roles land in later
commits.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): reset role fields on a failed companion load
loadRoles' freeLoaded only released the C contexts it had opened; it
left ctxPtr/diarCtx/tagCtx and companions pointing at those now-freed
contexts, so a later Free() on the same instance would double-free.
Zero all four alongside the CppFree calls.
Also route AudioTranscriptionStream and AudioTranscriptionLive through
notASRError when ctxPtr is unset but a diarization or sound model is
loaded, matching AudioTranscription: both used to return the generic
model-not-loaded error instead of naming the loaded role.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(parakeet-cpp): add speaker diarization
Implement the Diarize RPC for the parakeet-cpp Go backend, wired to
Nemotron-3-Diarization through libparakeet.so's diarization C-API.
Plain diarization uses parakeet_capi_diarize_pcm; when include_text is
set and an ASR companion is loaded, parakeet_capi_transcribe_and_
diarize_json fills each segment's text instead. Speaker labels are the
decimal index, or "unknown" for -1 (no diarized speaker overlaps).
min_duration_off merges same-speaker segments across a short gap
before min_duration_on drops the segments still too short, then ids
are renumbered. num_speakers/min_speakers/max_speakers/clustering_
threshold have no Sortformer equivalent and are logged at debug
instead of rejected.
Verified against the real Nemotron-3-Diarization + parakeet-tdt_ctc-
110m checkpoints on the two_speakers.wav fixture: correct A-B-A-B
speaker segmentation and matching speaker-attributed transcripts.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(parakeet-cpp): add sound event detection
Wire the SoundDetection RPC to the CED tagger context (p.tagCtx)
loaded by Task 1's role classification. It runs the whole clip
through a one-shot parakeet_capi_sound_stream_* session (window
10s, hop 10s, top_k set to the tagger's class count so every
drained window carries a full score list), averages each class's
score across the drained windows, sorts descending, then applies
the request's threshold and top_k (0 keeps every class).
No tagCtx returns FailedPrecondition; a libparakeet.so missing the
sound_stream symbols returns Unimplemented. Every C call runs under
engineMu, and the stream is always freed, even when a feed or drain
call fails partway through.
Verified against a real ced-tiny-q8_0.gguf on the rooster.wav demo
clip: "Chicken, rooster" tops the list at score 0.91.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): cancel sound detection mid-feed, shrink the lock
SoundDetection now checks ctx before each 10 s feed slice (mirroring
driver.go's feedSlices) and returns Canceled if the caller gave up,
so a long clip can be interrupted instead of feeding to completion
regardless. The stream is still freed on every path, cancellation
included.
Also narrow engineMu to the C calls: the drained JSON document is
now decoded after the lock is released, splitting soundStreamScores
into a locked soundStreamDrain (opts, begin, feed, drain, free) and
an unlocked json.Unmarshal.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(parakeet-cpp): stream speaker and sound events during live transcription
Add two additive proto fields, LiveSpeakerSegment and LiveSoundEvent,
repeated on TranscriptLiveResponse. When a diarization or sound
companion model is loaded, AudioTranscriptionLive now runs a no-ASR
scene stream (parakeet_capi_scene_stream_begin) beside the ASR
streaming session, feeding it the same PCM slices and forwarding any
closed speaker or sound events alongside the matching ASR delta, or
on their own when a slice has no ASR output.
The scene stream is freed and reopened on a mid-stream Config reset,
flushed with is_last before the closing FinalResult, and degrades
gracefully (a warning, not an error) when begin or a later feed call
fails, so live transcription keeps working ASR-only. Existing live
behavior is unchanged when no companion is configured, and no scene
C call is made in that case.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): keep scene events off the ASR critical path in live
Emit each slice's ASR result right after the ASR feed, before the
scene feed for that slice runs, so a companion diarization/sound
model never adds scene compute latency in front of the delta or
<EOU> that drives realtime turn detection. Closed speakers/sounds go
out afterward as their own response, so a slice with both now
produces two responses, ASR first. The live feed log line now
reports ASR and scene wall time separately.
Re-check the diarization/sound contexts a scene stream was begun
with against the live contexts before every feed, under the same
lock: Free() can race between an ASR feed and the matching scene
feed and free the model the stream borrows. A mismatch now returns
without touching the C side. Freeing the stream itself stays
unconditional; the scene stream's destructor only releases its own
buffers and never touches the borrowed contexts.
Also recover a panicking stub inside the live test goroutine instead
of crashing the test binary, and reset the live decode-lag tracker on
a mid-stream config reset, matching what its own comment already
promised.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): surface live speaker and sound events
Carry the backend's closed speaker segments and sound events
(TranscriptLiveResponse fields 7/8) through LiveTranscriptionEvent
as LiveSpeakerSegment/LiveSoundEvent (nanoseconds mapped to
seconds), and forward them from the semantic_vad live path.
Each speaker segment emits
conversation.item.input_audio_transcription.segment with speaker,
start, end and empty text under the turn's item id. Each sound
event emits conversation.item.sound_detection with one tag
(label, score = peak, index) and the event's new optional
start/end seconds fields, omitted when unset so the existing
unary/windowed sound-detection path is unaffected.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): keep start/end on a zero-second transcription segment
ConversationItemInputAudioTranscriptionSegmentEvent.Start/End used
omitempty, so a speaker segment starting at 0.0s dropped its
"start" key. Nothing emitted this event before the live scene-event
path, so drop omitempty: the segment always carries real times.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(gallery): add parakeet-cpp diarization, CED and realtime scene models
Add gallery entries for the new parakeet-cpp capabilities: standalone
Nemotron-3-Diarization, the same paired with the Parakeet TDT+CTC
110M ASR model for speaker-attributed text, CED-Tiny and CED-Base
sound classifiers, and a realtime scene bundle combining the
streaming EOU ASR model with diarization and sound companions.
SHA256 taken from the Hub API; licenses from each model card
(openmdw-1.1 for Nemotron-3-Diarization, apache-2.0 for CED,
cc-by-4.0 for the Parakeet ASR models).
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: document parakeet-cpp diarization, sound detection and live scene events
Cover the new parakeet-cpp capabilities across the feature pages:
Nemotron-3-Diarization as a diarization backend (with and without
speaker text, the ignored speaker-count hints, the Sortformer
voice-like-sound quirk), CED as a sound classification backend, the
asr_model/diarization_model/sound_model/diarization_latency companion
options, and the realtime live speaker/sound events (event shapes,
the speech-turn-only limitation, and using this or
pipeline.sound_detection but not both).
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): correct the realtime-scene license and wording nits
parakeet-cpp-realtime-scene mistakenly copied cc-by-4.0 from the
existing realtime_eou_120m-v1 entry; the model card lists the NVIDIA
open model license instead. Switch to the gallery's usual spelling
for that license and keep the diarization/CED licenses called out in
the description.
Also: audio-diarization.md now says getting per-segment text needs
both an asr_model companion and include_text=true on the request, and
audio-to-text.md's option table reads "Use on" (a pairing the loader
does not enforce) instead of "Allowed on".
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): reject a companion role that duplicates the primary's
loadRoles let a companion option (asr_model:/diarization_model:/
sound_model:) assign into a role field the primary already occupied,
for example asr_model: on an already-ASR primary. The companion's
context silently overwrote ctxPtr/diarCtx/tagCtx, and Free() only
walks those three fields, so the original primary context was never
freed again.
Reject a companion whose role the primary already holds before its
GGUF is even loaded, freeing everything loadRoles opened so far, the
same way a wrong-kind companion is already rejected.
Also warn, rather than silently fall through, when
parakeet_capi_model_kind reports PARAKEET_MODEL_KIND_NONE for a
successfully loaded primary; the primary is still treated as ASR,
matching today's behavior.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): cap live scene sound score retention
sceneBegin started the live diarization/sound companion stream with
the C API's default sound options, whose top_k keeps 5 scores per
window forever until drained. The live scene path never drains sound
scores (only the offline SoundDetection RPC does, with its own fresh
stream), so this window queue on the C side grew for the whole
session's lifetime.
Set opts.Sound.TopK = 0 before starting the scene stream: this
disables score retention while leaving sound event detection (onset/
offset), which the live path actually consumes, unaffected.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): merge diarization segments per speaker, harden Diarize
mergeCloseSegments only compared neighbors in the single start-sorted
segment list, so two same-speaker segments never merged once another
speaker's turn fell between them (A, B, A): the short B segment broke
the adjacency the merge relied on. Group segments by speaker first,
merge within each speaker's own start-ordered run, then re-sort the
result by start so interleaved speakers come back out in timeline
order.
Also harden Diarize's entry points the same way streamFeedDoc/
sceneFeed already are: diarizeCall re-checks p.diarCtx (and, on the
include_text path, p.ctxPtr) under engineMu right before the C call,
so a Free() racing between Diarize's own checks and the lock can no
longer reach the C side with a freed context. When the include_text
call returns NULL, last_error is now read from both contexts and
whichever came back non-empty is reported, since either side of the
pairing can be the one that failed. A WAV decode failure is reported
as InvalidArgument instead of an unwrapped/untyped error.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): harden SoundDetection's engine checks
soundStreamDrain ran every C call under engineMu but never re-checked
p.tagCtx there, so a Free() racing between SoundDetection's own
tagCtx==0 check and this lock could still reach the C side with a
freed context. Re-check p.tagCtx under the lock and return
ModelNotLoaded when it was cleared, mirroring diarizeCall's own
re-check. A WAV decode failure is now reported as InvalidArgument
instead of an unwrapped/untyped error.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(parakeet-cpp): cover a mid-session scene feed failure
feedSlicesScene already degrades gracefully when a scene feed call
fails mid-session: it frees the broken stream and carries the ASR-only
session forward. Add a spec covering that path end to end: the scene
stream is freed exactly once, later audio slices still produce ASR
responses, and no speaker/sound events appear before or after the
failure.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: fix the parakeet-cpp companion role table and realtime scene docs
audio-to-text.md's companion option table read "Use on" with a note
that the loader did not enforce the pairing; it now rejects a
companion whose role duplicates the primary's, so restore the
"Allowed on" wording and describe the real enforcement.
openai-realtime.md's live speaker/sound section claimed a mid-stream
session.update resets the companion stream and that it flushes on
session close; neither happens, since the realtime core opens one
live stream (and so one scene stream) per speech turn and closes it
at that turn's commit, with no mid-stream Config in between. Document
that lifecycle instead, state precisely that start/end are seconds
from the start of the turn's own audio, and note that the diarization
model starts a fresh session every turn, so a speaker index is only
meaningful within one turn. The example sound tag ("Rooster", index
17) did not match any real CED label; index 17 in ced-tiny-q8_0.gguf
is "Baby laughter". Replaced with "Chicken, rooster" at its real
index, 99.
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(parakeet-cpp): use CED's real index for Chicken, rooster
The scene feed comment and the live test's canned document gave
"Chicken, rooster" index 365. In CED's AudioSet label list it is 99,
which is also what the realtime docs show.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): call the test event accessor
The scene-event tests range over a method instead of its returned slice.
Call the synchronized accessor so the OpenAI test package compiles.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(parakeet-cpp): pin parakeet.cpp master with sound events
mudler/parakeet.cpp#75 (sound events, scene stream, model kinds) and
#74 (the missing <algorithm> include that broke the image builds) are
on master now. Pin 6dea76a instead of the #75 PR head, and update the
header comment the bump bot reads.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(parakeet-cpp): pin parakeet.cpp with ced.cpp on main
parakeet.cpp #76 moved its ced.cpp submodule from the head of
localai-org/ced.cpp#3 (a branch-only commit) to ced.cpp main, where
#3 landed with an identical tree. Pin 623a968 so the image builds no
longer depend on that branch.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(transcription): carry speaker labels on words and streamed segments
A diarizing backend could label transcript segments, but two paths
dropped the label: TranscriptWord had no speaker field, so live
transcription words and word-level timestamps could not carry one, and
the stream=true transcript.text.done event left the speaker out of
its segments.
TranscriptWord gains an optional speaker (proto field 4, additive).
It flows through the live event and result mapping, the JSON word
output of the endpoint and the CLI, and transcript.text.done now
includes a segment's speaker when there is one. Empty labels are
omitted, so responses without diarization are unchanged.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
(cherry picked from commit
|
||
|
Ettore Di Giacinto
|
bdd4a73d05 |
feat(ui): edit failover chain targets with a dedicated field
Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
63221ee779 |
feat(config): add failover chain block to model configs
A chain is a model config with an ordered list of target models, probe, trip and recovery settings. Like an alias it has no backend. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d55474a149 |
feat(audio): list available TTS voices
Clients cannot discover the named voices that an installed TTS model accepts without consulting backend-specific documentation. Expose voice metadata through the audio API and let custom model configs declare their own catalog. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
419375baae |
fix(config): register original diffusers config field
Expose original_config_file through the model configuration metadata registry so registry completeness checks and generated configuration surfaces stay in sync. Assisted-by: Codex:gpt-5 [systematic-debugging] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
9bd7d17ff6 |
[model-config] feat: add environment variables support for backends (#10721)
* feat: add environment variables support for backends in model configurations - Add field to model configuration to pass environment variables to backend processes - Update backend options and model configuration handling - Add documentation for environment variables configuration with examples including CUDA_VISIBLE_DEVICES Assisted-by: qwen-agentworld-35b-a3b Signed-off-by: nold <nold42@pm.me> * fix(test): Test environment variables configuration parsing from YAML Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: nold <Nold360@users.noreply.github.com> --------- Signed-off-by: nold <nold42@pm.me> Signed-off-by: nold <Nold360@users.noreply.github.com> Co-authored-by: nold <nold42@pm.me> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Stefan Walcz
|
109244a76a |
[chat] feat: template.system_messages_after_first — merge or forward late system turns (#11906)
* feat(chat): template.system_messages_after_first — merge or forward late system turns Tokenizer chat templates such as Qwen3.8 / Qwen3.8-Flash-Next raise 'System message must be at the beginning' for system-role messages that appear after the leading system block, while agent frameworks (cogito tool selection and adjustment prompts) legitimately append system instructions mid-conversation. Every such request failed with a 500 (48 errors in one 10-task agent run). New per-model option template.system_messages_after_first: merge fold late system turns into the leading system message user forward them as user-role turns at their original position Default (unset) keeps the current pass-through behaviour. Fixes #11876 Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * docs(model-config): document template.system_messages_after_first Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(config/meta): register template.system_messages_after_first in the field registry TestAllFieldsHaveRegistryEntries requires every model-config field to have a registry entry. Adds the entry (templates section, select component) and the option list for the new field so the coverage gate passes. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> --------- Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
0761bd02c7 |
feat(chat): add end-to-end context compression (#11556)
* feat(config): add context compression policy Define the opt-in model configuration contract before the chat middleware consumes it. Document each policy field so later request handling does not invent a second schema.\n\nRefs #9534\n\nAssisted-by: Codex:gpt-5 * fix(config): register compression fields The model editor metadata gate rejects new config fields without descriptions and suitable controls. Register the compression policy so operators can edit its six fields safely. Assisted-by: Codex:gpt-5 [monitoring-prs] * feat(chat): compress long contexts Long conversations currently fail once they reach the model context window. The opt-in policy now summarizes complete older turns before primary inference and preserves the newest tool chains. Both OpenAI and MCP chat routes share the same transformation. Usage metadata and metrics expose each compression event. Refs #9534 Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
d10374f849 |
feat(router): make KNN a first-class classifier with a persisted, curated corpus (#10652)
* feat(router): make KNN a first-class classifier with a persisted, curated corpus
Add `classifier: knn` — similarity-weighted voting over labelled
example prompts. Unlike score/colbert it needs no classifier model:
label knowledge lives in a corpus seeded and curated through the
admin API, so routing decisions are deterministic, auditable, and
grounded in graded experience rather than a model's opinion.
Epistemic gate: corpus entries below knn.similarity_threshold cannot
vote; when none clears it the classifier activates no labels and the
router uses the fallback — a prompt unlike all labelled experience is
treated as undecidable, not guessed. Decisions record
nearest_similarity (also on fallback rows) so admins can see how far
the nearest labelled experience was; the Routing tab explains
out-of-corpus fallbacks and shows per-label corpus counts.
Persistence: one JSONL file per router under
<data path>/router-corpus (text, labels, vector, embedder
fingerprint). The file is the source of truth; the local-store index
is rebuilt from it at classifier build time and stays a pure
in-memory index. Entries recorded under a different embedding model
re-embed on load. Also corrects the docs' false claim that
local-store collections persist — the embedding cache never survived
restarts (and still doesn't); the corpus does.
Corpus input is API-only by design (entries may contain example user
content): POST /api/router/{name}/corpus seeds (labels validated
against declared policies, embedded server-side, indexed
immediately), GET .../corpus/stats inspects — label counts only,
entry texts are never returned by any surface — DELETE .../corpus
wipes. Admin-gated like the sibling router endpoints, and exposed as
MCP tools (seed_router_corpus / get_router_corpus_stats /
clear_router_corpus) in both the httpapi and inproc clients with
coverage-test route mappings.
Plumbing: VectorStore gains SearchK (top-K was hardcoded to 1);
local-store gets InsertBatch/Delete as optional fast paths;
RouterConfig gains a knn block (embedding_model, k,
similarity_threshold, vote_threshold, store_name) with meta-registry
fields; the classifier dropdown now offers knn and the
previously-missing colbert; embedding_cache is ignored (with a
warning) for knn — it IS an embedding-KNN lookup; the stale
/api/instructions intelligent-routing entry is rewritten (it
described a classifier that no longer exists); swagger regenerated.
Tests: KNN vote/gate specs with hand-computed vote shares, corpus
manager suite (restart reload without re-embedding, fingerprint
re-embed, dedupe, hostile store names), middleware specs (corpus
routing, gate fallback, config validation, cache-wrap refusal),
corpus endpoint specs pinning the texts-never-returned contract, MCP
catalog + route-mapping gates, and a Playwright spec for corpus
stats and the out-of-corpus decision detail.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): name consulted corpus neighbours in knn decisions
Every knn decision (decision log rows and the /api/router/decide
response) now carries neighbors: the K retrieved corpus entries by
descending similarity - including ones below the epistemic gate, which
is what makes fallback decisions diagnosable - each as {id, similarity,
labels}. The id is the entry's content hash (first 8 bytes of the
SHA-256 of its text, hex): stable across reseeds and re-embeds, and
text-free, so an external platform that seeded the corpus can recompute
text->id on its own copy and bucket decisions by corpus region (per-
region reliability accounting) without corpus text ever leaving the
server. A corrupt index payload surfaces as an id-less neighbour at a
real similarity instead of disappearing.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* refactor(router): deduplicate knn plumbing and cut corpus hot-path waste
Post-review cleanup of the knn-first-class-router branch; no behaviour
changes on the API surface.
Reuse/altitude:
- RouterKNNConfig.ResolvedStoreName is now the single source of the
router-corpus-<name> default (was hand-derived in four files).
- corpus.ResolveKNNRouter + corpus.Seed carry the shared model
resolution and seed validation; the REST endpoints and the assistant
MCP client are thin transport adapters over them, with sentinel
errors mapped to HTTP statuses at the echo boundary.
- middleware.NewClassifierDeps assembles the classifier dependency set
once for all five entry points (OpenAI, Anthropic, realtime, decide,
corpus) instead of five hand-copied literals.
- router.AllClassifiers feeds both the status endpoint and the
unknown-classifier error, ending the classifier-list drift.
- Per-classifier requirements moved out of validateRouterPolicies into
their buildClassifier arms; the knn arm owns its embedding_cache
opt-out instead of a name-check in the shared wrap tail.
- adminOnly replaces four inline copies of the admin gate in the
middleware routes.
- localVectorStore.Search delegates to SearchK (identical traces).
Efficiency:
- Manager.Add embeds outside the manager mutex and appends to the
JSONL file (O(new) instead of O(corpus) rewrite); a torn tail from a
crash mid-append is tolerated on read and repaired on next write.
- Stats memoises per store keyed on the file's stat fingerprint and no
longer takes the manager mutex, so the 5s status poll stops parsing
vector-laden JSONL and stops blocking behind seeds.
- KNN Classify decodes each neighbour payload once (was twice) and
builds refs and votes in a single pass with one fallback return.
- Corpus file writes fsync before rename/close.
- The corpus manager is built eagerly in newApplication (sync.Once
dropped); test helper dead branch removed.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): bind knn corpus vectors to an embedder fingerprint and fail closed on mismatch
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(mcp): align corpus tool prompts and the mutating-tool safety list
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(proto,backend): report embedding shape from the llama-cpp backend
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): Go-side pooling — mean/last/decayed_mean with half-life
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): accept chat messages[] and per-request pooling on /v1/embeddings
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(middleware): name the failing fields when post-merge validation 400s
An intermittent post-merge validation failure surfaced as an opaque 400
during integration (pooling scheme mismatch that no client had sent).
Log the model, the request's pooling override, and the merged config's
pooling fields at the failure point so the next occurrence identifies
whether the request or the stored config carried the bad value.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(embeddings): scheme override must not inherit the config's half-life
A model config defaulting to decayed_mean pooling carries
pooling_half_life_tokens; a request overriding the scheme to mean/last
without its own half-life inherited that value, and post-merge
validation rejected the pair the server itself had assembled. Zero the
inherited half-life when the overridden scheme is not decayed_mean; a
request that explicitly pairs a half-life with a non-decayed scheme
still 400s.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix embedding pooling validation and router bounds
Declare backend embedding layouts and reject incompatible pooling modes. Reset local-store dimensions after a full clear, validate KNN thresholds, and add real backend and store integration coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: run local-store integration tests
Build and install the local-store backend in the Linux test job, then run the existing store integration suite so new specs are discovered automatically.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
|
||
|
Richard Palethorpe
|
5c63969760 |
fix: Show MCP connection errors in the UI (#11495)
* fix(mcp): surface configured server failures Keep model-configured MCP servers visible when discovery or connection setup fails, propagate status through distributed discovery, and let the Chat UI show actionable errors while retrying unavailable servers. Add model-editor metadata for remote and stdio configuration and document the expected format, deployment networking boundary, and alternate MCP scopes. Assisted-by: Codex:gpt-5 Ordino golangci-lint Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(compose): match CUDA development image Configure the API image with the cublas, CUDA 13, auth-tagged build settings used by the local development Makefile invocation, including the 24-way Docker build. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> * revert: keep host build settings out of compose The CUDA development deployment is managed from ~/docker/localai, not the repository example Compose file. Restore the generic example and keep machine-specific build settings in the host deployment. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(docker): exclude local agent artifacts Keep Claude worktrees and locally installed verification tools out of the Docker build context. These host-only directories added roughly 1.9 GB to every root image build. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
06ff56e674 |
feat(pii): restore request-scoped pseudonyms (#11272)
* feat(pii): restore request-scoped pseudonyms Replace masked request values with unique per-request tokens when response restoration is enabled, then restore them across JSON and SSE write boundaries. Document the opt-in model setting and expose it in config metadata.\n\nAssisted-by: Codex:gpt-5 * fix(pii): wrap reversible redaction tokens Use configurable token delimiters to avoid restoring ordinary model text that happens to match an internal identifier. Rename the option and document the confidentiality tradeoff. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
49ef40a187 |
feat(classifier/VAD): support voice control on low power devices (#10804)
* feat(llama-cpp): route Score through the slot loop Score previously bypassed the slot loop with a direct llama_decode: a conflict guard aborted the whole process if scoring raced generation, the config validator had to reject score alongside chat/completion/embeddings, and every candidate re-decoded the full shared prompt. Add SERVER_TASK_TYPE_SCORE to the (patched) upstream server so score tasks are scheduled like any other slot work: generation and scoring serialize naturally, the shared prompt is decoded once per call, and the slot's prompt cache carries the conversation prefix across calls. Context checkpoints at the score boundary and at the cache-divergence point keep SWA/hybrid/recurrent models (e.g. LFM2.5) from re-prefilling the whole prompt per candidate: warm-turn scoring on a 6-option set drops from ~8s to ~0.5s on a desktop CPU. The conflict guard and the validation split are removed; declaring score with generation usecases on one config is now supported and shares the slot cache. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier wire types and pipeline config Wire types and YAML config for realtime classifier mode: sessions carry a localai_classifier extension (options with canned replies/tool calls, softmax threshold, normalization, history trimming, fallback modes, and a deterministic wake-word address gate), mirrored by pipeline.classifier in the model YAML and surfaced in the config-meta registry. The localai.classifier.result server event reports the full score distribution per turn. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier response flow Classifier-mode responses: instead of autoregressive generation, each user turn is prefill-scored against the option list (router.ScoreClassifier prompt/candidate shapes over the Score primitive) and the winning option's canned reply and tool call are emitted through the existing response machinery. Below-threshold turns take the configured fallback (none / canned reply / generate); empty transcripts and unaddressed turns (wake word not mentioned) skip scoring entirely. The scoring probe defaults to the latest user message only — small scorers echo canned replies from prior turns back as the top option otherwise. Built for hardware that can afford prompt processing but not decode: with slot-based Score the option list stays KV-cached across turns, so a turn costs roughly one forward pass over the new words. session_update_error events now carry the validation cause instead of a generic message. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): bound the VAD tick's scan window and buffer retention The VAD tick loop re-scanned the entire input buffer every 300ms and only trimmed it on zero-segment ticks or commits. Audio that keeps producing segments without a committing pause (steady noise a mic pipeline lets through, music, continuous speech) grew the buffer toward the 100MB cap with each tick rescanning all of it — O(n^2), measured at ~3.3ms of silero per buffered second: past ~90s retained, ticks run back to back and pin ~4 cores until the stream stops. Silero's recurrent state only carries a few hundred ms of context, so rescanning old audio buys nothing. Clip the slice handed to the VAD to the largest silence the commit test can need to measure (server_vad silence window or the semantic eagerness fallback) plus a warm-up margin, and rebase the returned segment times so every downstream consumer keeps whole-buffer coordinates. An open turn whose clipped window is all silence now commits (the silence outran the window) instead of being discarded as no-speech. Independently, retain at most 90s of raw buffer, rebasing the live-feed and EOU cursors on trim — this also bounds the previously unbounded VAD-error path. Turn boundaries are otherwise unchanged: no forced commits, no new coordinator states. pipeline.turn_detection.vad_window_sec can widen the scan window; values below the automatic floor are ignored. The tick body is extracted into vadTick so specs can drive turn detection synchronously (same shape as classifySoundWindow); the babble reproduction that pinned 4 cores now plateaus under 10% of one core. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(backend): let per-model threads override the global default ModelOptions overrode a set per-model threads value with the app-level --threads whenever the latter was non-zero — and WithThreads defaults it to the physical core count, so it always was. The YAML threads: knob has been dead config: a tiny VAD model could never opt down from the global pool size. SetDefaults already fills an unset per-model value from the app config, which is the intended precedence; resolve threads through a helper that honors it (explicit threads: 0 still means unset). Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(gallery): single-thread the silero VAD Silero is a ~2MB recurrent model with no exploitable graph parallelism: measured per-call latency is identical at 1 and 10 ORT threads, while every extra pool thread just spin-waits between the realtime loop's frequent tiny inferences. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * docs(realtime): classifier mode, VAD scan window, threads precedence Document the realtime classifier mode (options, threshold guidance, wake-word address gate, empty-transcript handling), the VAD scan window and 90s buffer retention (pipeline.turn_detection.vad_window_sec), the per-model threads precedence, and the M3 classifier note in the realtime state-machine design doc. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * perf(llama-cpp): score all candidates in one batched decode One scoring call is now a single SERVER_TASK_TYPE_SCORE task: the slot decodes the shared prefix (prompt + longest common candidate token prefix) once, then forks one sequence per candidate off it (metadata-only for the unified KV cache, copy-on-write for recurrent state) and decodes every candidate's unique tail in one llama_decode. Previously each candidate was its own task that restored the boundary checkpoint and re-decoded its full tail sequentially, paying per-candidate task and decode overhead. The context reserves SERVER_SCORE_FORK_SEQS extra sequence ids (and recurrent-state cells) beyond the parallel slots via the new common_params::n_seq_score_forks. Forking requires the unified KV cache (already this backend's default) since per-sequence streams would shrink n_ctx_seq; an explicit kv_unified:false disables forking and Score calls that need it fail cleanly. Candidates beyond the fork/output budget decode in successive chunks. Wire contract and scores are unchanged: per-token logprobs are stitched from the shared region and the forked tails. Verified bitwise deterministic call-to-call and independent of candidate order (no cross-fork leakage via equal-length candidate swap); ranking matches the per-candidate implementation on the drone battery (winner softmax 0.99996 vs 0.99997), and >16-candidate chunking, prefix-of-another and empty candidates all pass. Measured on a desktop CPU: warm /api/score calls 0.52s -> 0.23s; warm realtime classifier turns 196-303ms. The 9-candidate drone turn decodes ~17 unique tail tokens in one batch instead of nine sequential ~220ms checkpoint-restore tasks. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): gate scoring capacity by model usecase Reserve llama.cpp scoring slots only for models that explicitly declare the score usecase, while allowing score to coexist with chat and completion. Reject incompatible unified-KV settings and classifier activation on models without scoring capacity. Propagate application defaults when resolving realtime and preload pipeline stages so unset thread counts are resolved consistently without overriding explicit model settings. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): honor APT mirrors in the prebuilt llama-cpp compile step The builder-prebuilt path installs gcc-14 with apt directly and ignored the APT_MIRROR/APT_PORTS_MIRROR build args the from-source path already honors, so an ubuntu mirror outage broke every arm64 backend build. Pass the args into the stage and run apt-mirror.sh (already in the build context via COPY . /LocalAI) before the apt step. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier argument slots via constrained completion Hybrid classify-then-complete: a classifier option's canned tool call can declare typed argument slots (number | enum | string, with defaults and prompt hints) referenced as "{{name}}" in the arguments template. When the option wins, the slots are filled by a short grammar-constrained completion that continues the exact scoring prompt — rendered by the same cached ScoreClassifier, so the llama.cpp prompt cache is already warm — with the chosen route JSON re-opened at the first slot field. A GBNF grammar pins the field skeleton and frees only the values; temperature 0, a couple dozen tokens at most (~300ms on a desktop CPU for two slots). Slot declarations and hints ride the option descriptions in the shared system prompt, informing scoring and the fill alike at no per-turn token cost. The localai.classifier.result event carries the final arguments and a fill_latency_ms. On inference failure the slots' defaults apply; a slot without a default fails the response (or falls through with fallback.mode: generate). Slot filling requires completion alongside score in the scoring model's known_usecases. Verified end-to-end on the Pi drone demo: "fly forward three meters" in distance mode classifies forward and infers {"distance": 3, "units": "meters"} in ~310ms, and the drone flies exactly 3 units. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): splice filled slot values into classifier replies A classifier option's spoken reply can now reference its tool's argument slots ("Going forward {{distance}} {{units}}."): the values inferred by the slot-fill completion — or the recovery defaults — are spliced into the reply as plain text before it is emitted, so what the assistant says confirms what it actually inferred. Placeholders without a value stay literal, and options without slots are untouched. FillToolArguments now returns the raw slot values alongside the spliced arguments JSON to make the reply templating possible. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): harden classifier slot completion Reserve context for constrained slot filling, size completions from their encoded output, and encode enum grammar literals as valid JSON. Reject empty enum values and cover the failure modes with regression tests. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): prewarm the classifier scoring prompt on registration Swapping a session's classifier option list (a voice-switched command mode, for instance) made the next turns pay a full re-prefill of the new option-list prompt — measured 2.4s vs 0.3s warm on a desktop CPU, and worse: on hybrid-memory models like LFM2.5, whose state cannot be partially rewound (llama.cpp can only restore checkpoints), *every* probe change re-prefilled from scratch whenever the last checkpoint missed the probe boundary, so even same-list turns intermittently cost full prefills. Registering an option list (pipeline seed or session.update) now fires a best-effort background prewarm: two throwaway scores with distinct probes. The first prefills the new option-list prompt; the second, diverging exactly where per-turn probe text starts, plants the backend's rewind point (KV checkpoint) at the stable-prefix boundary that every real turn reuses. The prewarm hides behind the canned mode-switch reply — by the time it finishes speaking, the cache is warm. Idempotent per option set, detached from the registering request's lifetime. Measured on the drone demo (LFM2.5-1.2B, desktop CPU): first turn after a mode switch 2374ms -> 340ms; intermittent same-list full prefills (1.3-2.1s) all -> under 0.5s. For clients that swap lists frequently, options: [parallel:2] on the scoring model additionally keeps one slot per list via prefix-similarity routing (+26MB RSS, unified KV). Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * perf(llama-cpp): checkpoint scoring at the caller-declared stable prefix Hybrid-memory models (LFM2.5 shortconv, Qwen3.5 deltanet — where new small models are headed) cannot rewind their state, so any prompt-cache reuse that needs a rewind falls back to a full re-prefill. For classifier scoring that meant every probe change re-processed the whole option-list prompt: the server's checkpoints were placed reactively (at wherever the previous task happened to diverge), so a checkpoint past the next divergence was erased rather than restored — measured as intermittent 2-10s turns on prompts with a 95%+ common prefix. The classifier now computes the probe-invariant prompt prefix once (the byte-wise common prefix of two synthetic probe renders) and declares its length with every Score request; the server maps it to a token boundary and forces a KV checkpoint exactly there on each score prefill. That checkpoint sits at or before every future divergence under the same option list, so it always survives and always restores — repeat scoring costs probe+candidates regardless of how the probe changes. Also: - prewarm reruns on every option-list registration instead of memoizing per list: with boundary checkpoints a redundant rewarm costs two probe-sized decodes, while skipping one after a slot eviction (three lists sharing fewer slots evict in LRU cascades) silently moves a full re-prefill onto the user's next turn - new llama.cpp backend option rs_seq:N exposes bounded recurrent-state rollback outside speculative decoding; measured impractical for deltanet-scale states (65GB for 64 snapshots on Qwen3.5-4B) but cheap insurance for small-state models - docs: the multi-list recipe (parallel:N + sps:0.5 — the default slot similarity threshold funnels distinct lists onto one slot) Measured on the drone demo (LFM2.5-1.2B scorer, desktop CPU), steady state: every turn 285-421ms including mode switches, vs 2.4s post-switch and intermittent 1.3-2.9s re-prefills before. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): align classifier cache guidance Document the single-score prewarm behavior and clean the vendored score patch formatting. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(llama-cpp): guard score task for fork backends TurboQuant and Bonsai reuse the primary gRPC server against llama.cpp forks that do not carry LocalAI's slot-based Score patches. Compile the Score integration only for the patched primary backend and return UNIMPLEMENTED from fork builds instead of referencing absent task types and common_params fields. Assisted-by: Codex:gpt-5 [gh] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(dev): generate gRPC code before commit lint The coverage phase regenerates ignored protobuf bindings, but lint runs first and can fail against missing or stale output. Generate the pinned bindings before lint so the gate always type-checks the current schema. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
walcz-de
|
4b4faa4ac7 |
feat(cloud-proxy): optional Anthropic prompt-cache breakpoints in translate mode (#11158)
The Anthropic translate provider builds the upstream request from scratch and
never emitted cache_control, so prompt caching was impossible for OpenAI-format
clients routed through cloud-proxy — even though the entire system prompt + tools
prefix is re-sent on every agentic turn.
Add an opt-in cache_prompt flag (ProxyOptions.cache_prompt; model YAML
proxy.cache_prompt: true). On a translate+anthropic model, buildAnthropicRequest
injects cache_control:{type:ephemeral} on the stable prefix — the system block,
the last tool, and the last message block (at most 3 of Anthropic's 4 allowed
breakpoints). Anthropic then serves the repeated prefix at the cache-read rate
(0.1x input) on subsequent calls, cutting cost on multi-turn/agentic workloads.
No effect in passthrough mode, for non-Anthropic providers, or when unset.
System is widened to any so it can carry the block form required to attach
cache_control, while still marshalling as a bare string when caching is off.
Adds a unit test asserting exactly three breakpoints when on and none when off,
and documents the option in docs/content/operations/cloud-proxy.md.
Assisted-by: Claude:opus-4.8
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
|
||
|
LocalAI [bot]andEttore Di Giacinto
|
bcc41219f7 |
feat: materialize Hugging Face model artifacts (#10825)
* feat(config): add model artifact source contract Assisted-by: Codex:GPT-5 [Codex] * feat(downloader): add authenticated raw-byte progress Assisted-by: Codex:GPT-5 [Codex] * feat(huggingface): resolve immutable snapshot manifests Assisted-by: Codex:GPT-5 [Codex] * feat(models): add artifact storage primitives Assisted-by: Codex:GPT-5 [Codex] * feat(models): materialize pinned Hugging Face snapshots Assisted-by: Codex:GPT-5 [Codex] * feat(models): bind managed snapshots at runtime Assisted-by: Codex:GPT-5 [Codex] * feat(gallery): materialize model artifacts during install Assisted-by: Codex:GPT-5 [Codex] * feat(gallery): declare managed Hugging Face artifacts Assisted-by: Codex:GPT-5 [Codex] * feat(models): preload managed model artifacts Assisted-by: Codex:GPT-5 [Codex] * fix(gallery): retain shared artifact caches on delete Assisted-by: Codex:GPT-5 [Codex] * feat(models): report artifact acquisition progress Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): load managed models from ModelFile Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): load staged speech model snapshots Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): use staged snapshots in engine backends Assisted-by: Codex:GPT-5 [Codex] * test(distributed): cover staged artifact snapshots Assisted-by: Codex:GPT-5 [Codex] * docs: explain managed model artifacts Assisted-by: Codex:GPT-5 [Codex] * docs: add product design context Assisted-by: Codex:GPT-5 [Codex] * feat(ui): show model artifact download progress Assisted-by: Codex:GPT-5 [Codex] * Eagerly materialize Hugging Face artifacts Materialize HF-backed model references as managed GGUF artifacts during load, with lazy download retained only as fallback. Assisted-by: Codex:GPT-5 [shell] * Refactor HF downloads through a shared executor Assisted-by: Codex:GPT-5 [shell] * drop Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
4056283aa4 |
[voice] feat: add managed voice cloning profiles (#10799)
* feat(ui): add voice library workflow Give administrators a production-ready flow to record or upload consented reference audio, manage reusable profiles, inspect API usage, discover compatible models, and hand a saved voice directly to text-to-speech. Assisted-by: Codex:gpt-5 * feat(voice): add managed voice cloning profiles Make reusable reference voices manageable through the admin API instead of requiring model-directory and YAML edits. Discover compatible installed and gallery models from server-side backend capabilities, retain explicit model configuration controls, and stage saved references for supported backends. Expose profile management through REST and MCP, document backend-specific behavior, and cover the workflow from profile creation through real Qwen3-TTS synthesis. Harden the agent-job HTTP test against completion racing cancellation. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
b00422e45f |
feat(backends): add LongCat video and avatar generation (#10792)
* feat(backends): add LongCat video and avatar generation Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web] * refactor(config): declare model I/O modalities Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook. Assisted-by: Codex:GPT-5 [web] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
eb32cd9073 |
feat(realtime): eager blocking pipeline warm-up + /backend/load API (#10662)
Realtime sessions previously lazy-loaded each pipeline sub-model (VAD,
transcription, LLM, TTS) on first use, so every cold session paid a
per-request model-load stall and load errors only surfaced mid-stream.
Warm the whole pipeline eagerly and blockingly at session start
(including the voice-gate speaker-recognition model, which an enforced
gate blocks each utterance on; compaction's summary_model stays lazy
since it only runs off the response path):
- Add backend.PreloadModel / PreloadModelByName as the single load path
for every modality (no transcription special-case; backend-omitted
configs are deprecated).
- The realtime session blocks on Model.Warmup and returns a
model_load_error to the client if any stage fails to load;
updateSession warms in the background. Opt out per pipeline with
pipeline.disable_warmup, exposed as a UI toggle via the
config-metadata registry.
Add a LocalAI-native POST /backend/load (and /v1/backend/load) that
pre-loads a model -- expanding realtime pipelines into their sub-models
-- as the inverse of /backend/shutdown. There is one preload engine
(backend.PreloadStages): the realtime Warmup methods, /backend/load and
the --load-to-memory startup flag all use it, so --load-to-memory now
also expands pipeline models and records load-failure traces. Pipeline
sub-model alias resolution is likewise shared
(ModelConfigLoader.LoadResolvedModelConfig). Surface the endpoint
everywhere an admin manages models:
- MCP admin tool load_model (httpapi + inproc clients, safety/catalog
prompts, catalog/dispatch tests).
- "Load into memory" action in the React models UI.
- Swagger regenerated; docs moved to the general backend-monitor page
since it is not realtime-specific.
Fix a Traces UI crash ("json: unsupported value: -Inf"): audio-snippet
RMS/peak now floor at a finite dBFS, and backend-trace data is sanitized
to drop non-finite floats before marshaling. The sanitizer is
copy-on-write -- it runs on every RecordBackendTrace, so containers are
only re-allocated on the paths that actually changed.
Migrate core/http/openresponses_test.go onto the prebuilt mock-backend
the rest of the http suite already uses -- it was the last spec still
pointing at a real HuggingFace model, so it 404'd wherever no vision
backend was built -- and fix its item_reference specs to send the
spec's "id" field instead of "item_id", which the handler never
accepted.
Assisted-by: Claude:claude-opus-4-8 Claude Code
Signed-off-by: Richard Palethorpe <io@richiejp.com>
|
||
|
Richard Palethorpe
|
5d0c43ec6e |
feat(realtime): Semantic VAD EOU token (#10444)
* feat(realtime): EOU-driven semantic_vad turn detection Add a `semantic_vad` turn-detection mode to the realtime API that feeds the transcription model live and decides "the user finished speaking" from the `<EOU>` end-of-utterance token rather than from silence alone. When EOU fires the turn commits immediately (~0.3s); otherwise it falls back to an eagerness-scaled silence threshold (low/med/high = 8/4/2s). Plumbing, bottom to top: - proto: `AudioTranscriptionLive` bidirectional RPC (config-first oneof, mono float PCM @16k, ready-ack / Unimplemented degrade signal) plus `TranscriptResult.eou` for the unary retranscribe gate. - pkg/grpc: client/server/base/embed scaffolding for the bidi stream, modeled on AudioTransformStream; release stream conns on terminal Recv. - parakeet-cpp: live transcription RPC with per-C-call engine locking (one live stream per turn, finalize+free at commit); bump parakeet.cpp to ABI v5 — incremental StreamingMel (no more quadratic per-feed mel recompute that delayed EOU on long turns) and the <EOU>/<EOB> split; strip the literal <EOU>/<EOB> from offline text and set Eou. - core/backend: LiveTranscriptionSession wrapper + pipeline `turn_detection:` config block (type/eagerness/retranscribe). - realtime: semantic_vad integration — live input captions streamed as transcription deltas while the user speaks, EOU-immediate commit with eagerness fallback, optional retranscribe gate (batch re-decode must also end in <EOU> to confirm), clause synthesis off the LLM token callback, and per-turn live-transcription / model_load telemetry. - UI: show the realtime pipeline components as a vertical list. Docs and tests included; opt-in via the pipeline YAML or per-session `session.update`. Non-streaming STT backends degrade to silence-only. Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Write] [Bash] Assisted-by: Claude Code:claude-fable-5 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): explicit formally-verified state machines + parakeet streaming driver The realtime API had several implicit state machines whose state was inferred from scattered booleans, channels, and five separate mutexes, leaving illegal/inconsistent states reachable. Make them explicit and keep the implementation in step with a formal design; rework the parakeet streaming backend along the same lines. Realtime state machines (M1-M5). Each is a sealed sum-type State/Event/Effect with a total, pure Next(state,event)->(state,[]effect) behind a single-writer Coordinator: M1 conncoord connection lifecycle: VAD toggle + once-only teardown (replaces vadServerStarted + a `done` channel closed from two sites). M2 turncoord turn detection: collapses speechStarted and the live-stream "turn open" flag into one state, so discardTurn can no longer desync them and suppress the next onset. M3 respcoord response coordination: serializes the dual-writer start/cancel so at most one response is live; one response.done per response.create. M4 compactcoord conversation compaction: single-flight (replaces the `compacting atomic.Bool` CAS). M5 ttscoord TTS pipeline: open->closing->closed, idempotent wait(), rejects enqueue-after-close (was a silent drop). The Coordinator/Sink/Next plumbing — only the sealed types and Next differed per machine — is extracted once into core/http/endpoints/openai/coordinator as a generic Coordinator[S,E,F]; each machine keeps its public API via type aliases, so no sink, call-site, or test moved. Hierarchy. session_lifecycle.fizz models M1 as the parent region with its children (M2/M3/M4) as one statechart and asserts ChildrenDieWithParent (conn torn => all children terminal, none start after teardown). respcoord and compactcoord gain an absorbing Terminated state + Shutdown event; conncoord's teardown drives the children terminal. This closes a compaction teardown gap: a fire-and-forget compaction could outlive a torn session — compactionSink now takes a session-scoped cancellable context + WaitGroup and joins the in-flight summarize+evict on shutdown. Formal verification. formal-verification/ holds one authoritative FizzBee spec per machine plus the composition spec, each with an always-assertion and a documented one-line edit that makes the checker fail (verified non-vacuous). scripts/realtime-conformance.sh is fail-closed: all Go conformance suites under -race AND a model-check of every .fizz spec; a missing FizzBee is a hard error (only the loud REALTIME_CONFORMANCE_SKIP_FIZZBEE=1 bypasses it, never in CI). FizzBee is pinned by sha256 and installed via scripts/install-fizzbee.sh into .tools/ (gitignored). Wired as make test-realtime-conformance, a CI workflow, and a pre-commit path filter. Go conformance tests are Ginkgo/Gomega (per the repo's forbidigo lint): transition tables + fixed-seed property walks + concurrent/-race specs, no rapid dependency. Design map: docs/design/realtime-state-machines.md. Parakeet streaming backend. The same treatment applied to the parakeet-cpp streaming paths: - AudioTranscriptionStream returns codes.Unimplemented for non-streaming models instead of decoding offline and emitting it as one delta + final. A client that asked for streaming learns the model cannot stream rather than receiving a batch result shaped like a stream. New grpcerrors.StreamTranscriptionUnsupported carries that signal; the HTTP /v1/audio/transcriptions stream path surfaces it as an SSE error event. Mirrors AudioTranscriptionLive, which already did this. - utteranceBoundary (boundary.go): a single definition of the end-of-utterance latch, replacing three open-coded finalEou toggles. Modelled as a two-valued type so illegal states are unrepresentable. - Shared decode driver (driver.go): streamFeedResult (one per-feed event) + feedChunk (hides the ABI v4 JSON vs text-only split) + feedSlices + flushTail. The feed loop is written once. - AudioTranscriptionLive becomes a bidi adapter: it streams the per-feed {delta,eou,eob,words} the realtime turn detector consumes and a terminal FinalResult carrying only Text. Segments/duration/eou are offline-only and no longer produced (nor read) on the live path; liveTraceState drops the terminal eou and keeps the per-feed eou_events count. - AudioTranscriptionStream + streamJSON merge into one driver-based function; streamSegmenter is generalized to the unified event with a text-only fallback that preserves the legacy (no-words) library's per-utterance segmentation. Verified: build/vet/gofumpt clean, golangci-lint 0 issues, all coordinator and parakeet packages under -race, the fail-closed conformance gate green, and make test-realtime (12 e2e WS+WebRTC). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
fdf475ec5f |
feat(realtime): conversation compaction (summarize-then-drop) + OpenAI item.delete/truncate/clear (#10446)
* feat(realtime): add pipeline.compaction config + resolution Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(realtime): extract itemID helper, reuse in item.retrieve Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): drop duplicate Ginkgo bootstrap, fold specs into openai suite Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): implement conversation.item.delete Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): implement input_audio_buffer.clear Add a handler for the input_audio_buffer.clear client event that discards a partially-captured utterance (raw PCM + buffered Opus frames) via a unit-tested clearInputAudio helper, then acks with input_audio_buffer.cleared. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): implement conversation.item.truncate (text) Clears both .Text and .Transcript of the assistant content part at contentIndex so barge-in truncation also works for audio turns whose spoken words live in .Transcript. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): add Conversation.Memory + pair-safe compactionCut Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): compactionCut returns 0 for keep<=0 (no-cap sentinel, avoids panic) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(realtime): gofmt compaction test helper closures Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): inject rolling memory into the prompt + summary builders Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): server-side summarize-then-drop compactor Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): unit-test prefixMatches eviction-safety predicate Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): resolve summarizer model + schedule compaction per turn Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): document conversation compaction + new item events Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): resolve summary model inside compaction goroutine (lazy, off-path) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(realtime): reuse reasoning.ExtractReasoningComplete for summary stripping Replace the bespoke <think> regex in the compactor with the shared pkg/reasoning extractor (via spokenReasoningConfig), matching the rest of the realtime path and covering all reasoning tag families, not just <think>. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): register pipeline.compaction fields in meta registry TestAllFieldsHaveRegistryEntries requires every ModelConfig field to have a UI/meta registry entry; add the four pipeline.compaction.* leaves so they render with proper labels/descriptions instead of the reflection fallback. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
600dafd20b |
feat(ced): sound-event classification backend (CED audio tagger) (#10425)
* feat(ced): sketch sound-classification backend (CED audio tagger) Wires ced.cpp (CED, 527-class AudioSet sound-event tagger; baby cry, footsteps, glass, alarms, dog bark) into LocalAI as a Go/purego backend. SKETCH (backend skeleton real; core REST wiring + CI/gallery is a checklist in DESIGN.md): - backend/backend.proto: new SoundDetection rpc + SoundClass messages (run `make protogen-go` to regenerate pkg/grpc/proto). - backend/go/ced: main.go (purego dlopen libced.so + ced_capi.h), goced.go (Ced gRPC backend: Load + SoundDetection), Makefile (clone-at-pin CED_VERSION, ggml static-PIC shared build), run.sh, package.sh, .gitignore. - DESIGN.md: REST /v1/audio/classification wiring (handler/route/capability registration checklist), gallery/index + CI registration, and a scoping note for the realtime/websocket live-recognition path (sliding-window classify over the existing ws transport + voicegate; the ced C-API per-PCM entry point is already window-friendly). Backend code does not compile until protogen-go regenerates the pb types and a libced.so is built (Makefile clones+builds it). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): REST /v1/audio/classification endpoint + capability registration Wires the ced sound-event classification backend (AudioSet audio tagger) end to end through the REST surface, mirroring the transcription path. - Handler: core/http/endpoints/openai/sound_classification.go parses the multipart audio upload, temp-files it, resolves the model config and calls the SoundDetection RPC; returns {model, detections[]} JSON. - Backend wrapper: core/backend/sound_classification.go (ModelSoundDetection) loads the model and normalizes the proto response into schema types. - Schema: core/schema/sound_classification.go (SoundClassificationResult). - gRPC layer: SoundDetection wired through the LocalAI wrapper (interface, Backend client, Client, embed, server, base default) so the loader-typed client exposes the RPC; proto regenerated via make protogen-go. - Route: POST /v1/audio/classification (+ /audio/classification alias) with the audio/multipart default-model middleware in routes/openai.go. - Capability surfaces: swagger @Tags/@Router on the handler; FLAG_SOUND_ CLASSIFICATION usecase flag + UsecaseSoundClassification + UsecaseInfoMap + GuessUsecases + ModalityGroups + GetAllModelConfigUsecases; meta usecase option; /api/instructions audio area updated; auth RouteFeatureRegistry + FeatureAudioClassification (APIFeatures, default ON) + FeatureMetas; UI usecaseFilters, capabilities.js CAP_SOUND_CLASSIFICATION, Models.jsx filter + i18n; docs page features/audio-classification.md + whats-new + crosslink. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): realtime sound-event detection over the websocket API When a realtime pipeline configures a sound-classification model, each VAD-committed utterance (the same window the transcription path produces) is also run through the CED sound-event classifier and the scored AudioSet tags are emitted as a new server event. No new backend rpc is needed: the SoundDetection gRPC method already exists on this branch. - config: add Pipeline.SoundDetection (yaml/json sound_detection,omitempty) beside Transcription/VAD. - realtime: add Model.SoundDetection(ctx, audio, topK, threshold) to the ModelInterface; implement it on wrappedModel and transcriptOnlyModel by calling backend.ModelSoundDetection with the session's sound-classification model config (mirrors how Transcribe dispatches). Load the optional config in newModel / newTranscriptionOnlyModel; nil config keeps it additive. - types: add ConversationItemSoundDetectionEvent (item_id, content_index, detections[]{label,score,index}) with type conversation.item.sound_detection, its ServerEventType constant and MarshalJSON, mirroring the transcription completed event. - realtime: add emitSoundDetection (unary path: classify the committed window, build the event, t.SendEvent) and wire it at the utterance-commit hook right after emitTranscription; gated on session.SoundDetectionEnabled (resolved from Pipeline.SoundDetection at session setup, defaults top_k=5, threshold=0). Its error is logged via xlog but never aborts the turn. - test: Ginkgo specs for emitSoundDetection (tags emitted, empty detections, classifier error) plus a SoundDetection method on the fakeModel double. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ced): implement SoundDetection in nodes backend test doubles The SoundDetection method added to the grpc backend interface left two test doubles (fakeBackendClient, fakeGRPCBackend) incomplete, so core/services/nodes failed to compile under `go vet`/`go test` (go build missed it: the doubles live in _test.go). Add the method to both, mirroring their existing Detect mock. Repairs CI for the nodes package. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): decouple realtime sound detection from VAD (sound-only sessions) Sound-event detection must activate on sounds, not speech, so it no longer runs through the voice VAD/transcription path. A sound-detection-only pipeline (sound_detection set, no transcription/LLM) now: - is accepted by prepareRealtimeConfig (sound_detection counts as a pipeline stage), - builds a lightweight model via newSoundDetectionOnlyModel (no VAD/STT/LLM/TTS loaded), and - defaults the session to turn_detection none (no VAD) with no transcription stage, so the client drives windowing via input_audio_buffer.commit (option A: client-side sliding window). The per-PCM C-API already supports arbitrary windows. commitUtterance gains a sound-only branch: it emits the conversation.item.sound_detection event (scored AudioSet tags) and stops - no transcription, no LLM response. generateResponse is now guarded on a transcription stage being present, so a sound-only turn never invokes the LLM. Existing transcription/VAD sessions are unchanged (additive). Added a commitUtterance sound-only Ginkgo spec asserting it emits the sound event and neither transcribes nor generates a response. go vet + golangci-lint (new-from-merge-base) clean; openai suite green. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): register sound-classification backend in gallery + CI Mechanical backend-image registration for the ced sound-event classifier, mirroring the parakeet-cpp Go/purego backend everywhere it is wired up. - .github/backend-matrix.yml: add the ced build matrix, field-for-field copies of the parakeet-cpp entries (cpu amd64/arm64, cublas cuda 12/13 amd64, l4t cuda-13 arm64, l4t-jetpack cuda-12 arm64, sycl f32/f16, vulkan amd64/arm64, rocm hipblas, and the metal darwin entry), changing only backend and tag-suffix. dockerfile stays ./backend/Dockerfile.golang. - backend/index.yaml: add the &ced meta anchor (capabilities map per platform) plus ced-development and the per-arch image entries, each uri/mirror tag-suffix matching the matrix exactly. The model gallery (GGUF) entry is intentionally deferred pending the HuggingFace publish (TODO note inline). - scripts/changed-backends.js: add an explicit item.backend === "ced" branch in inferBackendPath mapping to backend/go/ced/, same mechanism and ordering as the parakeet-cpp branch (before the generic golang fallthrough). - .github/workflows/bump_deps.yaml: register mudler/ced.cpp -> CED_VERSION in backend/go/ced/Makefile so the daily bot bumps the pin. - swagger/{docs.go,swagger.json,swagger.yaml}: regenerated via make swagger so the existing /v1/audio/classification annotations land in the generated spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): server-side windowing for realtime sound detection (option B) Adds an optional server-driven sliding-window classifier so a sound-only realtime client only has to stream audio (no input_audio_buffer.commit): - Pipeline.sound_detection_window_ms / sound_detection_hop_ms config knobs. When both > 0 on a sound-only session, the server classifies the last window of streamed audio every hop and emits a conversation.item.sound_ detection event; the input buffer is trimmed to one window so a long stream stays bounded. When unset, the session stays client-driven (option A). Runs independent of VAD (sound events are not speech). - handleSoundWindow (ticker) + classifySoundWindow (one tick, extracted so it is unit-testable) + writeWindowWAV, which declares the true InputSampleRate (NewWAVHeaderWithRate) so the classifier resamples correctly. Goroutine is started after toggleVAD and torn down with the session (close + wg.Wait). - Register pipeline.sound_detection (+window_ms/hop_ms) in the config meta registry; the earlier realtime commit added pipeline.sound_detection without a registry entry, failing TestAllFieldsHaveRegistryEntries. This fixes that and covers the two new knobs. Tests: classifySoundWindow emits an event + trims the buffer to one window, no-ops on too-little audio; writeWindowWAV declares the given sample rate. go build/vet + golangci-lint (new-from-merge-base) clean; config + openai suites green. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): add ced-base GGUF model gallery entries (f16 + q8_0) The ced-base weights are now published at mudler/ced-base-gguf (Apache-2.0, converted from mispeech/ced-base). Adds gallery/ced.yaml (backend: ced + known_usecases: sound_classification) and two gallery/index.yaml entries (ced-base-f16 default, ced-base-q8 smallest) with sha256-pinned files, and removes the now-resolved TODO from backend/index.yaml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): add tiny/mini/small GGUF model gallery entries Publishes the rest of the CED family (same architecture, metadata-driven port verified end-to-end on ced-tiny) to mudler/ced-{tiny,mini,small}-gguf and adds their f16 + q8_0 gallery entries: ced-tiny (5.5M, edge/Pi-class) f16 11MB / q8_0 6MB ced-mini (9.6M) f16 19MB / q8_0 11MB ced-small (22M) f16 42MB / q8_0 23MB All sha256-pinned. ced-base remains the accuracy default. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): point gallery entries at the consolidated mudler/ced-gguf repo All CED quantizations (tiny/mini/small/base, f16/q8_0) now live in a single HuggingFace repo, mudler/ced-gguf, instead of per-model repos. Repoint the 8 gallery model entries' urls + file uris accordingly. sha256 and filenames are unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): bump CED_VERSION to the short-clip fix Pin the ced backend to ced.cpp 99c6ed3, which fixes a crash on any clip shorter than target_length (~10.11s): time_pos_embed was added at its full 63-frame grid instead of being sliced to the clip's actual time grid, tripping ggml_can_repeat in ggml_add. Surfaced by the live realtime e2e (sub-10s windows) and gated with a short-clip parity test upstream. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(ced): list ced.cpp as a LocalAI-team engine + backend-guide directive - README.md: add ced.cpp to the "native C/C++/GGML engines developed and maintained by the LocalAI project" table. - docs/content/features/backends.md: add a Sound Classification backend category (sound-event classification / audio tagging) listing ced.cpp. - .agents/adding-backends.md: add a "Documenting the backend" section and two verification-checklist items requiring new backends to be documented in the backends.md category list, and in-house native engines to be added to the README maintained-engines table. This directive was missing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): repin CED_VERSION to the v0.1.0 release commit ced.cpp history was squashed into a single release commit (tagged v0.1.0), so the previous pin (99c6ed3) no longer exists upstream. Pin to c04ac14, the v0.1.0 release commit, so the backend builds against a commit that exists. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ced): silence gosec G304/G103 + govet unsafeptr on audited paths - sound_classification.go: os.Create(dst) where dst = temp dir + path.Base of the upload (no traversal). #nosec G304, matching the depth-anything-cpp handler. - goced.go: reading a NUL-terminated C string from a libced-owned buffer. #nosec G103 (gosec) + //nolint:govet (golangci-lint's unsafeptr check), since the uintptr is a C-owned malloc'd buffer, not Go-GC memory. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
32c47706ae |
feat(realtime): speaker-aware conversations - surface identity to client and LLM (#10424)
* feat(realtime): add voice_recognition enforce + identity config Add Enforce *bool and Identity *VoiceIdentityConfig to PipelineVoiceRecognition, plus EnforceGate/IdentityEnabled/ AnnounceEnabled/PersonalizeEnabled helpers. Enforce nil defaults to gating (backward compatible); identity surfacing is independent of the gate. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): add Speaker type and conversation.item.speaker event Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(realtime): split voiceGate into Resolve + authorize Split the speaker authorization into a Resolve step (embed once, produce a types.Speaker identity) and a pure authorize policy step, with a 0..100 confidence score mirroring /v1/voice/identify. The legacy Authorize wrapper is kept so existing specs stay green. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): resolve speaker per turn and emit conversation.item.speaker Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): personalize LLM turns with recognized speaker Set the per-message name field on each recognized user turn and append a current-speaker note to the system message, both gated by the voice recognition identity config. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): document speaker identity surfacing and personalization Document the new voice_recognition keys (enforce, identity.*) and the LocalAI-extension conversation.item.speaker server event in the realtime feature docs. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): cover when:first+identity re-resolution and multi-speaker history Add two integration specs to harden the speaker-aware realtime path: - when:first with an Identity block re-resolves the speaker every turn even though re-authorization is skipped after the first match: a later resolve error now fails closed, while a clean later resolve still surfaces and names the speaker. - multi-speaker history attribution: each user turn carries its own per-message name and the injected system note reflects the latest speaker. Test-only change; no production behavior was modified. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): surface speaker labels in conversation.item.speaker Carry the registered speaker's labels (identify mode) on types.Speaker so they flow into the conversation.item.speaker event and the stored item. Verify mode has no labels, so the field is omitted there. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): cover conversation.item.speaker over a real websocket Add a realtime-pipeline-identity config (verify mode, enforce:false, identity announce+announce_unknown+personalize) and two e2e specs driving the real server over a real WebSocket with the mock VoiceEmbed backend: an authorized speaker yields a conversation.item.speaker event naming e2e-speaker (matched true) and reaches response.done; an unauthorized speaker yields an unknown (matched false, no name) event and still responds, proving enforce:false never drops a turn. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): register voice_recognition enforce + identity fields The meta registry coverage test (TestAllFieldsHaveRegistryEntries) requires every config field to have an entry in core/config/meta/registry.go. The new voice_recognition.enforce and voice_recognition.identity.* fields were missing, failing tests-linux and tests-apple. Add registry entries (toggles) so the fields are surfaced in the model-config editor and the coverage test passes. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
9565db5f94 |
feat(models): model aliases - redirect a model name to another configured model (#10414)
* feat(config): add model alias field and self-validation Add ModelConfig.Alias (yaml: alias), IsAlias(), and an alias short-circuit at the top of Validate() that rejects self-reference and forbids setting backend/parameters.model on a pure-redirect alias. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(config): resolve and validate model alias targets in the loader Assisted-by: Claude:opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(middleware): resolve model aliases and stamp requested/served identity Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(modeladmin): reject alias configs with invalid targets on create/edit Validate alias targets at create/swap entry points (ImportModelEndpoint, EditYAML, PatchConfig) so a dangling, chained, or disabled alias target is rejected at save time rather than surfacing as a runtime error. Assisted-by: Claude:opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(api): add GET /api/aliases to list model aliases Adds an admin-gated read-only endpoint that lists every model alias config as {name, target} pairs, backed by the loader's existing GetAllModelsConfigs(). Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(mcp): add set_alias and list_aliases tools Expose model-alias management over the LocalAI Assistant MCP surface: list_aliases (read-only, GET /api/aliases) and set_alias (mutating). SetAlias is swap-first: PATCH /api/models/config-json/:name swaps an existing alias's target (validated, non-destructive) and a 404 falls back to POST /models/import to create a fresh {name, alias} config. The inproc client mirrors this via ConfigService.PatchConfig + a create path modeled on ImportModelEndpoint. Deletion reuses delete_model. Assisted-by: Claude:claude-opus-4 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(mcp): replace em dashes in alias tool comments Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(config-meta): expose alias as a model-select field Add an 'alias' section to DefaultSections() and an 'alias' field override in DefaultRegistry() so the schema-driven React editor renders the new top-level ModelConfig.Alias field as a model picker in its own section. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add alias template card and Manage alias badge Add an 'Alias / Routing' template to the create-flow gallery that seeds a minimal name + alias config, and a read-only 'alias -> target' badge on the Manage Models tab. The capabilities row payload does not carry the alias field, so the badge resolves targets from GET /api/aliases looked up by name. Assisted-by: Claude:claude-opus-4 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: document model aliases Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(swagger): regenerate for GET /api/aliases Adds the /api/aliases path and AliasInfo schema generated from the ListAliasesEndpoint annotation. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(localai): check os.RemoveAll error in aliases_test Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix: correct alias conversion docs and advertise /api/aliases in instructions Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(mcp): write alias config 0600 to satisfy gosec G306 The inproc createAlias path wrote the alias YAML with 0644, which gosec flags as a new G306 finding on the PR. The LocalAI process is the sole reader/writer of model configs, so 0600 is correct and keeps the scan clean. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
3fa7b2955c |
feat(pii): NER tier engine — privacy-filter.cpp backend + NER-centric PII filter (#10360)
Squashed feat/pii-ner-tier-engine rebased onto master (was 45 commits; see backup/pii-ner-tier-engine-prerebase). Net change: - privacy-filter.cpp: standalone GGML engine for the openai-privacy-filter PII/NER token classifier, wired as a LocalAI gRPC backend (CPU/CUDA/Vulkan). TokenClassify moves off the patched llama.cpp path onto this backend. - PII filter reworked to be NER-centric (encoder/NER detection tier scanning whole conversations as one document), with a recreated bounded restricted- regex secret-matching pattern detector tier alongside it (per-model pii_detection.builtins / .patterns + core/services/routing/piipattern). - Detection labelled by source (ner vs pattern); backend trace / confidence / debug observability; analyze/redact exposed as a synchronous API. - Instance-wide default detector policy + per-usecase default-on; request filtering extended to completions, embeddings, edits & Ollama. - React UI: NER-centric PII editor, detector-models table, pattern/builtins editor, middleware default-policy UI. - Gallery: privacy-filter-multilingual token-classify model + NER install filter; token_classify known_usecase; batch sized to context for NER models. privacy-filter backend registered in the backend gallery (cpu/vulkan/cuda-13 meta + image entries with a capabilities map) matching its CI matrix jobs, and an /import-model auto-detect importer (PrivacyFilterImporter, narrow privacy-filter GGUF detection) replacing the prior pref-only registration. Reconciled against master's independent evolution: - Dropped master's PIIPatternOverrides feature (global-pattern runtime overrides + /api/pii/patterns API + runtime_settings.json persistence). The per-model NER + pattern-detector design supersedes it; it was built on the global redactor pattern set this branch replaced. - Reverted the llama.cpp Score carry-patch (0006-server-task-type-score): removed the patch and restored master's grpc-server.cpp Score RPC (direct llama_decode, slot-loop bypass) and LLAMA_VERSION pin, plus master's model_config validation forbidding score + chat/completion/embeddings on llama-cpp. token_classify is unaffected (it runs on the privacy-filter backend, not llama-cpp). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
294170d3ed |
feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery (#10352)
* feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery Mirrors the locate-anything-cpp backend to register a new depth-anything backend that wraps the Depth Anything 3 ggml port (depth-anything.cpp) via purego (cgo-less, no Python at inference). - backend/go/depth-anything-cpp/: gRPC backend (Load + Predict + GenerateImage), purego binding to the da_capi_* C ABI, CMake/Makefile/run/package/test scripts building depth-anything.cpp's DA_SHARED static .so per CPU variant. - backend/index.yaml: depth-anything backend meta + all hardware-variant capability entries (cpu/cuda12/cuda13/intel-sycl-f32+f16/vulkan/nvidia-l4t). - gallery/index.yaml: 8 Depth Anything 3 GGUF models (base q4_k/q8_0/f16/f32, small, large, giant, mono-large). - .github/backend-matrix.yml: one build entry per hardware variant. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(depth): typed Depth RPC + REST endpoint exposing full DA3 data Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): pin depth-anything.cpp to e0b6814 (ABI 3 dense C-API) The Depth RPC handler calls da_capi_depth_dense / da_capi_points (C-API ABI 3); pin the native build to the commit that exports them. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): pin depth-anything.cpp to v0.1.0 release (b515c31) Repoint the native version from the now-orphaned e0b6814 to the b515c31 release commit, kept alive by the upstream v0.1.0 tag. C-API is unchanged (da_capi_abi_version == 3). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): wire depth-anything-cpp into build, CI bump, and importer The backend dir, gallery index, and CI build-matrix were present but the backend was never wired into the integration points that adding-backends.md requires: - root Makefile: add to .NOTPARALLEL, the test-extra chain, a BACKEND_* definition, the docker-build target eval, and docker-build-backends (mirrors parakeet-cpp; the backend's own Makefile already documented that its `test` target is driven by test-extra). - bump_deps.yaml: register the DEPTHANYTHING_VERSION pin so the daily auto-bump bot tracks mudler/depth-anything.cpp master (it cannot see an unregistered Makefile pin). - import form: add a preference-only KnownBackend entry so depth-anything is selectable at /import-model (mirrors sam3-cpp; no reliable GGUF auto-detect signal, so pref-only per the doc's default). changed-backends.js needs no entry: the generic golang suffix branch already resolves backend/go/depth-anything-cpp/. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(depth): auto-detect importer for depth-anything GGUFs Replace the preference-only entry with a real auto-detect importer (mirrors parakeet-cpp / locate-anything): - DepthAnythingImporter matches a .gguf whose name carries a depth-anything token (depth-anything-<size>-<quant>.gguf), so /import-model recognises mudler/depth-anything.cpp-gguf repos and direct GGUF URLs without an explicit backend preference. preferences.backend= "depth-anything" still forces it. - Registered before LlamaCPPImporter so its GGUF bundles aren't claimed by the generic .gguf importer; the narrow name match means it cannot claim arbitrary llama GGUFs or the upstream safetensors PyTorch repos. - Multi-quant repos pick the smallest quant by default (q4_k -> ... -> f32, depth stays >0.998 corr even at q4_k); quantizations preference overrides. - Drops the now-redundant knownPrefOnlyBackends entry (importer-backed backends are not listed there, matching parakeet-cpp). - Table-driven Ginkgo test covers detection, negative cases (llama GGUF, upstream safetensors), default/override/fallback quant pick, and direct URL import. 10/10 specs pass. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): check conn.Close error in grpc Depth client (errcheck) The new Depth() client method used a bare `defer conn.Close()`. golangci-lint runs with new-from-merge-base, so although the 39 sibling methods use the same bare form (grandfathered), the newly added line trips errcheck. Drop the result explicitly to satisfy the linter. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): bump depth-anything.cpp to v0.1.1 (embeddable CMake) v0.1.0 (b515c31) used ${CMAKE_SOURCE_DIR} for its include dirs, which points at the parent project when built via add_subdirectory() as this backend does, so the container build failed with missing stb_image.h / da_gguf_keys.h. v0.1.1 (2d42897) switches to project-relative paths. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): resolve gosec findings in the backend wrapper The code-scanning gate flagged three new failure-level alerts in godepthanythingcpp.go (gosec runs with -no-fail; GitHub gates on new alerts): - G301: export dirs were created with 0o755. Tighten to 0o750 (no world access needed for backend-written export output). - G304: writeDepthPNG creates req.GetDst(). That path is chosen by the LocalAI core as the intended output destination (same pattern every image backend uses), not attacker input, so annotate with #nosec G304 and document why. The remaining G103 "audit unsafe" notes on the unsafe.Slice C-buffer copies are warning-level (the same purego interop whisper/parakeet use) and do not gate the check, per the supertonic exclusion precedent in secscan.yaml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): bump depth-anything.cpp to v0.1.2 (CUDA cross-build arch) v0.1.1 forced CMAKE_CUDA_ARCHITECTURES=native, which breaks the GPU-less l4t/cublas CI builds (nvcc "Unsupported gpu architecture 'compute_'" on CMake 3.22). v0.1.2 (442eea4) drops the override and lets ggml pick its default cross-build arch list. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
1ab61a0875 |
feat: generic chat_template_kwargs (model config + per-request metadata) (#10359)
* feat(config): add chat_template_kwargs model field + resolver Adds the ChatTemplateKwargs model-config map and RequestMetadata carrier, plus ResolveChatTemplateKwargs which layers the config map under coerced request metadata. Foundation for generic jinja chat-template kwargs (issue #10329). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend): forward resolved chat_template_kwargs blob to backends gRPCPredictOpts now merges per-request client metadata over the server-derived enable_thinking/reasoning_effort (reaching all backends via the standalone keys) and serialises the resolved chat_template_kwargs map into a JSON blob for llama.cpp, written last so a client cannot clobber it. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(http): wire request metadata to config.RequestMetadata The OpenAI request metadata field was parsed but unused; stamp it onto the per-request ModelConfig so gRPCPredictOpts forwards it as chat_template_kwargs overrides. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): generic chat_template_kwargs merge (drop per-key blocks) Replace the per-key enable_thinking/reasoning_effort handling in both the streaming and non-streaming chat paths with a single block that parses the chat_template_kwargs JSON blob resolved by the Go layer and merges every key into body_json. New jinja template levers (e.g. preserve_thinking) now need no C++ change. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: document custom chat_template_kwargs (model + per-request) Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(backend): pin reasoning_effort as a string in the chat_template_kwargs blob Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(http): e2e guard pinning chat_template_kwargs forwarded to gRPC Adds an ECHO_PREDICT_METADATA marker to the mock-backend that echoes the received PredictOptions.Metadata, and an app_test.go spec that drives a real /v1/chat/completions request (model chat_template_kwargs + per-request metadata override) and asserts the exact metadata + chat_template_kwargs blob the REST layer forwards to gRPC. Locks the REST->gRPC contract against regressions. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(config): grandfather chat_template_kwargs in registry coverage chat_template_kwargs is a free-form map[string]any (like engine_args, already on the list), not a scalar the config UI registry can surface, so it is exempt from the registry-entry requirement. Fixes the TestAllFieldsHaveRegistryEntries failure introduced by the new field. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
2df2876db2 |
feat(supertonic): add Supertonic ONNX TTS backend (CPU) (#10342)
* feat(supertonic): vendor upstream Go TTS pipeline (helper.go) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(supertonic): add gRPC backend (Load/TTS/TTSStream, CPU) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(supertonic): satisfy unused linter (use onnxProvider; exclude vendored helper.go) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(supertonic): unit tests for resolvers + gated end-to-end synthesis Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(supertonic): gofmt backend.go comment block Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(supertonic): add Makefile, run.sh, package.sh (CPU build) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(supertonic): wire backend into root Makefile Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(supertonic): check ort.DestroyEnvironment return (errcheck) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(supertonic): resolve voice_styles as sibling of onnx dir; guard trim; test voice Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(supertonic): add CPU build matrix + gallery index entries Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(supertonic): expose as pref-only importable backend Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(supertonic): add Supertonic/supertonic-3 TTS model to the gallery 16 files (4 onnx + tts.json + unicode_indexer.json + 10 voice styles) from HF Supertone/supertonic-3, served via the supertonic backend. Defaults to voice F1; onnx/ + sibling voice_styles/ layout matches the backend's resolveVoicesDir. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(meta): register pipeline.max_history_items config field Pre-existing on master: the field was added without a registry entry, failing TestAllFieldsHaveRegistryEntries (core/config/meta). Add the entry so it renders properly in the model-config UI. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(secscan): exclude vendored supertonic backend from gosec helper.go is vendored from supertone-inc/supertonic; its G304/G404/G104 findings are inherent to upstream and the math/rand use is correct for flow-matching noise (crypto/rand would be wrong). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
4ec6e3221e |
feat(realtime): gate realtime pipeline voice models behind voice recognition (#10319)
* feat(realtime): add pipeline voice_recognition gate config schema Add the PipelineVoiceRecognition config block that gates a realtime pipeline behind speaker verification (identify against the voice registry, or verify against reference audios), with Normalize defaults and Validate enum/shape checks. Register the new fields in the config meta registry so the UI renders them with proper labels/components (required by the registry-coverage gate). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * fix(realtime): range-check voice gate threshold and floor UI min Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * feat(realtime): add cosineDistance helper for voice gate Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * feat(realtime): add voiceGate identify-mode authorization Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * test(realtime): cover voice gate fail-closed error paths Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * feat(realtime): add voiceGate verify-mode authorization Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * feat(realtime): add voiceGate decide policy helper Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * feat(realtime): add newVoiceGate constructor Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * feat(realtime): gate pipeline responses behind voice recognition Run speaker verification concurrently with transcription and join on a hard barrier before generateResponse, so unauthorized utterances never reach the LLM, tools, or TTS. Supports identify (registry) and verify (reference) modes with multiple authorized speakers, per-utterance or first-utterance checking, and drop-with-event or silent-drop on reject. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * fix(realtime): harden voice gate goroutine lifecycle Only launch the verification goroutine on the transcription path and drain it before the temp WAV is removed on the transcription-error return, so an in-flight backend read never races the deferred cleanup. Drop the write-only voiceMatched field; log the matched speaker instead. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * docs(realtime): document the voice_recognition pipeline gate Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * fix(realtime): fail closed on an incomplete voice_recognition block A present voice_recognition block with no model previously disabled the gate silently, authorizing every speaker. Treat block presence as the intent signal and reject an empty model in Validate, so the session is refused instead of running unprotected. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * test(realtime): integration-test the voice gate through commitUtterance Drive the real commitUtterance path (gate goroutine, hard join before the LLM, reject event, when:first session trust) with the existing transport/model doubles: authorized speakers reach a full response, unauthorized ones are dropped before the LLM with a speaker_not_authorized event, backend errors fail closed, drop_silent stays quiet, and when:first trusts the session after one match. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
085fc53bbc |
fix(router): production-ready request router + auto-size batch for embedding/rerank (#10104)
* fix(router): score classifier production-readiness Conversation trimming runs through the classifier model's chat template and trims by exact token count, sized to the model's n_batch which is now scaled to context so long probes can't crash the backend. Missing chat_message templates are a hard error at router build time. Router- facing factories (Embedder/Scorer/Reranker/TokenCounter) re-resolve ModelConfig per call so a model installed post-startup doesn't bind a stub Backend="" config and silently fall into the loader's auto- iterate path. New 'vector_store' backend trace recorded inside localVectorStore on every Search/Insert — including the backend-load-failure path that previously vanished into an xlog.Warn — with outcome tagging (hit/miss/empty_store/backend_load_error/find_error/insert_error/ok). Companion cleanup drops misleading similarity:0 and input_tokens_count:0 from non-hit and text-mode traces. Gallery local-store-development aliases to 'local-store' so the master image satisfies pkg/model.LocalStoreBackend lookups from the embedding cache. Misc: llama-cpp TokenizeString reads the correct 'prompt' JSON key (the original bug); ModelTokenize nil-guard; non-fatal mitm proxy startup; PII 'route_local' renamed to 'allow' with docs/UI in sync; model-editor footer no longer eats the edit area on small screens; several config-editor template/dropdown/section fixes. Tests: e2e router specs (casual/code-hint + long-conversation trim), vector_store trace specs, lazy-factory specs, gallery dev-alias resolution, Playwright trace badge + scroll regression. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(backend): auto-size batch to context for embedding and rerank models Embedding and rerank models pool over the whole input in a single physical batch (n_ubatch). With batch left at the 512 default, the backend rejects longer inputs with "input is too large to process", silently capping a large-context embedder (e.g. 8k/32k) at 512 tokens. Size n_batch to the context for these single-pass usecases, mirroring the existing FLAG_SCORE behaviour; an explicit batch: still wins. Extracts EffectiveContextSize/EffectiveBatchSize from grpcModelOpts so the effective decode window has one home for other callers to reuse. Adds an e2e-aio regression test that embeds a >512-token input. The AIO embedding model is switched to nomic-embed-text-v1.5 (2048 context) because the previous granite model was capped at 512 tokens and could not exercise the larger batch. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(gallery): raise arch-router scoring output cap via parallel:64 Scoring decodes the whole prompt+candidate in a single llama_decode and reads one logit row per candidate token. The vendored llama.cpp server caps causal output rows at n_parallel, so the default of 1 aborts with GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max) on multi-token route labels. Set options: [parallel:64] on both arch-router quant entries to lift the cap; kv_unified (the grpc-server default) keeps the full context per sequence, so this does not split the KV cache. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
|
892fc49949 |
feat(realtime): stream the LLM / TTS / transcription pipeline stages (#10176)
* feat(realtime): pipeline streaming + disable_thinking config
Add a nested pipeline.streaming.{llm,tts,transcription} block plus
pipeline.disable_thinking, with StreamLLM/StreamTTS/StreamTranscription/
ThinkingDisabled helpers. Pointer-bools so unset keeps the unary path;
existing configs are unaffected. Wiring into the realtime handler follows.
Assisted-by: Claude:claude-opus-4-8 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): sentence segmenter for streamed LLM->TTS pipelining
streamSegmenter accumulates streamed LLM tokens and emits complete
sentence/clause segments (terminator+whitespace, or newline) so TTS can
synthesize each segment as it completes instead of waiting for the whole
reply. Pure helper; the streaming handler wiring consumes it next.
Assisted-by: Claude:claude-opus-4-8 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): streaming TTS/transcription methods on Model interface
Add TTSStream and TranscribeStream to the realtime Model interface and
implement them on wrappedModel (delegating to backend.ModelTTSStream /
ModelTranscriptionStream) and transcriptOnlyModel. ttsStream adapts the
backend's WAV-framed stream (44-byte header carrying the sample rate, then
PCM) into raw PCM + sample rate for the realtime transports. Handler wiring
that consumes these (flag-gated) follows.
Assisted-by: Claude:claude-opus-4-8 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): emitSpeech with flag-gated streaming TTS
emitSpeech synthesizes a piece of text and forwards audio to the client,
streaming one output_audio.delta per backend PCM chunk when the pipeline
sets streaming.tts, or one delta for the whole utterance otherwise. WebRTC
gets raw PCM (it resamples internally); WebSocket gets base64 PCM at the
session rate. It emits no transcript/audio-done events so a streamed reply
can be split into multiple spoken segments sharing one response.
Adds fakeModel/fakeTransport test doubles for the realtime Model/Transport
interfaces, driving streaming assertions deterministically.
Assisted-by: Claude:claude-opus-4-8 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): route response audio through emitSpeech (streaming TTS)
Replace the inline unary TTS block in the response handler with emitSpeech,
which streams a response.output_audio.delta per backend PCM chunk when
pipeline.streaming.tts is set and otherwise preserves the single-delta unary
behaviour. emitSpeech returns the accumulated base64 audio, stored on the
conversation item as before. Transcript and audio-done events stay in the
handler so later per-segment streaming can reuse emitSpeech.
Assisted-by: Claude:claude-opus-4-8 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): streaming transcription text deltas
Add emitTranscription and route commitUtterance through it. With
pipeline.streaming.transcription set it streams each transcript fragment as
a conversation.item.input_audio_transcription.delta via TranscribeStream
then a completed event; otherwise it preserves the single completed-event
unary behaviour. Returns the final transcript for response generation.
Assisted-by: Claude:claude-opus-4-8 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): pipeline disable_thinking maps to enable_thinking off
applyPipelineThinking forces the LLM's ReasoningConfig.DisableReasoning when
pipeline.disable_thinking is set, which gRPCPredictOpts turns into the
enable_thinking=false backend metadata. Applied at newModel construction on
the per-session LLM config copy, so it doesn't leak to other model users and
needs no realtime-specific request plumbing.
Assisted-by: Claude:claude-opus-4-8 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): speechStreamer for token-streamed LLM->TTS
emitSpeech now returns raw PCM (caller base64-encodes) so streamed segments
accumulate correctly. speechStreamer consumes streamed LLM tokens: it strips
reasoning via the streaming ReasoningExtractor, emits a transcript delta per
content fragment, and sentence-pipes content into emitSpeech so each sentence
is synthesized as soon as it's ready. Handler wiring (plain-content turns)
follows.
Assisted-by: Claude:claude-opus-4-8 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): wire streamLLMResponse for token-streamed replies
triggerResponseAtTurn takes a streamed path when pipeline.streaming.llm is
set, the turn has no tools, and audio is requested: streamLLMResponse
announces the assistant item, drives the LLM token callback through a
speechStreamer (reasoning-stripped transcript deltas + sentence-piped TTS),
and emits the terminal events. Tool turns and non-streaming pipelines keep
the existing buffered path unchanged, so this is strictly opt-in.
Assisted-by: Claude:claude-opus-4-8 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(realtime): document pipeline streaming + disable_thinking
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): register pipeline streaming/thinking config fields
TestAllFieldsHaveRegistryEntries (core/config/meta) requires every config
field to have a meta registry entry. The four new pipeline fields
(disable_thinking, streaming.{llm,tts,transcription}) had none, failing
tests-linux/tests-apple. Add toggle entries for them.
Also handle the os.Remove return in realtime_speech_test.go to satisfy
errcheck (golangci-lint).
Assisted-by: Claude:claude-opus-4-8 go test, golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): always strip reasoning from spoken output
disable_thinking maps to ReasoningConfig.DisableReasoning=true on the LLM
config, which the backend reads as enable_thinking=false. But the realtime
handler reads that SAME config to drive reasoning extraction, and there
DisableReasoning=true means "skip stripping". PredictConfig() returns this
LLM config, so both the streamed (speechStreamer) and buffered realtime
paths stopped stripping <think>…</think> exactly when disable_thinking was
on — leaking raw reasoning to the client whenever the model ignored the
enable_thinking hint (e.g. lfm2.5).
Add spokenReasoningConfig() which clears DisableReasoning for extraction
(keeping custom tokens/tag pairs) and route both realtime paths through it.
Spoken output now always strips reasoning, independent of the backend
suppression hint.
Assisted-by: Claude:claude-opus-4-8 go test, golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): clean TTS temp path before read (gosec G304)
emitSpeech reads the WAV file the TTS backend wrote. The read moved here
from realtime.go, so code-scanning flagged it as a new G304 alert even
though the path is backend-controlled (a temp file), not user input.
Wrap it in filepath.Clean — a real path normalization that also clears
the alert, keeping with the repo's no-#nosec convention.
Assisted-by: Claude:claude-opus-4-8 gosec, golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(realtime): buffer whole message for TTS, drop sentence segmenter
Per review (richiejp): the sentence segmenter pipelined unary TTS by
splitting on ASCII .!?/newline, which does nothing for languages without
those boundaries (CJK/Thai) — there it already degraded to buffering the
whole message anyway.
Replace it with a uniform model: stream the LLM transcript live, buffer the
full message, then synthesize it once. emitSpeech already streams the audio
chunks when the backend implements TTSStream and falls back to a single
unary delta otherwise, so this is real streaming TTS where supported and a
clean whole-message synthesis elsewhere — no per-sentence emulation, no
language assumptions. speechStreamer becomes transcriptStreamer (transcript
deltas only); the whole-message synthesis moves into streamLLMResponse.
Assisted-by: Claude:claude-opus-4-8 go test, golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): stream tool-call turns via tokenizer-template autoparser
Per review (richiejp): tool-call deltas exist, so streaming should work with
tools too. It does — for models that use their tokenizer template. The C++
autoparser then clears reply.Message and delivers content + tool calls via
ChatDeltas, so the streamed transcript carries only spoken content (no
tool-call JSON leak) and the tool calls are parsed from the final response.
- Drop the len(tools)==0 gate; stream when no tools OR use_tokenizer_template
(grammar-based function calling still buffers, since its call is emitted as
JSON in the token stream and would leak into the transcript).
- streamLLMResponse takes tools/toolChoice/toolTurn, reads ChatDelta content
in the token callback, parses tool calls from the final ChatDeltas, and
creates the assistant content item lazily so a content-less tool turn emits
only the tool calls.
- Extract emitToolCallItems from the buffered path so both paths finalize tool
calls, response.done, and server-side assistant-tool follow-ups identically.
Assisted-by: Claude:claude-opus-4-8 go test, golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): script-aware clause chunking + streamed-reply fixes
Opt-in pipeline.streaming.clause_chunking splits the streamed LLM reply
into speakable clauses and synthesizes each as soon as it completes,
lowering time-to-first-audio instead of buffering the whole message. The
splitter is script-aware (rivo/uniseg, pure Go): UAX#29 sentence
segmentation handles CJK 。！？ with no whitespace, CJK clause
punctuation (，、；：) and Thai/Lao spaces give finer cuts, and a UAX#14
line-break cap bounds an over-long punctuation-less run. Unlike the old
ASCII .!?/newline segmenter (dropped in 076dcdbe) it does not degrade to
whole-message buffering for CJK/Thai; scripts needing a dictionary
(Khmer/Burmese) stay buffered until a space or end-of-message. Clauses
are synthesized synchronously in the token callback (the LLM keeps
generating into the gRPC stream meanwhile), so audio still starts
mid-generation. Off by default — the whole-message path is unchanged.
Also fix the streamed-reply path and the Talk page:
- Don't swallow streamed autoparser content as reasoning: the
tokenizer-template path already delivers reasoning-free content via
ChatDeltas, so prefilling the thinking start token re-tagged it as an
unclosed reasoning block, leaving no spoken reply. Disable the prefill
on that path; closed tag pairs are still stripped (#9985).
- Generate collision-free realtime IDs (16 random bytes) instead of a
constant, so per-item bookkeeping (cancel, conversation.item.retrieve)
works.
- Key the Talk transcript by the server item_id and upsert entries.
Realtime events arrive over a WebRTC data channel — outside React's
event system — so React defers the setTranscript updaters while
synchronous ref writes in handler bodies run first; the old
index-tracking ref rendered a duplicate assistant bubble on
completion. Upserts by item_id are idempotent and order-independent.
- Drop the partial assistant bubble on a cancelled response (barge-in):
the server discards the interrupted item and sends response.done with
status "cancelled"; mirror that in the UI so the regenerated reply
isn't rendered as a second assistant message.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Richard Palethorpe <io@richiejp.com>
|
||
|
LocalAI [bot]andEttore Di Giacinto
|
e837921c2c |
feat: forward reasoning_effort to the backend so jinja models honor it (#10184)
* feat: forward reasoning_effort to the backend so jinja models honor it reasoning_effort was only mapped to the binary enable_thinking toggle and otherwise reached Go-side templates — it was never sent to the backend. So jinja-templated models whose chat template keys on reasoning_effort (gpt-oss Harmony, LFM2.5) could not be driven by it: LFM2.5 ignores enable_thinking and kept emitting <think>. Forward the effective reasoning_effort to the backend as a chat_template_kwarg (mirroring enable_thinking) in grpc-server.cpp, and put it in PredictOptions metadata (gRPCPredictOpts). Add a config-level default: ModelConfig.reasoning_effort and Pipeline.reasoning_effort, resolved by ModelConfig.ApplyReasoningEffort (request value overrides config default, none->disable / level->enable, an operator's reasoning.disable wins). request.go now uses that helper. Assisted-by: Claude:claude-opus-4-8 go test, golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): set the pipeline LLM's reasoning_effort Apply Pipeline.ReasoningEffort to the pipeline's LLM config when the realtime model is built (per-session copy, overrides the LLM's own reasoning_effort), and surface the resolved effort on the template input so Go-templated models get it too. jinja models receive it via the backend metadata. This lets a realtime pipeline disable thinking on models that only honor reasoning_effort (e.g. LFM2.5), which enable_thinking can't. Assisted-by: Claude:claude-opus-4-8 go test, golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
6a80e23733 |
feat(middleware): Model routing, PII filtering, Cloud model proxies (#9802)
Add a routing middleware stack and a cloud-proxy backend. * cloud-proxy: a Go gRPC backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers, with an optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response) and full tool-calling support. * routing: admission control, content-aware model routing (embedding cache + classifier + rerank + Arch-Router score), PII detection/redaction (regex + NER) with streaming filter and OpenAI/Anthropic adapters, and a per-user/per-key billing recorder backed by GORM or in-memory storage. * middleware: UsageMiddleware records usage via the billing recorder, plus admission, route-model, usage-stamp and trace middlewares. * observability: BackendTrace ring buffer stores full request bodies (capped), MITM proxy emits structured trace events, and router classifier decisions surface at /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0). * UI: cloud-proxy model-editor fields, classifier system-prompt and score-normalization config, and a Traces page rendering request bodies. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Richard Palethorpe
|
9ac1bdc587 |
feat(ui): Interactive model config editor with autocomplete (#9149)
* feat(ui): Add dynamic model editor with autocomplete Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(docs): Add link to longformat installation video Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Richard Palethorpe
|
557d0f0f04 |
feat(api): Allow coding agents to interactively discover how to control and configure LocalAI (#9084)
Signed-off-by: Richard Palethorpe <io@richiejp.com> |