2 Commits

Include renames

Author	SHA1	Message	Date
mudler-agentandEttore Di Giacinto	2cadf319a0	feat(realtime): LFM3-Audio speech-to-speech on llama.cpp, remove liquid-audio (#12958) ... * chore(backends): remove the liquid-audio Python backend The backend targets liquid-audio, which loads LFM2 checkpoints only. Remove its directory, the gallery file and the LFM2.5-Audio model entries, the library and generated backend gallery entries, the backend matrix and CI job, the Makefile targets and the golden test fixtures that listed it. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(config): drop liquid-audio from capabilities, importers and UI With the liquid-audio backend gone, remove its capability entry, its importer, its artifact-gate entry and the realtime_audio heuristic that only matched that backend. realtime_audio is now set only through known_usecases. Update the Talk and fine-tuning pages, the docs and the swagger text that named the backend. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump llama.cpp to e486a4b5 Move the llama.cpp pin to e486a4b559617dc88bf79978996f6c670b6bbc9d, the master head of 2026-10-28. It contains the LFM3-Audio support (ggml-org/llama.cpp#31257) and the realtime server split (ggml-org/llama.cpp#31269). The score task patch is regenerated against the new pin. It applies with git apply and with patch -p1, with no offset and no fuzz. Its hunks in server-task.h and server-context.cpp were rebased by hand next to SERVER_TASK_TYPE_REALTIME and the control dispatch. The C++ backend was built on CPU for this change: the llama, mtmd and server-context libraries and the grpc-server binary link. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): compile server-realtime.cpp into grpc-server server-context.cpp includes server-realtime.h, and its realtime slot type is defined in server-realtime.cpp. Upstream compiles that file into the server library. The gRPC server is one translation unit, so it must include the file too, or the link fails. The include is guarded by __has_include. Forks that predate the realtime split do not ship the file, so they keep building as before. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): probe realtime and TTS voice support in prepare.sh prepare.sh writes llama_compat.h from probes on the checkout. Add two: LOCALAI_HAS_REALTIME (SERVER_TASK_TYPE_REALTIME in server-task.h) and LOCALAI_HAS_TTS_VOICE (set_voice in mtmd-helper.h). The probes let the gRPC server choose the realtime and voice code paths per checkout, so forks without them keep compiling. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): advertise speech recognition and speech-to-speech llama-cpp now registers the AudioTranscription and AudioToAudioStream RPCs, with the transcript and realtime_audio usecases. The default usecase stays chat, so a plain GGUF is still a chat model. The backend implements the RPCs in a follow-up change; until then the calls return UNIMPLEMENTED. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(config): add pipeline.any_to_any for speech-to-speech models A realtime_audio model that serves the whole speech loop itself sets pipeline.any_to_any. The flag is off by default, so chained pipelines and existing realtime_audio configs are unchanged. The realtime_audio heuristic stays explicit: a llama-cpp model is never a Talk-page model unless known_usecases says so. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(config): read the audio encoder and speech-to-speech projector flags Add HasAudioEncoderProjector for clip.has_audio_encoder and HasSpeechToSpeechProjector, which requires the LFM3 audio encoder family together with the gen-audio decoder. Qwen3-TTS sets both encoder flags too, so the flags alone cannot tell it apart from LFM3. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(importer): import LFM3-Audio projectors as speech-to-speech An mmproj that carries the LFM3 audio encoder and the gen-audio decoder now imports with known_usecases realtime_audio, chat, transcript and tts, pipeline.any_to_any: true, the parallel:2 option and the tokenizer template. Other gen-audio projectors keep the TTS-only result, Qwen3-TTS included. The probe runs against a local HTTP server in the tests. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): named-voice TTS and the LFM3 ASR prompt Models without a speaker reference (LFM3-Audio) take a named voice through their "tts" chat template instead of a reference file. The request is validated with needs_reference=false, the prompt is rendered with common_chat_get_tts_prompt(), and the voice is set on the gen-audio input. Qwen3-TTS Base keeps the reference-file rule. The branch is guarded by LOCALAI_HAS_TTS_VOICE, so forks without named voices keep the old behaviour. Transcription takes its prompt from common_chat_get_asr_prompt(). LFM3 gets a system turn that names the language and an empty user turn. Qwen3-ASR's preset is unchanged: fixed user text, no system turn, and a caller prompt is still appended after a newline. The preset call is guarded by a new LOCALAI_HAS_ASR_PRESET probe. The pure parts sit in tts_request_options.h and transcription_prompt.h, with standalone tests run by backend/cpp/run-unit-tests.sh. Add the test-extra-backend-llama-cpp-lfm3 target. It runs the tts, ttsstream and transcription specs against LFM3-Audio GGUF weights. It is a manual run and is not wired into CI. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): mock AudioToAudioStream and pending realtime S2S specs The mock backend serves AudioToAudioStream deterministically: session created, commit with RMS, a tool call on the first turn with tools, then transcript pieces and paced 24 kHz tone frames, with response.cancel ending a reply as cancelled. The realtime S2S specs cover the mock and real-weights variants. They skip until the /v1/realtime S2S path lands, so the mock suite stays green in CI. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): speech-to-speech AudioToAudioStream on the realtime slot Implement AudioToAudioStream on upstream's realtime slot, so a model that answers after the audio (LFM3-Audio) can hold a conversation over the existing duplex RPC. The proto schema does not change: tools travel in the config, tool calls leave as function_call_arguments.done and tool results arrive as a function_call_output item. realtime_session.h keeps what needs no model: the turn text derived from the chat template (with a tool turn), the conversation state, a filter that keeps <tool_call> blocks out of the transcript and drops audio produced inside them, and the translation of slot results into events. grpc-server.cpp wires it to the slot with control tasks and parses calls with the parser the chat template selects. The unit test uses a fake renderer. A ctest-only check renders the real LFM3 template and compares the slot text with the full render. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add the LFM3-Audio-2B entries Two llama-cpp entries for LiquidAI/LFM3-Audio-2B-GGUF, the Q4_K_M default (lfm3-audio-2b) and the Q8_0 build (lfm3-audio-2b-q8), both with the Q8_0 audio projector. Each declares pipeline.any_to_any and the realtime_audio, chat, transcript and tts usecases the importer writes for an LFM3 projector, with parallel:2 and the tokenizer template. The Q4 entry lists the Q8 one as a variant. File sizes and sha256 come from the Hugging Face API for the pinned repo revision. Speaker reference files and the VAD model are not part of the entries: VAD stays optional. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe LFM3-Audio speech-to-speech and its gallery entries Adds a speech-to-speech section to the Realtime API page (any_to_any configuration, VAD versus manual commit, tools, transcripts, barge-in, limits), a pipeline.any_to_any row, a llama.cpp speech-to-speech row in the compatibility table, and LFM3-Audio mentions on the TTS and ASR pages. The feature is marked experimental: the behaviour follows the backend session contract and has not been run with real weights. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): run any-to-any models on AudioToAudioStream A realtime_audio model with pipeline.any_to_any now opens one duplex stream to the backend and keeps it for the session, replacing the "any-to-any models not implemented" branch. The chained pipeline is unchanged. The endpoint forwards input audio (with a pre-roll from speech start when a VAD model is set, otherwise every chunk and a client commit), sends response.create, and turns the backend's events into the usual realtime events: audio and transcript deltas, tool calls through the existing function_call items, and a single response.done. A function call output is forwarded before the next response. Barge-in sends response.cancel ahead of the new speech and stops forwarding audio at once. The user's speech is transcribed on the side and reported on the user item; a failure there does not touch the response. Server VAD is on only when pipeline.vad is set. Classifier mode, compaction, streaming.llm, voice gating, sound detection and semantic_vad are rejected for these models. The mock backend now matches the llama.cpp contract (session.updated, the cancel error when idle) and its VAD reports the loud stretches of the audio, so the S2S e2e specs run a VAD turn. The specs are enabled. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): add the real-weights realtime S2S variant and the gRPC s2s spec The realtime S2S specs now check reply audio on real models: duration, loudness, clipping, and keywords found in the model transcript or in a separate ASR of the output audio. run-realtime-lfm3.sh stages the llama-cpp and sherpa-onnx backends, the LFM3-Audio files and the sherpa VAD, ASR and TTS models, then runs the real variant. It is manual and never part of CI: `make test-realtime` still filters out real-models. The backend suite gains an s2s capability (spoken turn, tool call, tool result, spoken answer over AudioToAudioStream) and optional TTS duration and RMS bounds plus a transcript keyword check, all enabled for test-extra-backend-llama-cpp-lfm3. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): expect the idle-cancel error from the mock S2S The mock now answers a response.cancel with no reply running with response_cancel_not_active, as the llama.cpp backend does. The cancel spec still treated that cancel as a no-op and read the error as extra data after the stream ended. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(talk): show any-to-any models as one speech-to-speech model /api/pipeline-models lists a pipeline.any_to_any model with any_to_any set and its optional VAD. A realtime_audio model that is not any-to-any is listed only when it has the four pipeline slots, since the session takes the chained path for it. The Talk page shows a speech-to-speech model as one model: no four-slot view, the VAD slot only when a VAD model is set, and the female and male voices offered. Without a VAD model the server does not detect turns, so the page offers Done speaking, which sends input_audio_buffer.commit and then response.create. Text-only replies stream through response.output_text.delta and .done. self_contained stays in the response for clients built before any_to_any. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): refuse a llama-cpp realtime model without pipeline.any_to_any llama.cpp has no VAD, transcription or TTS RPCs, so filling the empty pipeline slots with the model name failed later with UNIMPLEMENTED. Fail at session start and name the missing setting instead. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): hold manual-commit S2S audio until the commit In manual-commit mode the endpoint streamed every appended chunk to the backend, so input_audio_buffer.clear only emptied the Go buffer while the model kept the audio, and idle audio filled its context between turns. Keep the audio in the input buffer and send it, behind pending text and tool-result items, when the client commits or asks for a response. Frames and controls now go through one ordered write queue, so no gRPC send happens under the audio buffer lock. A tool result created while a VAD turn is open is refused with a clear error. When the stream ends outside a response the session reports it and closes, and a bare EOF from Send reports the status that Recv holds. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): drop the S2S readiness gate and stale comments The gate was always true. Some comments referred to work items that readers of the repository cannot see. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): add the liquid-audio migration note and the clear semantics Say that the Python liquid-audio backend is gone, which gallery entries replace it and that LFM2.5-Audio has no replacement. Describe what input_audio_buffer.clear drops for speech-to-speech sessions now that manual-commit audio is held back until the commit. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): do not promise turn detection when no VAD model is set The idle note said the server notices when you stop talking. A speech-to-speech model without a VAD model needs Done speaking, so the note now says that. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): make the mock S2S backend keep the llama-cpp session limits The mock accepted what the real backend refuses and hid two endpoint bugs. It now treats output_modalities from the config as a fixed session mode and picks the default per response from the tools set at that time. A fixed text plus audio mode with tools reads the tool call out loud. After the first commit or message, session.update of instructions, voice or tools is an error that does not belong to a response, committing an empty buffer is an error, and response.create while a reply runs is refused. Errors of a refused response.create carry response_scoped in their meta. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): leave the S2S output mode to the backend unless the client sets one The stream config sent text plus audio when the session had no tools yet. The llama-cpp backend keeps a mode from the config as a fixed session mode, and tools added later by session.update never cleared it. The model then read tool calls out loud. Send output_modalities only when the client chose them. The backend picks the default of each response from the tools it has when the response starts. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): mark the realtime errors that answer a response.create The error event now carries response_scoped: true when it ends the request for a response (a refused response.create, a failed commit made by it, or a stream failure). Other errors, such as a refused session.update or a commit of an empty buffer, are not about a response and one that runs keeps generating. The client needs the difference to know whether a response.done is still coming. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end an S2S response only on the errors that belong to it Every backend error ended the response in flight, although the backend also reports errors about other things (a refused session.update, a commit of an empty buffer) and goes on generating. The rest of the reply was lost and the next response.create met either a refusal or the leftover events of the old reply. End the response only on an error marked response_scoped, forward the others to the client and keep waiting for response.done. Drop the events buffered by an earlier response when a new one starts, and skip the commit of a turn that forwarded no audio. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): run the S2S input transcription detached from the commit slot The commit waited for the side transcription of the user audio while it held the commit slot, and response.create waits for that slot. The ASR latency was added before every reply, and on a backend with few slots a busy ASR task could stall the reply. The model already has the audio, so run the transcription detached under the session context and release the slot at once. The transcript is still reported on the user item. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): do not pin the place of the input transcript in the S2S order The transcript of the user audio is reported on the side and may land before or after the first response events. Check the order of the response events without it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): stop an S2S reader leak and a cancel that waits on the write queue The reader waited on a channel it closes itself when the events buffer was full, so it outlived the session after a turn ended on an error or timed out. It now also stops when the session closes. Stopping a response sent the cancel through the write queue and waited for it before the grace timer started. With backpressure on the stream the wait never ended and nothing read the events meanwhile. The timer now starts first and the cancel is only queued. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): let a chained pipeline run without a VAD model A pipeline that names no pipeline.vad failed to build, because the VAD slot was always loaded, and a new session still started with server_vad. The slot is now optional for chained pipelines as it already is for any-to-any models: the session starts with turn detection off and the client commits each turn. Turning server_vad on for such a session is refused, and a switch to a pipeline without a VAD drops turn detection. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add LFM3-Audio VAD and chained-pipeline entries Add lfm3-audio-2b-vad and -q8-vad (any-to-any with silero-vad-ggml as the turn detector), and lfm3-audio-2b-pipeline, -pipeline-vad and the -q8 forms (the model chained through its own transcription, chat and speech slots, with and without the VAD). The gallery has no dependencies between models, so the VAD entries state that silero-vad-ggml must be installed first. Drop the variants list from lfm3-audio-2b: it let the installer pick the larger Q8 build on any host with enough memory, so a plain install of the Q4 entry did not get Q4. All entries are now independent. Tests cover the entry shapes, the single-build install and their acceptance by the realtime endpoint. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): run the realtime specs for any-to-any and chained shapes The realtime specs covered one any-to-any model with a VAD. They now run for any-to-any and for a chained pipeline, each with and without a VAD, on the mock backend in CI. VAD-less shapes also check that turn detection starts off and that server_vad is refused. The manual real-weights script writes one config per shape that mirrors the gallery entries and runs the four shapes in turn (LFM3_MATRIX picks a subset). Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): document the LFM3-Audio VAD and chained-pipeline entries List the eight LFM3-Audio gallery entries, explain that silero-vad-ggml must be installed before the -vad entries, describe the chained pipeline on one model and the behaviour of a pipeline without a VAD, and describe the shape matrix of the e2e specs. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): wait for the detached input transcription in the S2S specs The input transcription runs in its own goroutine, so the transcription completed and failed events can arrive after the response ends. Poll for them instead of reading the event list right after the response. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): release the S2S stream before running server-side tool calls A response held the stream slot until it returned, and it ran the tool calls from inside that scope. A server-side tool starts a follow-up response on the same stream, which then waited for the slot its parent still held, so the client never got response.done. Run the tool calls once the slot is free; the event order is unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end an S2S session whose stream dies, and stop queuing after close A stream that died during a response left the session open: later requests failed one by one and VAD frames kept being forwarded. The response now ends the session through the transport, and enqueue drops requests once the session is closed so the write queue cannot grow. The writer is now the only caller of Send and of CloseSend. close() used to call CloseSend while the writer could be inside Send, which gRPC does not allow. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): close the open S2S turn when the VAD stops Switching turn detection to manual in the middle of a VAD turn left the turn forwarding, and the next client commit sent the whole buffer a second time. Stopping the VAD now commits the forwarded audio on the backend and drops it from the input buffer. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): keep a tool result pending while an S2S user turn is open The check that refuses a tool result during a user turn ran when the item arrived, but the item was forwarded later, and the result was marked as sent before the backend took it. If the user spoke in between, the backend refused it and it was never retried. The result is now claimed and queued under the audio lock only when no turn is open, and otherwise stays pending for the next commit or response. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): empty the input buffer for models without a VAD Without voice detection the microphone fills the server's input buffer for the whole session, so Done speaking committed idle silence and the echo of the previous reply. Clear the buffer when the session starts and when a reply ends. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(importer): keep an existing parallel option on LFM3-Audio imports The speech-to-speech import added parallel:2 even when the options already carried another parallel value. Use the same check as the hardware defaults, which also covers n_parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): list chained pipelines that have no VAD /api/pipeline-models required a VAD for a chained pipeline, so the VAD-less LFM3-Audio pipeline entries never showed up on the Talk page. Any pipeline with transcription, LLM and speech slots now qualifies. The Talk page treats every model without a VAD as manual-turn, and the session sheet says so for chained pipelines too. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump llama.cpp to 0ac1a7fd Move the llama.cpp pin to 0ac1a7fd35c127a18c0cbc6c57d12e256a6a385b, the master head of 2026-10-29. It contains tool calls in the realtime server (ggml-org/llama.cpp#31287). The score task patch is regenerated so that it applies with no offset and no fuzz. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): keep a null tool argument null A tagged parameter whose text was "null" came out as an empty string: unmarshalling JSON null into a string succeeds and leaves it empty, so the value was taken for a string. Check the first byte of the JSON value instead. A parameter with the text "null" now becomes JSON null, as in llama.cpp. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(functions): add an lfm3 XML format preset The LFM3 chat template writes a call as <tool_call>, <function=NAME>, one <parameter=KEY> block per argument, then the closing tags. The qwen3-coder format has the same shape and auto-detection already found it, but xml_format_preset: lfm3 was not a known name. Add the preset, with tests for one call, a call without arguments, two calls, string and non-string values (numbers, booleans, null, arrays, objects), a call between text, and unfinished input. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): parse tool calls with the chat template, as upstream Since ggml-org/llama.cpp#31287 the realtime server handles tools with the parser and grammar of the chat template. Do the same in the speech-to-speech stream instead of a hard-coded tag filter and a parse of the whole text at the end. - Each response gets a common_chat_session with the tools of the session and parallel calls. Its grammar and preserved tokens go into the sampling of the response, so the slot keeps the call tokens in the text and constrains the call. - The text of the response is fed to the session piece by piece (task_result_state::update_chat_msg). Content deltas become the transcript; tool-call deltas are collected into one function_call_arguments.done per call at the end. Audio after the start of a call is dropped. - A tool_choice of auto, none or required is accepted in the params of the config, in session.update and in response.create, with the response overriding the session. - The tool turn is derived from a conversation in which the assistant called a tool, like upstream, not from a plain reply. - response.done carries raw_text, everything the model wrote with the call markup, for a caller that parses it another way. If the parser throws, the rest of the response is content and the raw text is the way out. ToolCallFilter is no longer used and goes in the next commit. The unit tests cover the splitter contract (content, call deltas, the final piece, whole characters only). realtime_chat_check feeds LFM3-format text through the session in pieces of 1, 7 and all bytes and checks the call, the transcript and the grammar. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(realtime): drop the tool-call text filter The parser of the chat template now splits content from tool calls, so ToolCallFilter, which looked for the literal <tool_call> tags, has no caller. Remove it with its tests. Every case it covered (a call between text, split tags, several calls, an unfinished call, text after a call) is handled by the parser, and realtime_chat_check runs the LFM3 format through it in pieces of different sizes. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): fall back to the raw text for S2S tool calls For an any-to-any model, the calls the backend parsed still win. When there are none and the request had tools, parse the raw text that response.done now carries the way the chained pipeline parses the text of its LLM (CleanupLLMResult, ParseFunctionCall, ParseTextContent), so a call format the backend parser does not know can still be served with function.xml_format_preset or the other function settings of the model. The content around the calls replaces the streamed transcript only if the config captures it, as in the chained path. Also pass tool_choice to the backend: the session mode with the stream config and with session.update, and the mode of a response.create override with that event. A named function cannot be forced, so it is not sent. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): carry raw_text in the mock S2S response.done Match the llama-cpp session: with tools set, response.done has raw_text, the call markup of the LFM3 template for a tool turn and the transcript for any other reply. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): parse tool blocks that have text between them The iterative XML parser stopped at the first scope block that was preceded by text, so "<tool_call>A</tool_call> then <tool_call>B" gave one call. Skip text between blocks once a call was found; text before the first block still ends the parse as before. Blocks that follow each other, or only whitespace, already worked. Tests cover three calls back to back and with text between them, with the lfm3 preset and with auto-detection. They also pin the current reading of argument values: the parser has no tool schema, so a value such as 42 or true is read as JSON even when the schema types it as a string. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): read LFM3-Audio tool calls with the lfm3 preset Set function.xml_format_preset: lfm3 on the eight LFM3-Audio-2B entries and in the config the importer writes for a speech-to-speech projector, so the server parses the call format of the LFM3 chat template when the backend returns no call (and in the chained pipeline entries, which use the regular chat path). gallery/index.yaml is regenerated from the library. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): describe how S2S tool calls are read Replace the description of the hard-coded tag filter with the parser and grammar of the chat template, tool_choice, and the raw-text fallback in the server. Document function.xml_format_preset, and set it in the real-weights e2e model configs like the gallery entries do. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): keep earlier calls when a later block does not parse After a valid call, text that holds the opening tag again (a prose mention, a markdown example, a malformed second block) made the parser treat it as a new block, fail, and return no calls at all. Remember the state after each complete block and fall back to it when a later one fails: the calls found so far are kept, parsing stops there and the remaining text stays in the content. Text skipped between two valid blocks now goes to the content instead of being dropped, so StripToolCallMarkup keeps it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): do not drop audio after a tool call started In mixed mode the speech for text written before a call can still be pending when the call starts, so dropping every audio frame after the first call delta cut the speech while its transcript was sent. The llama.cpp server never drops audio when it reads a result; do the same. If a model speaks the call markup aloud, that is model behaviour to observe on real weights, not something to filter here. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): flush held-back text when a response is cancelled The parser got is_partial=true on a cancelled response, so text it held back (a possible start of a tag) never reached the transcript. The llama.cpp server parses a cancelled response with is_partial=false; do the same. A cancelled response still makes no tool call. The path where the parser throws already sends the text as content and needs no change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): stop the fake S2S stream from sending after Close The barge-in spec failed now and then with a data race and "send on closed channel": its script pushes audio from a goroutine, and the session closes the stream while that goroutine still runs. A push now does nothing once the stream is closed, and Close waits for a push in flight. The production close order is right: the writer closes the stream only after it stops sending. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): align the S2S tool-call fallback with the backend Four gaps in how the Go side reads the tool calls of an any-to-any reply: - With tool_choice none (the response value wins over the session value) the backend builds no tool parser, yet the raw-text fallback still turned a call the model wrote into a function_call item. Skip it. - Without a capture regex the fallback never cleaned the transcript, so the stored item and transcript.done kept the raw call markup. When the call is XML, use the text left after the markup is cut out; a reply that is only a call now has an empty transcript. - A named function as tool_choice mapped to nothing, so the backend kept the previous mode. Refuse it with an error in session.update and response.create. - A response-level tools list changed what the fallback looked for, but the backend only has the tools of the session. Use those in both. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): say how S2S tool_choice and response tools behave Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(e2e): use the real llama-cpp backend in the real realtime run The suite always registered the mock backend under the name llama-cpp, so a run with REALTIME_S2S_MODEL set never reached the llama-cpp backend found in REALTIME_BACKENDS_PATH. Register the mock only when that run is off. The real run converts the input audio with ffmpeg. Check for it in the suite and in run-realtime-lfm3.sh before any download or backend start, with a message that says what to install, and list it as a prerequisite in the script header and the Makefile target. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): cover the tools path of a client session.update to the S2S backend A tool call that never came back from an any-to-any model raised the question whether the tools of the session reach the backend at all. Add a spec that sends the session.update of the real-weights e2e (flat Responses API tool, no session type) and checks that the backend gets the tool in the chat-completions form. Log the number of tools sent when the stream opens and when a session.update carries them, so a run log shows it without reading the wire. The "No tools received" line of the llama-cpp backend does not show it: the server prints it for the input transcription request, which has no tools by design. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): interrupt the chained pipeline reply while it is in flight With unary stages, the default of the lfm3 pipeline gallery entries, the chained pipeline synthesizes the whole reply and sends it as one audio delta right before response.done. The barge-in spec waited for that delta and then rendered the interruption with the TTS, so the first response was already done when the speech started and ended completed instead of cancelled. Real runs on a GPU showed it with both quantizations. For a pipeline, start the interruption at response.created, which comes before the LLM and TTS stages. Render the interruption audio before the first utterance is sent, so nothing sits between the reply and the interruption. The any-to-any shape still waits for the first audio delta, which that backend streams while it generates. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): accept a number word and its homophone in the spoken-answer check The check that the reply audio says what the transcript says needs a shared word of 4+ letters. A short answer failed it: the model wrote "Four" and the ASR heard "for". Compare numbers in one spoken form, so digits, number words and their homophones match each other. A homophone only counts when the other text has the number itself, so "for" on both sides still shares nothing and the check does not pass on a stop word. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): ask LFM3-Audio for function calls when a realtime session has tools The default system prompt of a realtime session was "Respond with interleaved text and audio." whatever the tools. Under that prompt LFM3-Audio answers in words ("I'll check the weather ...") and never writes the call. The model card, and the llama.cpp realtime example client, use "Respond with function calls, and interleaved text and audio." when the session has tools, and the model then calls the tool. Pick the default by the tools of the session, and treat the plain default that the LocalAI server sends back as no choice: it follows the tools too, also when they arrive with a session.update before the first audio. A prompt the client wrote is kept as it is. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): transcribe the input of an any-to-any turn between responses The input transcript of a speech-to-speech turn ran at the same time as the reply, on the same model. llama.cpp serves both from one context: the realtime slot wants embeddings on, the chat slot of the transcript wants them off, and each flip makes it reserve its compute buffers again. In a GB10 run that was two reserves of about 175 ms per step, the reply fell to 2.6 tokens per second and the transcript to 380 ms per token, until the transcript was done. When pipeline.transcription is the model of the stream, run the transcript after the response, or before one that is not started yet, and keep responses out while it runs. A separate transcription model still runs in parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): send the input transcript of an any-to-any turn before its reply The previous change moved the same-model transcript after the response. A client that reads until response.done then never saw conversation.item.input_audio_transcription.completed: the transcript was made (the log shows it) but sent after the end of the window. Run it in line with the commit instead. A VAD turn transcribes and then responds. A client response.create already waits for the commit slot, so it starts once the transcript is out. The transcript event now comes before any output of the response of its turn, for a manual commit and for a VAD turn, and never overlaps the response on the llama.cpp context. A transcription model of its own still runs in parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): transcribe the input of an any-to-any model only when asked The stream answers from the audio, so the transcript of the input is for the client only. On the model of the stream it is costly: llama.cpp serves the transcript request and the realtime stream from one context and rebuilds its compute buffers on every step while both are active. On a GB10 that was 26 to 29 reserves of about 175 ms per turn, against 0 to 1 on the llama.cpp server, and the reply ran at a few tokens per second. An any-to-any model no longer fills pipeline.transcription with its own name. Without it the session does not transcribe the input; with another model it does, in parallel with the reply; with the model itself named it does, before the reply. The reply transcript always comes from the stream. The gallery entries set none, so they get the fast path. The mock configs and run-realtime-lfm3.sh name a transcription model, so the real-weights spec still covers the input transcript. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end a response as cancelled when a barge-in lands during the LLM call The chained pipeline cancels the response context on a barge-in. When that happens while the model is generating, the backend call fails with the context error, and the response ended as failed: an error event and no response.done. A client that waits for the terminal of the interrupted response then waits forever. On a GB10 the q4 model answered before the barge-in arrived, so only the slower q8 model showed it: the spec read events for 30 s and timed out. When the context is cancelled, a failed Predict call or a failed result is a cancel, as the TTS step and the streamed LLM path already treat it. The response ends with response.done{cancelled} and no error. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): wait for the input transcript that comes from a model of its own An any-to-any model now gets its input transcript from a separate ASR that runs next to the reply. The reply is often over first, so a spec that stops reading at response.done saw no transcript in the VAD shape. Keep reading, up to 30 s, until the transcript arrives. The contract is that it comes, not when. Add a guard spec for what the input transcription is handed: the whole turn, in a WAV that says the rate it was recorded at (the session rate for a client commit, 16 kHz for a VAD turn). Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): check the errors of Close in the realtime specs golangci-lint (errcheck) flagged the unchecked Close of the websocket connections, the response body and the fake stream. Use the `defer func() { _ = x.Close() }()` form the other e2e specs use. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): write the mock S2S specs with Ginkgo and drop the env read forbidigo only allows Ginkgo/Gomega specs and no os.Getenv outside the sanctioned places. Convert the mock S2S tests to Describe/It with a suite bootstrap (the package had none), with the same 11 cases. The mock no longer reads MOCK_S2S_FRAMES: only these specs set it, so the reply length is a package variable they change per spec and restore afterwards. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-10-29 22:06:41 +01:00
mudler's LocalAI [bot]andEttore Di Giacinto	7b9167eaad	feat(llama-cpp): serve Qwen3-TTS through the llama.cpp backend (#11392) ... * fix(config): do not read a TTS speaker-encoder mmproj as vision support Qwen3-TTS on llama-cpp ships an mmproj holding the speaker encoder and code predictor. VisionSupported() treated any non-empty MMProj as proof of image input, so every such model would be advertised as vision-capable. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): add TTS request option parsing helper Validates text and speaker reference presence and strictly parses the top_k / top_p per-request params, in a header with no llama.cpp or gRPC dependencies so the standalone C++ unit test gate picks it up. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): range-check the TTS top_k and top_p request params Format validation alone let NaN, infinity and out-of-range values through. The consumer copies both values into the audio generation input unconditionally and only guards its separate sampler assignment with "> 0", a test NaN also fails, so a NaN reached llama.cpp with the guard never firing. top_k must now be >= 0 and top_p must fall within 0.0 to 1.0 inclusive, with the bound written as a negated in-range test so NaN is rejected rather than silently accepted. Also cover the two checks the suite could not previously kill: the whole-string check in the float parser and the int32 range check. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump pin to f9e832c10 and carry the TTS server task Picks up ggml-org/llama.cpp#26254 (Qwen3-TTS via mtmd) and #26536 (the short-input audio chunk fix). Adds 0002-add-server-task-type-tts.patch, the server-side half of the still-draft #26603, so TTS runs through the slot scheduler instead of racing it. Remove that patch when #26603 merges. The patch is rebased on top of the score patch: its tokenize-switch hunk collided with the SERVER_TASK_TYPE_SCORE case, and its lone SRV_WRN call passes no variadic argument, which the macro cannot expand. The score patch itself needed no refresh. Also fixes fallout from the bump in grpc-server.cpp: upstream dropped the per-slot n_ctx argument from server_schema::eval_llama_cmpl_schema. Only the schema branch loses it, since forks predating the server-schema split still expect the old argument list. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): implement the TTS and TTSStream RPCs Both were declared in backend.proto but unimplemented. They now submit a SERVER_TASK_TYPE_TTS task and drain the response reader, the same shape PredictStream uses. The streaming path emits a leading sample_rate message and then raw PCM, because ModelTTSStream builds the WAV header itself; the non-streaming path emits a complete WAV to the requested dst. The streamed samples are converted from the pipeline's float32 to signed 16-bit first. MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM hands back floats, while the header ModelTTSStream writes announces 16-bit samples, so shipping the floats verbatim would decode as noise. prepare.sh and CMakeLists.txt now stage tts_request_options.h alongside the other grpc-server helpers, and register its standalone test with ctest the way passthrough_options_test is registered. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): mask non-codec tokens for Qwen3-TTS generation The Qwen3-TTS gen-audio pipeline maps a sampled backbone token to a codebook row with an unchecked subtraction, in mtmd-helper-gen.cpp: inp.code0 = sampled - codec_0; For ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF the vocab is 155008 tokens, <|codec_0|> is 151936 and the codec codes end at 153983. The model's own tokenizer.ggml.suppress_tokens holds 1023 ids covering 153984..155007, every special above the codec range except <|codec_eos_token|> (154086) which stays reachable as the stop token. Nothing masks the text range 0..151935, so the backbone can sample a text token at any step, the subtraction goes negative, and ggml_compute_forward_get_rows aborts the whole backend process on GGML_ASSERT(i01 >= 0 && i01 < ne01). Complete the mask upstream started: bias every token below <|codec_0|> to -INFINITY for TTS tasks so only codec codes and the codec EOS remain reachable. The biases are appended to task.params.sampling.logit_bias, which common_sampler_init already merges with the model's suppress tokens into one llama_sampler_init_logit_bias, so no sampler is added to the chain. Measured cost is 0.082 ms per sampled token and 1.16 MB, set against a forward pass in the multi-millisecond range. It lands in launch_slot_with_task rather than in a route handler so that llama.cpp's own POST /tts and LocalAI's TTS/TTSStream RPCs are both covered, and <|codec_0|> is resolved from the vocab rather than hardcoded so a model without it is left alone. This is reproducible with upstream's own llama-tts and no LocalAI code loaded, aborting at frame 55 on Q4_K_M and frame 71 on Q8_0, so it is neither a quantization artifact nor an artifact of the gRPC adapter. Two further defects in the same draft pipeline still prevent end-to-end audio; they are independent of this one and are recorded in the task report for an upstream bug report. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump pin to 9de0fcf2b and drop the TTS codec mask Upstream fixed the Qwen3-TTS abort in ggml-org/llama.cpp c8e03ce81 ("mtmd/ggml: add ggml_build_forward_order", #26649), landed one hour after the previous pin. ggml_build_forward_expand marks a tensor and all its ancestors for compute, so using it as a pure ordering hint defeated ggml_build_forward_select and made GEN_WAV calls execute the GEN_CODE branch against a stale inp_code0, hitting the get_rows bound assert in ggml_compute_forward_get_rows. That single defect accounts for every abort seen on this model, so 0003-mask-non-codec-tokens-for-tts.patch is removed rather than rebased. The mask changed the observed behavior, but it was perturbing a graph ordering bug rather than fixing a sampling one: at the new pin the whole path works without it. Keeping it would have meant carrying a 152k-entry logit bias, and rebasing it on every pin bump, for no benefit. Verified at 9de0fcf2b with only 0001 and 0002 applied, which both apply clean with no fuzz and needed no rebase: non-streaming HTTP 200, 410924 bytes, 8.56 s RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz streaming HTTP 200, 560684 bytes, 11.68 s, exactly one RIFF at byte 0, same format, which also exercises the float32-to-s16 conversion at runtime for the first time Pristine unpatched llama-tts at the same pin now also completes, 130 frames to a valid WAV, where it aborted at frame 55 before. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): clear the TTS slot sequence between requests Only the first TTS request in a backend process succeeded. Every later one failed instantly, in about 0.13 s, with "TTS prompt processing failed" from step_prompt, regardless of streaming or non-streaming and regardless of the text. With LOCALAI_SINGLE_ACTIVE_BACKEND=true the process is kept alive between requests, so a deployment would have served exactly one utterance per backend start. The cause is missing KV hygiene, not anything in the gRPC adapter. TTS slots never enter the shared batch: pre_decode() returns early for them and process_tts_slots() drives them instead, so they skip the prompt-cache bookkeeping that clears a slot's sequence between requests. Nothing in the gen-audio path makes up for it: mtmd_helper_gen_audio_reset only clears host-side buffers, and the pipeline always decodes from position 0 into the sequence identified by slot.id. So the second task on a slot writes positions 0..N over the first task's tokens and llama_decode fails. Fix is one call to slot.prompt_clear(), the same helper the normal path uses, in the SERVER_TASK_TYPE_TTS branch of launch_slot_with_task before set_input. It goes into 0002 rather than a new patch file because it is a defect in the code that patch introduces, and the header now records it as ours so we know whether it still needs carrying if #26603 merges without it. Verified in one backend process, different text on every request: three consecutive non-streaming requests, three consecutive streaming requests, and an interleaved non-streaming, streaming, non-streaming, streaming run. All ten returned HTTP 200 with RIFF ... WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz, the streamed ones carrying exactly one RIFF header at byte 0, and every output measured as real speech rather than silence or a truncated fragment. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): expose max_frames for TTS requests The Qwen3-TTS backbone does not always emit <|codec_eos_token|>, and when it does not, generation runs to upstream's 512-frame n_predict default. At the model's 12.5 Hz frame rate that is 40.96 s of audio, which a short input can trigger: one request in this session produced 40.96 s for a ten-word sentence. prepareTTSTask hardcoded n_predict to -1, so callers had no way to bound it. Add a max_frames key alongside top_k and top_p, parsed with the same strict whole-string parsing so a typo is an error rather than a silently truncated value, and rejected with a field-naming message when negative. 0 keeps the existing sentinel convention and means unset, so a request that omits it behaves exactly as before. Named max_frames rather than n_predict because frames are what the parameter means at a TTS endpoint: one frame is 0.08 s of audio. The 512-frame default is deliberately unchanged. Lowering it would truncate legitimately long inputs, which is a worse failure than an occasionally overlong one. Verified end to end on one text of thirty words: max_frames=25 HTTP 200, 96044 bytes, 2.00 s, exactly 25 frames max_frames=50 HTTP 200, 192044 bytes, 4.00 s, exactly 50 frames no max_frames HTTP 200, 572204 bytes, 11.92 s, stopped at its own codec EOS after 149 frames, unchanged behavior max_frames=-1 InvalidArgument "max_frames must be >= 0, got \"-1\"" max_frames=many InvalidArgument "max_frames must be an integer, got \"many\"" Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): send the TTS sample rate up front, and tidy three review items Four items from the Task 4 review. Streaming first-byte latency. TTSStream sent the sample-rate reply only once the first audio result arrived, and a chunk needs a whole 72-frame window, roughly 5.8 s of audio and far longer in wall time on CPU. The Go side blocks on that reply before it can emit the WAV header, so a streaming client sat at zero bytes for the whole stretch. The rate is a property of the loaded model and is available synchronously from mtmd_gen_audio_get_info, so it now goes out immediately after post_task and the rate_sent bookkeeping is gone. Measured on a warm model, first byte drops from 30.48 s to 0.014 s, and the output is still a valid WAV with exactly one RIFF header at byte 0. Unchecked close. The non-streaming path ignored ofstream::close(), so a failure that only surfaces on flush was reported as success while leaving a truncated file at dst. It now returns INTERNAL like the other write failures. Wrong comment on set_lang. gen_audio::inp::get() already maps a stored blank to nullptr, so our guard is behavior-preserving, not behavior-fixing. The comment claimed otherwise; the code was right. Repetition penalty. penalty_last_n = -1 is inert at this pin, because llama_sampler_init_penalties clamps it with std::max(penalty_last_n, 0) and then builds a disabled sampler, so the 1.05 penalty never applies. Upstream's README attributes looping to a missing repeat_penalty, so it was worth testing as a root-cause fix for the model running to the frame cap. Dropping the line lets the sampling default of 64 apply, which was confirmed in the sampler chain trace as penalty_last_n = 64 with repeat_penalty = 1.050. Over 15 uncapped short requests each way it did not help: 0 of 15 ran to the cap with the penalty inert, 1 of 15 with it active. Both lines are therefore kept for parity with upstream's draft, and a comment now records that the pair is inert and why, so the next reader does not believe a penalty is applied. max_frames remains the way to bound output. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(llama-cpp): let unpatched forks opt out of the TTS task turboquant and bonsai copy grpc-server.cpp into llama.cpp forks that do not carry our patches. disable-tts-task.sh injects the same kind of preprocessor switch disable-score-task.sh already uses, so those builds answer UNIMPLEMENTED rather than failing to compile. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): keep a TTS speaker-encoder projector out of vision detection Task 1 exempted a declared-TTS model's mmproj from VisionSupported, but the first real gallery entry with an mmproj still came back vision-capable through two paths the earlier fix did not close. GuessUsecases has no FLAG_VISION branch, so it falls through to true for any chat-ish model. That is not just a wrong answer at the call site: syncKnownUsecasesFromString rewrites KnownUsecaseStrings from HasUsecases, and the loader calls it more than once per config file, so the guessed FLAG_VISION is written out and parsed back into KnownUsecases as if the operator had declared it. Give GuessUsecases a FLAG_VISION branch that defers to the same explicit signals VisionSupported uses. Second, llama.cpp builds an mtmd context for the speaker-encoder projector and reports its media marker on the first chat probe, which resurrected vision after the model had been used once. Apply the same declared-TTS exemption to MediaMarker that the mmproj check already had. Verified against the qwen3-tts-llamacpp-q4 gallery entry: no vision capability and no image input modality, before load, after a TTS request, and after a chat probe. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add Qwen3-TTS entries for the llama-cpp backend Two entries over upstream's own GGUF conversion, Q8_0 and Q4_K_M, each pairing a backbone with the Q8_0 projector. Named to sit alongside the existing qwen3-tts-cpp entries rather than replace them. Also tags the llama-cpp backend text-to-speech / TTS so the backend browser surfaces the capability. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: cover Qwen3-TTS on the llama-cpp backend Adds the gallery variants, the two-file mmproj configuration, the required voice reference, and the language and sampling knobs. Also corrects the streaming-support list, which named only voxcpm. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): register llama-cpp as a TTS and voice-cloning backend The branch taught the llama-cpp backend to serve Qwen3-TTS and shipped two gallery entries for it, but never told the capability table. llama-cpp still declared only the text RPCs and usecases, so: - VoiceCloningForModel returned nil at the capability check, before it ever reached the model's own tts.voice_cloning override, and /tts answered 400 "selected model does not support reference-audio voice cloning" for any localai://voice-profiles/... voice. No model YAML could opt back in. - GET /api/backends/usecases did not list tts for llama-cpp, so the gallery greyed out the TTS filter for the entries this branch adds. - The React TTS page saw voice_cloning: null and kept both models out of the Voice Library. Add the TTS RPCs and usecase, and the reference-audio contract. The contract needs narrowing, because the per-backend switch in VoiceCloningForModel ends in a permissive default: an unnarrowed entry would have advertised reference-audio cloning on every GGUF chat model in the gallery. Narrow on the declared TTS usecase rather than the model name. The TTS checkpoints are the only llama-cpp models carrying known_usecases: [tts]; name matching would have to guess at third-party repacks, and "base", the substring the neighbouring Qwen and vLLM cases key on, is a routine word in text-model names. The check reads the declared bit directly instead of going through HasUsecases, which falls through to GuessUsecases and would hand the decision to a heuristic that never had a llama.cpp TTS model in mind. DefaultUsecases stays [chat]: a bare GGUF served by llama.cpp is a chat model, and both the gallery filter and the importer read that field. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): declare what nemotron-3-nano-omni actually accepts The entry is backend: vllm-omni with known_usecases: [chat, completion], no mmproj and no media marker, so it used to report vision only through the blanket GuessUsecases fallthrough that the vision branch in this branch removed. Nemotron 3 Nano Omni is a multimodal understanding model: image, video and audio in, text out. Declaring that is what the sibling vllm-omni-qwen3-omni-30b already does. known_usecases gains vision only. FLAG_VIDEO is video GENERATION, an output modality, and this model generates none; video and audio input belong in known_input_modalities, which is where AudioInputSupported and VideoInputSupported read them from. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(importers): import a Qwen3-TTS GGUF repo as TTS, not chat The llama-cpp importer hardcodes known_usecases: [chat] and assigns any mmproj-matching file as a vision projector, so ggml-org/Qwen3-TTS-12Hz-1.7B- Base-GGUF imported as a chat model with vision. Both fields were wrong, and the model was unreachable from /tts and from the Voice Library. Filenames cannot fix this. A Qwen3-TTS repo has the exact shape of a vision repo, one backbone GGUF plus one mmproj-*.gguf, so the projector's own header is the only honest signal: mtmd writes clip.has_gen_audio_encoder for the projectors it can drive as a speech pipeline and refuses to build one without it. Probe the selected mmproj for that flag, reusing the range-fetch the MTP detection already does, and declare tts when it is set. The mmproj assignment then stops reading as vision on its own, since a declared-TTS model already exempts its projector from vision detection. The probe is best-effort like the MTP one: a network blip leaves the chat default in place rather than failing the import. Verified against the real artifacts on disk: the Qwen3-TTS projector reports gen-audio, its backbone does not. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): stop non-TTS models crashing on the new pin Two regressions, both hit every ordinary llama-cpp model and neither was caught locally because every test on this branch loaded a TTS model. The first is a null dereference. server_slot::tts_ctx::reset() called mtmd_helper_gen_audio_reset() unconditionally, but the gen-audio pipeline is only allocated for models carrying a gen-audio mmproj, and upstream's implementation reads ctx->pipeline before null-checking anything. Since server_slot::reset() runs during slot initialization for every model, any non-TTS model segfaulted the backend the moment it loaded. Guard the call on the is_supported() predicate already defined beside it, and keep the plain field resets unconditional. The second is unrelated to TTS and came in with the pin bump. PredictOptions.Penalty is a bare proto float, so a caller that names no repetition penalty sends 0 rather than omitting the field. Since 9de0fcf2b, common_sampler_init() rejects a non-positive penalty_repeat outright because it would divide logits by zero, turning every such request into "Failed to initialize samplers". Treat 0 as unset and leave llama.cpp's own neutral default in place. Verified with the same suite CI runs, which is what caught both: tests/e2e-backends passes 6 of 6 including the load and predict specs that were red. Qwen3-TTS still synthesises on both paths, 24 kHz mono 16-bit WAV with exactly one RIFF header on the streamed output. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-08-10 10:18:47 +02:00