From 2cadf319a0e1a91030867b20c5bf069bd8c6bb3e Mon Sep 17 00:00:00 2001 From: mudler-agent Date: Thu, 29 Oct 2026 22:06:41 +0100 Subject: [PATCH] feat(realtime): LFM3-Audio speech-to-speech on llama.cpp, remove liquid-audio (#12958) * chore(backends): remove the liquid-audio Python backend The backend targets liquid-audio, which loads LFM2 checkpoints only. Remove its directory, the gallery file and the LFM2.5-Audio model entries, the library and generated backend gallery entries, the backend matrix and CI job, the Makefile targets and the golden test fixtures that listed it. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * refactor(config): drop liquid-audio from capabilities, importers and UI With the liquid-audio backend gone, remove its capability entry, its importer, its artifact-gate entry and the realtime_audio heuristic that only matched that backend. realtime_audio is now set only through known_usecases. Update the Talk and fine-tuning pages, the docs and the swagger text that named the backend. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * chore(llama-cpp): bump llama.cpp to e486a4b5 Move the llama.cpp pin to e486a4b559617dc88bf79978996f6c670b6bbc9d, the master head of 2026-10-28. It contains the LFM3-Audio support (ggml-org/llama.cpp#31257) and the realtime server split (ggml-org/llama.cpp#31269). The score task patch is regenerated against the new pin. It applies with git apply and with patch -p1, with no offset and no fuzz. Its hunks in server-task.h and server-context.cpp were rebased by hand next to SERVER_TASK_TYPE_REALTIME and the control dispatch. The C++ backend was built on CPU for this change: the llama, mtmd and server-context libraries and the grpc-server binary link. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * feat(llama-cpp): compile server-realtime.cpp into grpc-server server-context.cpp includes server-realtime.h, and its realtime slot type is defined in server-realtime.cpp. Upstream compiles that file into the server library. The gRPC server is one translation unit, so it must include the file too, or the link fails. The include is guarded by __has_include. Forks that predate the realtime split do not ship the file, so they keep building as before. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * chore(llama-cpp): probe realtime and TTS voice support in prepare.sh prepare.sh writes llama_compat.h from probes on the checkout. Add two: LOCALAI_HAS_REALTIME (SERVER_TASK_TYPE_REALTIME in server-task.h) and LOCALAI_HAS_TTS_VOICE (set_voice in mtmd-helper.h). The probes let the gRPC server choose the realtime and voice code paths per checkout, so forks without them keep compiling. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * feat(llama-cpp): advertise speech recognition and speech-to-speech llama-cpp now registers the AudioTranscription and AudioToAudioStream RPCs, with the transcript and realtime_audio usecases. The default usecase stays chat, so a plain GGUF is still a chat model. The backend implements the RPCs in a follow-up change; until then the calls return UNIMPLEMENTED. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * feat(config): add pipeline.any_to_any for speech-to-speech models A realtime_audio model that serves the whole speech loop itself sets pipeline.any_to_any. The flag is off by default, so chained pipelines and existing realtime_audio configs are unchanged. The realtime_audio heuristic stays explicit: a llama-cpp model is never a Talk-page model unless known_usecases says so. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * feat(config): read the audio encoder and speech-to-speech projector flags Add HasAudioEncoderProjector for clip.has_audio_encoder and HasSpeechToSpeechProjector, which requires the LFM3 audio encoder family together with the gen-audio decoder. Qwen3-TTS sets both encoder flags too, so the flags alone cannot tell it apart from LFM3. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * feat(importer): import LFM3-Audio projectors as speech-to-speech An mmproj that carries the LFM3 audio encoder and the gen-audio decoder now imports with known_usecases realtime_audio, chat, transcript and tts, pipeline.any_to_any: true, the parallel:2 option and the tokenizer template. Other gen-audio projectors keep the TTS-only result, Qwen3-TTS included. The probe runs against a local HTTP server in the tests. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * feat(llama-cpp): named-voice TTS and the LFM3 ASR prompt Models without a speaker reference (LFM3-Audio) take a named voice through their "tts" chat template instead of a reference file. The request is validated with needs_reference=false, the prompt is rendered with common_chat_get_tts_prompt(), and the voice is set on the gen-audio input. Qwen3-TTS Base keeps the reference-file rule. The branch is guarded by LOCALAI_HAS_TTS_VOICE, so forks without named voices keep the old behaviour. Transcription takes its prompt from common_chat_get_asr_prompt(). LFM3 gets a system turn that names the language and an empty user turn. Qwen3-ASR's preset is unchanged: fixed user text, no system turn, and a caller prompt is still appended after a newline. The preset call is guarded by a new LOCALAI_HAS_ASR_PRESET probe. The pure parts sit in tts_request_options.h and transcription_prompt.h, with standalone tests run by backend/cpp/run-unit-tests.sh. Add the test-extra-backend-llama-cpp-lfm3 target. It runs the tts, ttsstream and transcription specs against LFM3-Audio GGUF weights. It is a manual run and is not wired into CI. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): mock AudioToAudioStream and pending realtime S2S specs The mock backend serves AudioToAudioStream deterministically: session created, commit with RMS, a tool call on the first turn with tools, then transcript pieces and paced 24 kHz tone frames, with response.cancel ending a reply as cancelled. The realtime S2S specs cover the mock and real-weights variants. They skip until the /v1/realtime S2S path lands, so the mock suite stays green in CI. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * feat(llama-cpp): speech-to-speech AudioToAudioStream on the realtime slot Implement AudioToAudioStream on upstream's realtime slot, so a model that answers after the audio (LFM3-Audio) can hold a conversation over the existing duplex RPC. The proto schema does not change: tools travel in the config, tool calls leave as function_call_arguments.done and tool results arrive as a function_call_output item. realtime_session.h keeps what needs no model: the turn text derived from the chat template (with a tool turn), the conversation state, a filter that keeps blocks out of the transcript and drops audio produced inside them, and the translation of slot results into events. grpc-server.cpp wires it to the slot with control tasks and parses calls with the parser the chat template selects. The unit test uses a fake renderer. A ctest-only check renders the real LFM3 template and compares the slot text with the full render. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * feat(gallery): add the LFM3-Audio-2B entries Two llama-cpp entries for LiquidAI/LFM3-Audio-2B-GGUF, the Q4_K_M default (lfm3-audio-2b) and the Q8_0 build (lfm3-audio-2b-q8), both with the Q8_0 audio projector. Each declares pipeline.any_to_any and the realtime_audio, chat, transcript and tts usecases the importer writes for an LFM3 projector, with parallel:2 and the tokenizer template. The Q4 entry lists the Q8 one as a variant. File sizes and sha256 come from the Hugging Face API for the pinned repo revision. Speaker reference files and the VAD model are not part of the entries: VAD stays optional. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * docs: describe LFM3-Audio speech-to-speech and its gallery entries Adds a speech-to-speech section to the Realtime API page (any_to_any configuration, VAD versus manual commit, tools, transcripts, barge-in, limits), a pipeline.any_to_any row, a llama.cpp speech-to-speech row in the compatibility table, and LFM3-Audio mentions on the TTS and ASR pages. The feature is marked experimental: the behaviour follows the backend session contract and has not been run with real weights. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * feat(realtime): run any-to-any models on AudioToAudioStream A realtime_audio model with pipeline.any_to_any now opens one duplex stream to the backend and keeps it for the session, replacing the "any-to-any models not implemented" branch. The chained pipeline is unchanged. The endpoint forwards input audio (with a pre-roll from speech start when a VAD model is set, otherwise every chunk and a client commit), sends response.create, and turns the backend's events into the usual realtime events: audio and transcript deltas, tool calls through the existing function_call items, and a single response.done. A function call output is forwarded before the next response. Barge-in sends response.cancel ahead of the new speech and stops forwarding audio at once. The user's speech is transcribed on the side and reported on the user item; a failure there does not touch the response. Server VAD is on only when pipeline.vad is set. Classifier mode, compaction, streaming.llm, voice gating, sound detection and semantic_vad are rejected for these models. The mock backend now matches the llama.cpp contract (session.updated, the cancel error when idle) and its VAD reports the loud stretches of the audio, so the S2S e2e specs run a VAD turn. The specs are enabled. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): add the real-weights realtime S2S variant and the gRPC s2s spec The realtime S2S specs now check reply audio on real models: duration, loudness, clipping, and keywords found in the model transcript or in a separate ASR of the output audio. run-realtime-lfm3.sh stages the llama-cpp and sherpa-onnx backends, the LFM3-Audio files and the sherpa VAD, ASR and TTS models, then runs the real variant. It is manual and never part of CI: `make test-realtime` still filters out real-models. The backend suite gains an s2s capability (spoken turn, tool call, tool result, spoken answer over AudioToAudioStream) and optional TTS duration and RMS bounds plus a transcript keyword check, all enabled for test-extra-backend-llama-cpp-lfm3. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): expect the idle-cancel error from the mock S2S The mock now answers a response.cancel with no reply running with response_cancel_not_active, as the llama.cpp backend does. The cancel spec still treated that cancel as a no-op and read the error as extra data after the stream ended. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * feat(talk): show any-to-any models as one speech-to-speech model /api/pipeline-models lists a pipeline.any_to_any model with any_to_any set and its optional VAD. A realtime_audio model that is not any-to-any is listed only when it has the four pipeline slots, since the session takes the chained path for it. The Talk page shows a speech-to-speech model as one model: no four-slot view, the VAD slot only when a VAD model is set, and the female and male voices offered. Without a VAD model the server does not detect turns, so the page offers Done speaking, which sends input_audio_buffer.commit and then response.create. Text-only replies stream through response.output_text.delta and .done. self_contained stays in the response for clients built before any_to_any. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): refuse a llama-cpp realtime model without pipeline.any_to_any llama.cpp has no VAD, transcription or TTS RPCs, so filling the empty pipeline slots with the model name failed later with UNIMPLEMENTED. Fail at session start and name the missing setting instead. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): hold manual-commit S2S audio until the commit In manual-commit mode the endpoint streamed every appended chunk to the backend, so input_audio_buffer.clear only emptied the Go buffer while the model kept the audio, and idle audio filled its context between turns. Keep the audio in the input buffer and send it, behind pending text and tool-result items, when the client commits or asks for a response. Frames and controls now go through one ordered write queue, so no gRPC send happens under the audio buffer lock. A tool result created while a VAD turn is open is refused with a clear error. When the stream ends outside a response the session reports it and closes, and a bare EOF from Send reports the status that Recv holds. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): drop the S2S readiness gate and stale comments The gate was always true. Some comments referred to work items that readers of the repository cannot see. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * docs(realtime): add the liquid-audio migration note and the clear semantics Say that the Python liquid-audio backend is gone, which gallery entries replace it and that LFM2.5-Audio has no replacement. Describe what input_audio_buffer.clear drops for speech-to-speech sessions now that manual-commit audio is held back until the commit. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(talk): do not promise turn detection when no VAD model is set The idle note said the server notices when you stop talking. A speech-to-speech model without a VAD model needs Done speaking, so the note now says that. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): make the mock S2S backend keep the llama-cpp session limits The mock accepted what the real backend refuses and hid two endpoint bugs. It now treats output_modalities from the config as a fixed session mode and picks the default per response from the tools set at that time. A fixed text plus audio mode with tools reads the tool call out loud. After the first commit or message, session.update of instructions, voice or tools is an error that does not belong to a response, committing an empty buffer is an error, and response.create while a reply runs is refused. Errors of a refused response.create carry response_scoped in their meta. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): leave the S2S output mode to the backend unless the client sets one The stream config sent text plus audio when the session had no tools yet. The llama-cpp backend keeps a mode from the config as a fixed session mode, and tools added later by session.update never cleared it. The model then read tool calls out loud. Send output_modalities only when the client chose them. The backend picks the default of each response from the tools it has when the response starts. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * feat(llama-cpp): mark the realtime errors that answer a response.create The error event now carries response_scoped: true when it ends the request for a response (a refused response.create, a failed commit made by it, or a stream failure). Other errors, such as a refused session.update or a commit of an empty buffer, are not about a response and one that runs keeps generating. The client needs the difference to know whether a response.done is still coming. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): end an S2S response only on the errors that belong to it Every backend error ended the response in flight, although the backend also reports errors about other things (a refused session.update, a commit of an empty buffer) and goes on generating. The rest of the reply was lost and the next response.create met either a refusal or the leftover events of the old reply. End the response only on an error marked response_scoped, forward the others to the client and keep waiting for response.done. Drop the events buffered by an earlier response when a new one starts, and skip the commit of a turn that forwarded no audio. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): run the S2S input transcription detached from the commit slot The commit waited for the side transcription of the user audio while it held the commit slot, and response.create waits for that slot. The ASR latency was added before every reply, and on a backend with few slots a busy ASR task could stall the reply. The model already has the audio, so run the transcription detached under the session context and release the slot at once. The transcript is still reported on the user item. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(realtime): do not pin the place of the input transcript in the S2S order The transcript of the user audio is reported on the side and may land before or after the first response events. Check the order of the response events without it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): stop an S2S reader leak and a cancel that waits on the write queue The reader waited on a channel it closes itself when the events buffer was full, so it outlived the session after a turn ended on an error or timed out. It now also stops when the session closes. Stopping a response sent the cancel through the write queue and waited for it before the grace timer started. With backpressure on the stream the wait never ended and nothing read the events meanwhile. The timer now starts first and the cancel is only queued. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * feat(realtime): let a chained pipeline run without a VAD model A pipeline that names no pipeline.vad failed to build, because the VAD slot was always loaded, and a new session still started with server_vad. The slot is now optional for chained pipelines as it already is for any-to-any models: the session starts with turn detection off and the client commits each turn. Turning server_vad on for such a session is refused, and a switch to a pipeline without a VAD drops turn detection. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * feat(gallery): add LFM3-Audio VAD and chained-pipeline entries Add lfm3-audio-2b-vad and -q8-vad (any-to-any with silero-vad-ggml as the turn detector), and lfm3-audio-2b-pipeline, -pipeline-vad and the -q8 forms (the model chained through its own transcription, chat and speech slots, with and without the VAD). The gallery has no dependencies between models, so the VAD entries state that silero-vad-ggml must be installed first. Drop the variants list from lfm3-audio-2b: it let the installer pick the larger Q8 build on any host with enough memory, so a plain install of the Q4 entry did not get Q4. All entries are now independent. Tests cover the entry shapes, the single-build install and their acceptance by the realtime endpoint. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): run the realtime specs for any-to-any and chained shapes The realtime specs covered one any-to-any model with a VAD. They now run for any-to-any and for a chained pipeline, each with and without a VAD, on the mock backend in CI. VAD-less shapes also check that turn detection starts off and that server_vad is refused. The manual real-weights script writes one config per shape that mirrors the gallery entries and runs the four shapes in turn (LFM3_MATRIX picks a subset). Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * docs(realtime): document the LFM3-Audio VAD and chained-pipeline entries List the eight LFM3-Audio gallery entries, explain that silero-vad-ggml must be installed before the -vad entries, describe the chained pipeline on one model and the behaviour of a pipeline without a VAD, and describe the shape matrix of the e2e specs. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(realtime): wait for the detached input transcription in the S2S specs The input transcription runs in its own goroutine, so the transcription completed and failed events can arrive after the response ends. Poll for them instead of reading the event list right after the response. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): release the S2S stream before running server-side tool calls A response held the stream slot until it returned, and it ran the tool calls from inside that scope. A server-side tool starts a follow-up response on the same stream, which then waited for the slot its parent still held, so the client never got response.done. Run the tool calls once the slot is free; the event order is unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): end an S2S session whose stream dies, and stop queuing after close A stream that died during a response left the session open: later requests failed one by one and VAD frames kept being forwarded. The response now ends the session through the transport, and enqueue drops requests once the session is closed so the write queue cannot grow. The writer is now the only caller of Send and of CloseSend. close() used to call CloseSend while the writer could be inside Send, which gRPC does not allow. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): close the open S2S turn when the VAD stops Switching turn detection to manual in the middle of a VAD turn left the turn forwarding, and the next client commit sent the whole buffer a second time. Stopping the VAD now commits the forwarded audio on the backend and drops it from the input buffer. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): keep a tool result pending while an S2S user turn is open The check that refuses a tool result during a user turn ran when the item arrived, but the item was forwarded later, and the result was marked as sent before the backend took it. If the user spoke in between, the backend refused it and it was never retried. The result is now claimed and queued under the audio lock only when no turn is open, and otherwise stays pending for the next commit or response. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(talk): empty the input buffer for models without a VAD Without voice detection the microphone fills the server's input buffer for the whole session, so Done speaking committed idle silence and the echo of the previous reply. Clear the buffer when the session starts and when a reply ends. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(importer): keep an existing parallel option on LFM3-Audio imports The speech-to-speech import added parallel:2 even when the options already carried another parallel value. Use the same check as the hardware defaults, which also covers n_parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(talk): list chained pipelines that have no VAD /api/pipeline-models required a VAD for a chained pipeline, so the VAD-less LFM3-Audio pipeline entries never showed up on the Talk page. Any pipeline with transcription, LLM and speech slots now qualifies. The Talk page treats every model without a VAD as manual-turn, and the session sheet says so for chained pipelines too. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * chore(llama-cpp): bump llama.cpp to 0ac1a7fd Move the llama.cpp pin to 0ac1a7fd35c127a18c0cbc6c57d12e256a6a385b, the master head of 2026-10-29. It contains tool calls in the realtime server (ggml-org/llama.cpp#31287). The score task patch is regenerated so that it applies with no offset and no fuzz. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(functions): keep a null tool argument null A tagged parameter whose text was "null" came out as an empty string: unmarshalling JSON null into a string succeeds and leaves it empty, so the value was taken for a string. Check the first byte of the JSON value instead. A parameter with the text "null" now becomes JSON null, as in llama.cpp. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * feat(functions): add an lfm3 XML format preset The LFM3 chat template writes a call as , , one block per argument, then the closing tags. The qwen3-coder format has the same shape and auto-detection already found it, but xml_format_preset: lfm3 was not a known name. Add the preset, with tests for one call, a call without arguments, two calls, string and non-string values (numbers, booleans, null, arrays, objects), a call between text, and unfinished input. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * feat(realtime): parse tool calls with the chat template, as upstream Since ggml-org/llama.cpp#31287 the realtime server handles tools with the parser and grammar of the chat template. Do the same in the speech-to-speech stream instead of a hard-coded tag filter and a parse of the whole text at the end. - Each response gets a common_chat_session with the tools of the session and parallel calls. Its grammar and preserved tokens go into the sampling of the response, so the slot keeps the call tokens in the text and constrains the call. - The text of the response is fed to the session piece by piece (task_result_state::update_chat_msg). Content deltas become the transcript; tool-call deltas are collected into one function_call_arguments.done per call at the end. Audio after the start of a call is dropped. - A tool_choice of auto, none or required is accepted in the params of the config, in session.update and in response.create, with the response overriding the session. - The tool turn is derived from a conversation in which the assistant called a tool, like upstream, not from a plain reply. - response.done carries raw_text, everything the model wrote with the call markup, for a caller that parses it another way. If the parser throws, the rest of the response is content and the raw text is the way out. ToolCallFilter is no longer used and goes in the next commit. The unit tests cover the splitter contract (content, call deltas, the final piece, whole characters only). realtime_chat_check feeds LFM3-format text through the session in pieces of 1, 7 and all bytes and checks the call, the transcript and the grammar. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * refactor(realtime): drop the tool-call text filter The parser of the chat template now splits content from tool calls, so ToolCallFilter, which looked for the literal tags, has no caller. Remove it with its tests. Every case it covered (a call between text, split tags, several calls, an unfinished call, text after a call) is handled by the parser, and realtime_chat_check runs the LFM3 format through it in pieces of different sizes. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * feat(realtime): fall back to the raw text for S2S tool calls For an any-to-any model, the calls the backend parsed still win. When there are none and the request had tools, parse the raw text that response.done now carries the way the chained pipeline parses the text of its LLM (CleanupLLMResult, ParseFunctionCall, ParseTextContent), so a call format the backend parser does not know can still be served with function.xml_format_preset or the other function settings of the model. The content around the calls replaces the streamed transcript only if the config captures it, as in the chained path. Also pass tool_choice to the backend: the session mode with the stream config and with session.update, and the mode of a response.create override with that event. A named function cannot be forced, so it is not sent. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): carry raw_text in the mock S2S response.done Match the llama-cpp session: with tools set, response.done has raw_text, the call markup of the LFM3 template for a tool turn and the transcript for any other reply. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(functions): parse tool blocks that have text between them The iterative XML parser stopped at the first scope block that was preceded by text, so "A then B" gave one call. Skip text between blocks once a call was found; text before the first block still ends the parse as before. Blocks that follow each other, or only whitespace, already worked. Tests cover three calls back to back and with text between them, with the lfm3 preset and with auto-detection. They also pin the current reading of argument values: the parser has no tool schema, so a value such as 42 or true is read as JSON even when the schema types it as a string. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * feat(gallery): read LFM3-Audio tool calls with the lfm3 preset Set function.xml_format_preset: lfm3 on the eight LFM3-Audio-2B entries and in the config the importer writes for a speech-to-speech projector, so the server parses the call format of the LFM3 chat template when the backend returns no call (and in the chained pipeline entries, which use the regular chat path). gallery/index.yaml is regenerated from the library. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * docs(realtime): describe how S2S tool calls are read Replace the description of the hard-coded tag filter with the parser and grammar of the chat template, tool_choice, and the raw-text fallback in the server. Document function.xml_format_preset, and set it in the real-weights e2e model configs like the gallery entries do. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(functions): keep earlier calls when a later block does not parse After a valid call, text that holds the opening tag again (a prose mention, a markdown example, a malformed second block) made the parser treat it as a new block, fail, and return no calls at all. Remember the state after each complete block and fall back to it when a later one fails: the calls found so far are kept, parsing stops there and the remaining text stays in the content. Text skipped between two valid blocks now goes to the content instead of being dropped, so StripToolCallMarkup keeps it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): do not drop audio after a tool call started In mixed mode the speech for text written before a call can still be pending when the call starts, so dropping every audio frame after the first call delta cut the speech while its transcript was sent. The llama.cpp server never drops audio when it reads a result; do the same. If a model speaks the call markup aloud, that is model behaviour to observe on real weights, not something to filter here. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): flush held-back text when a response is cancelled The parser got is_partial=true on a cancelled response, so text it held back (a possible start of a tag) never reached the transcript. The llama.cpp server parses a cancelled response with is_partial=false; do the same. A cancelled response still makes no tool call. The path where the parser throws already sends the text as content and needs no change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(realtime): stop the fake S2S stream from sending after Close The barge-in spec failed now and then with a data race and "send on closed channel": its script pushes audio from a goroutine, and the session closes the stream while that goroutine still runs. A push now does nothing once the stream is closed, and Close waits for a push in flight. The production close order is right: the writer closes the stream only after it stops sending. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): align the S2S tool-call fallback with the backend Four gaps in how the Go side reads the tool calls of an any-to-any reply: - With tool_choice none (the response value wins over the session value) the backend builds no tool parser, yet the raw-text fallback still turned a call the model wrote into a function_call item. Skip it. - Without a capture regex the fallback never cleaned the transcript, so the stored item and transcript.done kept the raw call markup. When the call is XML, use the text left after the markup is cut out; a reply that is only a call now has an empty transcript. - A named function as tool_choice mapped to nothing, so the backend kept the previous mode. Refuse it with an error in session.update and response.create. - A response-level tools list changed what the fallback looked for, but the backend only has the tools of the session. Use those in both. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * docs(realtime): say how S2S tool_choice and response tools behave Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(e2e): use the real llama-cpp backend in the real realtime run The suite always registered the mock backend under the name llama-cpp, so a run with REALTIME_S2S_MODEL set never reached the llama-cpp backend found in REALTIME_BACKENDS_PATH. Register the mock only when that run is off. The real run converts the input audio with ffmpeg. Check for it in the suite and in run-realtime-lfm3.sh before any download or backend start, with a message that says what to install, and list it as a prerequisite in the script header and the Makefile target. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(realtime): cover the tools path of a client session.update to the S2S backend A tool call that never came back from an any-to-any model raised the question whether the tools of the session reach the backend at all. Add a spec that sends the session.update of the real-weights e2e (flat Responses API tool, no session type) and checks that the backend gets the tool in the chat-completions form. Log the number of tools sent when the stream opens and when a session.update carries them, so a run log shows it without reading the wire. The "No tools received" line of the llama-cpp backend does not show it: the server prints it for the input transcription request, which has no tools by design. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): interrupt the chained pipeline reply while it is in flight With unary stages, the default of the lfm3 pipeline gallery entries, the chained pipeline synthesizes the whole reply and sends it as one audio delta right before response.done. The barge-in spec waited for that delta and then rendered the interruption with the TTS, so the first response was already done when the speech started and ended completed instead of cancelled. Real runs on a GPU showed it with both quantizations. For a pipeline, start the interruption at response.created, which comes before the LLM and TTS stages. Render the interruption audio before the first utterance is sent, so nothing sits between the reply and the interruption. The any-to-any shape still waits for the first audio delta, which that backend streams while it generates. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): accept a number word and its homophone in the spoken-answer check The check that the reply audio says what the transcript says needs a shared word of 4+ letters. A short answer failed it: the model wrote "Four" and the ASR heard "for". Compare numbers in one spoken form, so digits, number words and their homophones match each other. A homophone only counts when the other text has the number itself, so "for" on both sides still shares nothing and the check does not pass on a stop word. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(llama-cpp): ask LFM3-Audio for function calls when a realtime session has tools The default system prompt of a realtime session was "Respond with interleaved text and audio." whatever the tools. Under that prompt LFM3-Audio answers in words ("I'll check the weather ...") and never writes the call. The model card, and the llama.cpp realtime example client, use "Respond with function calls, and interleaved text and audio." when the session has tools, and the model then calls the tool. Pick the default by the tools of the session, and treat the plain default that the LocalAI server sends back as no choice: it follows the tools too, also when they arrive with a session.update before the first audio. A prompt the client wrote is kept as it is. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): transcribe the input of an any-to-any turn between responses The input transcript of a speech-to-speech turn ran at the same time as the reply, on the same model. llama.cpp serves both from one context: the realtime slot wants embeddings on, the chat slot of the transcript wants them off, and each flip makes it reserve its compute buffers again. In a GB10 run that was two reserves of about 175 ms per step, the reply fell to 2.6 tokens per second and the transcript to 380 ms per token, until the transcript was done. When pipeline.transcription is the model of the stream, run the transcript after the response, or before one that is not started yet, and keep responses out while it runs. A separate transcription model still runs in parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): send the input transcript of an any-to-any turn before its reply The previous change moved the same-model transcript after the response. A client that reads until response.done then never saw conversation.item.input_audio_transcription.completed: the transcript was made (the log shows it) but sent after the end of the window. Run it in line with the commit instead. A VAD turn transcribes and then responds. A client response.create already waits for the commit slot, so it starts once the transcript is out. The transcript event now comes before any output of the response of its turn, for a manual commit and for a VAD turn, and never overlaps the response on the llama.cpp context. A transcription model of its own still runs in parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * feat(realtime): transcribe the input of an any-to-any model only when asked The stream answers from the audio, so the transcript of the input is for the client only. On the model of the stream it is costly: llama.cpp serves the transcript request and the realtime stream from one context and rebuilds its compute buffers on every step while both are active. On a GB10 that was 26 to 29 reserves of about 175 ms per turn, against 0 to 1 on the llama.cpp server, and the reply ran at a few tokens per second. An any-to-any model no longer fills pipeline.transcription with its own name. Without it the session does not transcribe the input; with another model it does, in parallel with the reply; with the model itself named it does, before the reply. The reply transcript always comes from the stream. The gallery entries set none, so they get the fast path. The mock configs and run-realtime-lfm3.sh name a transcription model, so the real-weights spec still covers the input transcript. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * fix(realtime): end a response as cancelled when a barge-in lands during the LLM call The chained pipeline cancels the response context on a barge-in. When that happens while the model is generating, the backend call fails with the context error, and the response ended as failed: an error event and no response.done. A client that waits for the terminal of the interrupted response then waits forever. On a GB10 the q4 model answered before the barge-in arrived, so only the slower q8 model showed it: the spec read events for 30 s and timed out. When the context is cancelled, a failed Predict call or a failed result is a cancel, as the TTS step and the streamed LLM path already treat it. The response ends with response.done{cancelled} and no error. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): wait for the input transcript that comes from a model of its own An any-to-any model now gets its input transcript from a separate ASR that runs next to the reply. The reply is often over first, so a spec that stops reading at response.done saw no transcript in the VAD shape. Keep reading, up to 30 s, until the transcript arrives. The contract is that it comes, not when. Add a guard spec for what the input transcription is handed: the whole turn, in a WAV that says the rate it was recorded at (the session rate for a client commit, 16 kHz for a VAD turn). Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): check the errors of Close in the realtime specs golangci-lint (errcheck) flagged the unchecked Close of the websocket connections, the response body and the fake stream. Use the `defer func() { _ = x.Close() }()` form the other e2e specs use. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto * test(e2e): write the mock S2S specs with Ginkgo and drop the env read forbidigo only allows Ginkgo/Gomega specs and no os.Getenv outside the sanctioned places. Convert the mock S2S tests to Describe/It with a suite bootstrap (the package had none), with the same 11 cases. The mock no longer reads MOCK_S2S_FRAMES: only these specs set it, so the reply length is a package variable they change per spec and restore afterwards. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto --------- Signed-off-by: Ettore Di Giacinto Co-authored-by: Ettore Di Giacinto --- .agents/llama-cpp-backend.md | 10 + .github/backend-matrix.yml | 82 - .github/workflows/test-extra.yml | 27 - Makefile | 41 +- addons/fine-tuning/ui/src/FineTune.jsx | 33 - addons/fine-tuning/ui/src/i18n/en.js | 4 - addons/fine-tuning/ui/src/style.css | 4 +- .../ui/src/components/talk/SessionSheet.jsx | 29 +- addons/talk/ui/src/i18n/talk.en.json | 10 +- addons/talk/ui/src/pages/Talk.jsx | 51 +- backend/backend.proto | 25 +- backend/cpp/llama-cpp/CMakeLists.txt | 21 + backend/cpp/llama-cpp/Makefile | 2 +- backend/cpp/llama-cpp/grpc-server.cpp | 835 ++++++++- .../0001-add-server-task-type-score.patch | 54 +- backend/cpp/llama-cpp/prepare.sh | 34 + backend/cpp/llama-cpp/realtime_chat_check.cpp | 233 +++ backend/cpp/llama-cpp/realtime_session.h | 707 ++++++++ .../cpp/llama-cpp/realtime_session_test.cpp | 593 +++++++ backend/cpp/llama-cpp/transcription_prompt.h | 49 + .../llama-cpp/transcription_prompt_test.cpp | 83 + backend/cpp/llama-cpp/tts_request_options.h | 35 +- .../llama-cpp/tts_request_options_test.cpp | 99 +- backend/index.yaml | 506 ------ backend/python/common/model_utils_test.py | 1 - backend/python/liquid-audio/Makefile | 23 - backend/python/liquid-audio/backend.py | 867 --------- backend/python/liquid-audio/install.sh | 24 - backend/python/liquid-audio/protogen.sh | 11 - .../python/liquid-audio/requirements-cpu.txt | 13 - .../liquid-audio/requirements-cublas12.txt | 13 - .../liquid-audio/requirements-cublas13.txt | 13 - .../liquid-audio/requirements-hipblas.txt | 13 - .../liquid-audio/requirements-l4t13.txt | 13 - .../python/liquid-audio/requirements-mps.txt | 13 - backend/python/liquid-audio/requirements.txt | 3 - backend/python/liquid-audio/run.sh | 10 - backend/python/liquid-audio/test.py | 89 - backend/python/liquid-audio/test.sh | 11 - core/backend/audio_to_audio.go | 60 + core/config/backend_capabilities.go | 23 +- core/config/backend_capabilities_test.go | 20 + core/config/gguf.go | 41 +- core/config/gguf_audio_encoder_test.go | 94 + core/config/hardware_defaults.go | 4 + core/config/meta/registry.go | 7 + core/config/model_artifact_inference.go | 2 +- core/config/model_config.go | 21 +- core/config/model_config_test.go | 27 + core/gallery/backend_library_test.go | 6 +- core/gallery/importers/importers.go | 2 - core/gallery/importers/liquid-audio.go | 145 -- core/gallery/importers/liquid-audio_test.go | 91 - core/gallery/importers/llama-cpp.go | 26 + .../gallery/importers/llama-cpp_audio_test.go | 157 ++ core/gallery/lfm3_audio_entries_test.go | 143 ++ .../testdata/backend_library/digests.tsv | 16 - .../backend_library/gallery-before.yaml | 113 -- .../gallery/testdata/backend_library/pins.tsv | 14 - .../testdata/model_library/digests.tsv | 4 - .../http/endpoints/localai/audio_transform.go | 2 +- .../localai/audio_transform_internal_test.go | 2 +- core/http/endpoints/openai/realtime.go | 172 +- .../openai/realtime_chained_novad_test.go | 102 ++ .../endpoints/openai/realtime_conncoord.go | 2 + .../endpoints/openai/realtime_doubles_test.go | 45 +- .../endpoints/openai/realtime_gate_test.go | 26 +- .../openai/realtime_interrupt_test.go | 2 +- .../openai/realtime_lfm3_gallery_test.go | 80 + core/http/endpoints/openai/realtime_model.go | 218 ++- core/http/endpoints/openai/realtime_s2s.go | 1570 +++++++++++++++++ .../endpoints/openai/realtime_s2s_test.go | 1359 ++++++++++++++ .../endpoints/openai/realtime_semantic_vad.go | 10 + .../endpoints/openai/realtime_stream_test.go | 30 + core/http/react-ui/e2e/talk.spec.js | 77 + .../react-ui/public/locales/en/tools.json | 4 - .../react-ui/src/components/tools/tools.css | 4 +- core/http/routes/ui.go | 27 +- core/http/routes/ui_pipeline_models_test.go | 85 +- docs/content/advanced/model-configuration.md | 4 +- docs/content/features/audio-to-text.md | 4 + docs/content/features/audio-transform.md | 2 +- docs/content/features/openai-realtime.md | 107 +- docs/content/features/text-to-audio.md | 4 + docs/content/reference/compatibility-table.md | 7 +- gallery/backends/index.yaml | 628 ------- gallery/index.yaml | 539 +++++- gallery/liquid-audio.yaml | 46 - gallery/models/index.yaml | 635 +++++-- pkg/functions/iterative_parser.go | 56 +- pkg/functions/parse.go | 19 +- pkg/functions/peg/chat.go | 5 +- pkg/functions/peg_integration_test.go | 187 ++ swagger/docs.go | 2 +- swagger/swagger.json | 2 +- swagger/swagger.yaml | 7 +- tests/e2e-backends/backend_test.go | 237 +++ tests/e2e/REALTIME_S2S.md | 79 + tests/e2e/e2e_suite_test.go | 68 +- tests/e2e/mock-backend/main.go | 61 +- tests/e2e/mock-backend/s2s.go | 537 ++++++ tests/e2e/mock-backend/s2s_test.go | 353 ++++ tests/e2e/realtime_s2s_test.go | 661 +++++++ tests/e2e/run-realtime-lfm3.sh | 248 +++ 104 files changed, 10688 insertions(+), 3327 deletions(-) create mode 100644 backend/cpp/llama-cpp/realtime_chat_check.cpp create mode 100644 backend/cpp/llama-cpp/realtime_session.h create mode 100644 backend/cpp/llama-cpp/realtime_session_test.cpp create mode 100644 backend/cpp/llama-cpp/transcription_prompt.h create mode 100644 backend/cpp/llama-cpp/transcription_prompt_test.cpp delete mode 100644 backend/python/liquid-audio/Makefile delete mode 100644 backend/python/liquid-audio/backend.py delete mode 100755 backend/python/liquid-audio/install.sh delete mode 100755 backend/python/liquid-audio/protogen.sh delete mode 100644 backend/python/liquid-audio/requirements-cpu.txt delete mode 100644 backend/python/liquid-audio/requirements-cublas12.txt delete mode 100644 backend/python/liquid-audio/requirements-cublas13.txt delete mode 100644 backend/python/liquid-audio/requirements-hipblas.txt delete mode 100644 backend/python/liquid-audio/requirements-l4t13.txt delete mode 100644 backend/python/liquid-audio/requirements-mps.txt delete mode 100644 backend/python/liquid-audio/requirements.txt delete mode 100755 backend/python/liquid-audio/run.sh delete mode 100644 backend/python/liquid-audio/test.py delete mode 100755 backend/python/liquid-audio/test.sh create mode 100644 core/backend/audio_to_audio.go create mode 100644 core/config/gguf_audio_encoder_test.go delete mode 100644 core/gallery/importers/liquid-audio.go delete mode 100644 core/gallery/importers/liquid-audio_test.go create mode 100644 core/gallery/importers/llama-cpp_audio_test.go create mode 100644 core/gallery/lfm3_audio_entries_test.go create mode 100644 core/http/endpoints/openai/realtime_chained_novad_test.go create mode 100644 core/http/endpoints/openai/realtime_lfm3_gallery_test.go create mode 100644 core/http/endpoints/openai/realtime_s2s.go create mode 100644 core/http/endpoints/openai/realtime_s2s_test.go delete mode 100644 gallery/liquid-audio.yaml create mode 100644 tests/e2e/REALTIME_S2S.md create mode 100644 tests/e2e/mock-backend/s2s.go create mode 100644 tests/e2e/mock-backend/s2s_test.go create mode 100644 tests/e2e/realtime_s2s_test.go create mode 100755 tests/e2e/run-realtime-lfm3.sh diff --git a/.agents/llama-cpp-backend.md b/.agents/llama-cpp-backend.md index 67d09ced4..bb7b3d3c7 100644 --- a/.agents/llama-cpp-backend.md +++ b/.agents/llama-cpp-backend.md @@ -81,3 +81,13 @@ The `spec_type` option in `grpc-server.cpp` delegates to upstream's `common_spec - `llama.cpp/common/chat-parser.cpp` - Format presets and model-specific handlers - `llama.cpp/common/chat.h` - Format enums and parameter structures - `llama.cpp/tools/server/server-context.cpp` - Server configuration options + +## Speech-to-speech (AudioToAudioStream) + +`grpc-server.cpp` implements `AudioToAudioStream` on upstream's realtime slot (`server_realtime_slot`, task type `SERVER_TASK_TYPE_REALTIME`, `server-realtime.cpp`). It is compiled only when `prepare.sh` finds the realtime task type (`LOCALAI_HAS_REALTIME`); forks without it answer `UNIMPLEMENTED`. + +- `RealtimeStreamSession` (in `grpc-server.cpp`) feeds the slot with `SERVER_TASK_TYPE_CONTROL` tasks and writes the events. A reader thread handles client messages, the RPC thread consumes slot results. +- `realtime_session.h` holds everything that needs no model: the turn text derived from the chat template, the conversation state machine and the translation of slot results into events. The text of a response goes through the splitter that `grpc-server.cpp` builds from a `common_chat_session` (grammar, preserved tokens and incremental parser of the chat template). `realtime_session_test.cpp` tests it (picked up by `backend/cpp/run-unit-tests.sh`). `realtime_chat_check.cpp` checks it against the LFM3 template of the llama.cpp tree and runs under ctest (`-DLLAMA_GRPC_BUILD_TESTS=ON`). +- No proto change: tools travel in `AudioToAudioConfig.tools` (OpenAI format) and `tool_choice` in its `params`, tool calls leave as `response.function_call_arguments.done`, `response.done` carries `raw_text` for a Go-side parse when the C++ parser found no call, tool results arrive as a `conversation.item.create` control with a `function_call_output` item. The event names are listed in the comments of `backend.proto`. +- Upstream's slot has no voice selection and no VAD. Tool calls follow upstream's `server-realtime.cpp`: the session grammar goes into the sampling of the response (`server_control_response::sampling`), the tool turn is derived from the template. A requested voice is reported in `session.created` with `voice_applied: false`. +- Upstream renames controls and moves the slot often. After a llama.cpp bump, build the backend and run the `realtime_*` tests first. diff --git a/.github/backend-matrix.yml b/.github/backend-matrix.yml index 6fd62baf6..3993a58eb 100644 --- a/.github/backend-matrix.yml +++ b/.github/backend-matrix.yml @@ -341,19 +341,6 @@ include: dockerfile: "./backend/Dockerfile.python" context: "./" ubuntu-version: '2404' - - build-type: 'cublas' - cuda-major-version: "12" - cuda-minor-version: "8" - platforms: 'linux/amd64' - tag-latest: 'auto' - tag-suffix: '-gpu-nvidia-cuda-12-liquid-audio' - runs-on: 'ubuntu-latest' - base-image: "ubuntu:24.04" - skip-drivers: 'false' - backend: "liquid-audio" - dockerfile: "./backend/Dockerfile.python" - context: "./" - ubuntu-version: '2404' - build-type: 'cublas' cuda-major-version: "12" cuda-minor-version: "8" @@ -1082,19 +1069,6 @@ include: dockerfile: "./backend/Dockerfile.python" context: "./" ubuntu-version: '2404' - - build-type: 'cublas' - cuda-major-version: "13" - cuda-minor-version: "0" - platforms: 'linux/amd64' - tag-latest: 'auto' - tag-suffix: '-gpu-nvidia-cuda-13-liquid-audio' - runs-on: 'ubuntu-latest' - base-image: "ubuntu:24.04" - skip-drivers: 'false' - backend: "liquid-audio" - dockerfile: "./backend/Dockerfile.python" - context: "./" - ubuntu-version: '2404' - build-type: 'cublas' cuda-major-version: "13" cuda-minor-version: "0" @@ -1430,19 +1404,6 @@ include: backend: "vibevoice" dockerfile: "./backend/Dockerfile.python" context: "./" - - build-type: 'l4t' - cuda-major-version: "13" - cuda-minor-version: "0" - platforms: 'linux/arm64' - tag-latest: 'auto' - tag-suffix: '-nvidia-l4t-cuda-13-arm64-liquid-audio' - runs-on: 'ubuntu-24.04-arm' - base-image: "ubuntu:24.04" - skip-drivers: 'false' - ubuntu-version: '2404' - backend: "liquid-audio" - dockerfile: "./backend/Dockerfile.python" - context: "./" - build-type: 'l4t' cuda-major-version: "13" cuda-minor-version: "0" @@ -2427,19 +2388,6 @@ include: dockerfile: "./backend/Dockerfile.python" context: "./" ubuntu-version: '2404' - - build-type: 'hipblas' - cuda-major-version: "" - cuda-minor-version: "" - platforms: 'linux/amd64' - tag-latest: 'auto' - tag-suffix: '-gpu-rocm-hipblas-liquid-audio' - runs-on: 'ubuntu-latest' - base-image: "rocm/dev-ubuntu-24.04:7.2.1" - skip-drivers: 'false' - backend: "liquid-audio" - dockerfile: "./backend/Dockerfile.python" - context: "./" - ubuntu-version: '2404' - build-type: 'hipblas' cuda-major-version: "" cuda-minor-version: "" @@ -2942,19 +2890,6 @@ include: dockerfile: "./backend/Dockerfile.python" context: "./" ubuntu-version: '2404' - - build-type: 'intel' - cuda-major-version: "" - cuda-minor-version: "" - platforms: 'linux/amd64' - tag-latest: 'auto' - tag-suffix: '-gpu-intel-liquid-audio' - runs-on: 'ubuntu-latest' - base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04" - skip-drivers: 'false' - backend: "liquid-audio" - dockerfile: "./backend/Dockerfile.python" - context: "./" - ubuntu-version: '2404' - build-type: 'intel' cuda-major-version: "" cuda-minor-version: "" @@ -6348,20 +6283,6 @@ include: dockerfile: "./backend/Dockerfile.python" context: "./" ubuntu-version: '2404' - - build-type: '' - cuda-major-version: "" - cuda-minor-version: "" - platforms: 'linux/amd64' - platform-tag: 'amd64' - tag-latest: 'auto' - tag-suffix: '-cpu-liquid-audio' - runs-on: 'ubuntu-latest' - base-image: "ubuntu:24.04" - skip-drivers: 'false' - backend: "liquid-audio" - dockerfile: "./backend/Dockerfile.python" - context: "./" - ubuntu-version: '2404' - build-type: '' cuda-major-version: "" cuda-minor-version: "" @@ -6876,9 +6797,6 @@ includeDarwin: - backend: "trl" tag-suffix: "-metal-darwin-arm64-trl" build-type: "mps" - - backend: "liquid-audio" - tag-suffix: "-metal-darwin-arm64-liquid-audio" - build-type: "mps" - backend: "piper" tag-suffix: "-metal-darwin-arm64-piper" build-type: "metal" diff --git a/.github/workflows/test-extra.yml b/.github/workflows/test-extra.yml index 4ca43d986..892b34100 100644 --- a/.github/workflows/test-extra.yml +++ b/.github/workflows/test-extra.yml @@ -28,7 +28,6 @@ jobs: qwen-asr: ${{ steps.detect.outputs.qwen-asr }} nemo: ${{ steps.detect.outputs.nemo }} voxcpm: ${{ steps.detect.outputs.voxcpm }} - liquid-audio: ${{ steps.detect.outputs.liquid-audio }} llama-cpp-quantization: ${{ steps.detect.outputs.llama-cpp-quantization }} llama-cpp: ${{ steps.detect.outputs.llama-cpp }} ik-llama-cpp: ${{ steps.detect.outputs.ik-llama-cpp }} @@ -455,32 +454,6 @@ jobs: run: | make --jobs=5 --output-sync=target -C backend/python/voxcpm make --jobs=5 --output-sync=target -C backend/python/voxcpm test - # liquid-audio: LFM2.5-Audio any-to-any backend. The CI smoke test - # exercises Health() and LoadModel(mode:finetune) — fine-tune mode - # short-circuits before pulling weights (backend.py:192), so no - # HuggingFace download or GPU is needed. The full-inference path is - # gated on LIQUID_AUDIO_MODEL_ID, which we don't set here. - tests-liquid-audio: - needs: detect-changes - if: needs.detect-changes.outputs.liquid-audio == 'true' || needs.detect-changes.outputs.run-all == 'true' - runs-on: ubuntu-latest - steps: - - name: Clone - uses: actions/checkout@v7 - with: - submodules: true - - name: Dependencies - run: | - sudo apt-get update - sudo apt-get install -y build-essential ffmpeg - sudo apt-get install -y ca-certificates cmake curl patch python3-pip - # Install UV - curl -LsSf https://astral.sh/uv/install.sh | sh - pip install --user --no-cache-dir grpcio-tools==1.64.1 - - name: Test liquid-audio - run: | - make --jobs=5 --output-sync=target -C backend/python/liquid-audio - make --jobs=5 --output-sync=target -C backend/python/liquid-audio test tests-llama-cpp-quantization: needs: detect-changes if: needs.detect-changes.outputs.llama-cpp-quantization == 'true' || needs.detect-changes.outputs.run-all == 'true' diff --git a/Makefile b/Makefile index f5d56cbd6..cd63de9e8 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ # Disable parallel execution for backend builds -.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/nemo-speech-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/qdrant-store backends/cloud-proxy backends/localai-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/mlx-video backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/timestral-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin +.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/nemo-speech-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/qdrant-store backends/cloud-proxy backends/localai-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/mlx-video backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/supertonic backends/depth-anything-cpp backends/timestral-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin .NOTPARALLEL: backends/whisper-medusa .NOTPARALLEL: backends/funasr @@ -637,7 +637,6 @@ prepare-test-extra: protogen-python $(MAKE) -C backend/python/mlx-video $(MAKE) -C backend/python/sglang $(MAKE) -C backend/python/vibevoice - $(MAKE) -C backend/python/liquid-audio $(MAKE) -C backend/python/moonshine $(MAKE) -C backend/python/pocket-tts $(MAKE) -C backend/python/qwen-tts @@ -675,7 +674,6 @@ test-extra: prepare-test-extra $(MAKE) -C backend/python/longcat-video test $(MAKE) -C backend/python/mlx-video test $(MAKE) -C backend/python/vibevoice test - $(MAKE) -C backend/python/liquid-audio test $(MAKE) -C backend/python/moonshine test $(MAKE) -C backend/python/pocket-tts test $(MAKE) -C backend/python/qwen-tts test @@ -823,6 +821,39 @@ test-extra-backend-llama-cpp-transcription: docker-build-llama-cpp BACKEND_TEST_CTX_SIZE=2048 \ $(MAKE) test-extra-backend +## LFM3-Audio (LiquidAI) through the llama-cpp backend: a named-voice TTS +## (no speaker reference), the LFM3 ASR preset, and the tts/ttsstream/transcription +## s2s specs. Real weights, so it is a manual run and not part of CI. CPU is enough. +test-extra-backend-llama-cpp-lfm3: docker-build-llama-cpp + BACKEND_IMAGE=local-ai-backend:llama-cpp \ + BACKEND_TEST_MODEL_URL=https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/main/LFM3-Audio-2B-Q4_K_M.gguf \ + BACKEND_TEST_MMPROJ_URL=https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/main/mmproj-LFM3-Audio-2B-Q8_0.gguf \ + BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \ + BACKEND_TEST_CAPS=health,load,predict,tts,ttsstream,transcription,s2s \ + BACKEND_TEST_CTX_SIZE=4096 \ + BACKEND_TEST_TTS_TEXT="Hello from LocalAI." \ + BACKEND_TEST_TTS_MIN_SECONDS=0.3 \ + BACKEND_TEST_TTS_MAX_SECONDS=15 \ + BACKEND_TEST_TTS_MIN_RMS=100 \ + BACKEND_TEST_TRANSCRIPT_CONTAINS=country \ + BACKEND_TEST_OPTIONS=parallel:2 \ + $(MAKE) test-extra-backend + +## Real-weights realtime e2e: LFM3-Audio on llama-cpp behind /v1/realtime, as +## an any-to-any model and as a chained pipeline, each with and without a VAD +## (LFM3_MATRIX picks the shapes). The client speaks generated audio (sherpa +## VITS) and a separate sherpa ASR checks the reply audio. MANUAL: not run in CI +## (the CI variant is `make test-realtime`, which uses the mock backend and the +## `!real-models` label filter). A GPU is recommended. Build the llama-cpp image +## for your hardware first. The host needs ffmpeg on PATH (input audio +## conversion). LFM3_VARIANT=q8 selects lfm3-audio-2b-q8. +test-realtime-s2s-real: build-mock-backend docker-build-llama-cpp docker-build-sherpa-onnx protogen-go + bash tests/e2e/run-realtime-lfm3.sh + +## Same, without rebuilding the backend images (they must already exist). +test-realtime-s2s-real-nobuild: build-mock-backend protogen-go + bash tests/e2e/run-realtime-lfm3.sh + ## privacy-filter: the PII/NER token-classification backend. Exercises the ## TokenClassify RPC and asserts byte-correct, UTF-8-aligned span offsets ## against the openai-privacy-filter multilingual GGUF (CPU-runnable, ~50M @@ -1404,7 +1435,6 @@ BACKEND_SGLANG = sglang|python|.|false|true BACKEND_DIFFUSERS = diffusers|python|.|--progress=plain|true BACKEND_CHATTERBOX = chatterbox|python|.|false|true BACKEND_VIBEVOICE = vibevoice|python|.|--progress=plain|true -BACKEND_LIQUID_AUDIO = liquid-audio|python|.|--progress=plain|true BACKEND_MOONSHINE = moonshine|python|.|false|true BACKEND_POCKET_TTS = pocket-tts|python|.|false|true BACKEND_QWEN_TTS = qwen-tts|python|.|false|true @@ -1516,7 +1546,6 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SGLANG))) $(eval $(call generate-docker-build-target,$(BACKEND_DIFFUSERS))) $(eval $(call generate-docker-build-target,$(BACKEND_CHATTERBOX))) $(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE))) -$(eval $(call generate-docker-build-target,$(BACKEND_LIQUID_AUDIO))) $(eval $(call generate-docker-build-target,$(BACKEND_MOONSHINE))) $(eval $(call generate-docker-build-target,$(BACKEND_POCKET_TTS))) $(eval $(call generate-docker-build-target,$(BACKEND_QWEN_TTS))) @@ -1552,7 +1581,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC))) docker-save-%: backend-images docker save local-ai-backend:$* -o backend-images/$*.tar -docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-localai-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-nemo-speech-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-qdrant-store docker-build-audio-cpp docker-build-timestral-cpp +docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-localai-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-nemo-speech-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-qdrant-store docker-build-audio-cpp docker-build-timestral-cpp docker-build-backends: docker-build-whisper-medusa docker-build-backends: docker-build-funasr diff --git a/addons/fine-tuning/ui/src/FineTune.jsx b/addons/fine-tuning/ui/src/FineTune.jsx index 364402a8d..8e9f2ec17 100644 --- a/addons/fine-tuning/ui/src/FineTune.jsx +++ b/addons/fine-tuning/ui/src/FineTune.jsx @@ -591,8 +591,6 @@ export function createFineTune(ctx, api, shared) { const [seed, setSeed] = useState(0) const [mixedPrecision, setMixedPrecision] = useState('') const [extraOptions, setExtraOptions] = useState([]) - const [liquidAudioVoice, setLiquidAudioVoice] = useState('') - const [liquidAudioValDataset, setLiquidAudioValDataset] = useState('') const [hfToken, setHfToken] = useState('') const [showAdvanced, setShowAdvanced] = useState(false) const [resumeFromCheckpoint, setResumeFromCheckpoint] = useState('') @@ -666,11 +664,6 @@ export function createFineTune(ctx, api, shared) { extra.eval_strategy = 'no' } for (const { key, value } of extraOptions) if (key.trim()) extra[key.trim()] = value - // The Python backend reads `voice` and `val_dataset` from extra_options. - if (backend === 'liquid-audio') { - if (liquidAudioVoice) extra.voice = liquidAudioVoice - if (liquidAudioValDataset.trim()) extra.val_dataset = liquidAudioValDataset.trim() - } const req = { model, @@ -740,10 +733,6 @@ export function createFineTune(ctx, api, shared) { const getFormConfig = () => { const extra = {} for (const { key, value } of extraOptions) if (key.trim()) extra[key.trim()] = value - if (backend === 'liquid-audio') { - if (liquidAudioVoice) extra.voice = liquidAudioVoice - if (liquidAudioValDataset.trim()) extra.val_dataset = liquidAudioValDataset.trim() - } return { model, backend, training_method: trainingMethod, @@ -828,10 +817,6 @@ export function createFineTune(ctx, api, shared) { if (eo) restoreEval(eo.eval_strategy, eo.eval_steps, eo.eval_split, eo.eval_dataset_source, eo.eval_split_ratio) if (eo?.save_total_limit != null) setSaveTotalLimit(Number(eo.save_total_limit)) - // liquid-audio extras; they are also kept out of the free-form list below. - if (eo?.voice != null) setLiquidAudioVoice(String(eo.voice)) - if (eo?.val_dataset != null) setLiquidAudioValDataset(String(eo.val_dataset)) - if (config.extra_options && typeof config.extra_options === 'object') { setExtraOptions(Object.entries(config.extra_options) .filter(([k]) => !EXTRA_HANDLED.includes(k)) @@ -1228,24 +1213,6 @@ export function createFineTune(ctx, api, shared) { )} - {backend === 'liquid-audio' && ( -
-

Liquid Audio

-

{t('form.liquidHint')}

-
- - - - - setLiquidAudioValDataset(e.target.value)} placeholder="/data/jenny_tts/val" /> - -
-
- )} -

{t('form.extra')}

{t('form.extraHint')}

diff --git a/addons/fine-tuning/ui/src/i18n/en.js b/addons/fine-tuning/ui/src/i18n/en.js index 34b977f86..9b3229ba9 100644 --- a/addons/fine-tuning/ui/src/i18n/en.js +++ b/addons/fine-tuning/ui/src/i18n/en.js @@ -147,10 +147,6 @@ export default { "form.rewardsHint": "GRPO needs at least one reward function. Pick built-in ones or add your own.", "form.resume": "Resume from checkpoint", "form.clearCheckpoint": "Clear the checkpoint", - "form.liquidHint": "The dataset must be preprocessed by LFM2AudioChatMapper: a directory of arrow files the LFM2 data loader can read.", - "form.liquidVoice": "Speech voice", - "form.liquidInherit": "Inherit from the system prompt", - "form.liquidVal": "Validation dataset", "form.extra": "Extra options", "form.extraHint": "Backend-specific key and value pairs.", "form.summaryFull": "full model", diff --git a/addons/fine-tuning/ui/src/style.css b/addons/fine-tuning/ui/src/style.css index a025d0868..0573c196b 100644 --- a/addons/fine-tuning/ui/src/style.css +++ b/addons/fine-tuning/ui/src/style.css @@ -80,8 +80,8 @@ .bt-kv__row { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1fr) auto; gap: var(--dk-space-2); width: 100%; align-items: center; } .bt-banner { display: flex; align-items: center; gap: var(--dk-space-3); padding: var(--dk-space-3) var(--dk-space-4); border-radius: var(--dk-radius-2); background: var(--dk-accent-wash); color: var(--dk-text); font-size: var(--dk-text-13); flex-wrap: wrap; } .bt-banner code { overflow-wrap: anywhere; } -.bt-rewards, .bt-liquid { display: flex; flex-direction: column; gap: var(--dk-space-3); padding-top: var(--dk-space-3); border-top: 1px solid var(--dk-edge); } -.bt-rewards .dk-hint, .bt-liquid .dk-hint { margin: 0; } +.bt-rewards { display: flex; flex-direction: column; gap: var(--dk-space-3); padding-top: var(--dk-space-3); border-top: 1px solid var(--dk-edge); } +.bt-rewards .dk-hint { margin: 0; } .bt-reward { display: flex; flex-direction: column; gap: var(--dk-space-2); padding: var(--dk-space-3); border: 1px solid var(--dk-edge); border-radius: var(--dk-radius-2); background: var(--dk-card); min-width: 0; } .bt-reward[data-on="true"] { background: var(--dk-accent-wash); border-color: transparent; } .bt-reward__name { display: block; font-size: var(--dk-text-13); font-weight: var(--dk-weight-semibold); } diff --git a/addons/talk/ui/src/components/talk/SessionSheet.jsx b/addons/talk/ui/src/components/talk/SessionSheet.jsx index b714b428e..b4a114119 100644 --- a/addons/talk/ui/src/components/talk/SessionSheet.jsx +++ b/addons/talk/ui/src/components/talk/SessionSheet.jsx @@ -46,7 +46,10 @@ export default function SessionSheet({ } }, []) - const slots = pipeline && !pipeline.self_contained + // A speech-to-speech model answers every role itself. Its only optional + // part is the VAD; the four-slot view would show names that do not run. + const anyToAny = Boolean(pipeline && (pipeline.any_to_any || pipeline.self_contained)) + const slots = pipeline && !anyToAny ? [ ['vad', pipeline.vad], ['transcription', pipeline.transcription], @@ -82,7 +85,13 @@ export default function SessionSheet({

{t('settings.voiceLanguage')}

- onVoice(e.target.value)} placeholder={t('settings.voicePlaceholder')} /> + onVoice(e.target.value)} placeholder={t('settings.voicePlaceholder')} list={anyToAny ? 'talk-voice-options' : undefined} /> + {anyToAny && ( + + + )}
@@ -135,8 +144,17 @@ export default function SessionSheet({ {pipeline && (

{t('settings.pipeline', { name: pipeline.name })}

- {pipeline.self_contained ? ( -

{t('settings.selfContained')}

+ {anyToAny ? ( + <> +

{t('settings.anyToAny')}

+ {pipeline.vad ? ( +
+
{t('settings.slot.vad')}
{pipeline.vad}
+
+ ) : ( +

{t('settings.noVad')}

+ )} + ) : (
{slots.map(([key, value]) => ( @@ -144,9 +162,10 @@ export default function SessionSheet({ ))}
)} + {!anyToAny && !pipeline.vad &&

{t('settings.noVad')}

} {!connected && ( )}
diff --git a/addons/talk/ui/src/i18n/talk.en.json b/addons/talk/ui/src/i18n/talk.en.json index 5e7a1c34f..47a32c2c6 100644 --- a/addons/talk/ui/src/i18n/talk.en.json +++ b/addons/talk/ui/src/i18n/talk.en.json @@ -14,7 +14,8 @@ "idle": { "title": "Ready", "body": "Start a session and talk. The microphone is only used while a session is open.", - "note": "Your microphone streams while the session runs. The server notices when you stop talking and answers on its own." + "note": "Your microphone streams while the session runs. The server notices when you stop talking and answers on its own.", + "noteManual": "Your microphone streams while the session runs. This model has no voice detection, so press Done speaking when you finish a turn." }, "connecting": { "title": "Connecting", @@ -23,7 +24,8 @@ "listening": { "title": "Listening", "body": "Say what you need. I answer when you stop.", - "hearing": "Hearing you speak." + "hearing": "Hearing you speak.", + "manual": "Say what you need, then press Done speaking." }, "interrupted": { "title": "Interrupted", @@ -74,6 +76,7 @@ "start": "Start session", "end": "End session", "testTone": "Test tone", + "endTurn": "Done speaking", "diagnostics": "Diagnostics", "viewTraces": "View traces", "retry": { @@ -111,7 +114,8 @@ "manageDesc": "Let the model query LocalAI: models, backends and system info.", "manageLocked": "Fixed while a session is open. End the session to change it.", "pipeline": "{{name}} in detail", - "selfContained": "Self-contained any-to-any model: it handles voice detection, transcription, the language model and speech by itself.", + "anyToAny": "Speech-to-speech model: one model hears you, answers and speaks. It needs no separate transcription, language or speech model.", + "noVad": "No voice detection model is set. Press Done speaking after each turn.", "editConfig": "Edit model config", "editPipeline": "Edit pipeline", "slot": { diff --git a/addons/talk/ui/src/pages/Talk.jsx b/addons/talk/ui/src/pages/Talk.jsx index 964219f51..86cfbd218 100644 --- a/addons/talk/ui/src/pages/Talk.jsx +++ b/addons/talk/ui/src/pages/Talk.jsx @@ -214,6 +214,21 @@ export default function Talk() { }, [mcpDisconnect]) const selectedModelInfo = pipelineModels.find(m => m.name === selectedModel) + // A model without a VAD model, speech-to-speech or chained, gets no turn + // boundaries from the server, so the user ends each turn with Done speaking. + const manualTurns = Boolean(selectedModelInfo && !selectedModelInfo.vad) + const manualTurnsRef = useRef(false) + manualTurnsRef.current = manualTurns + + // Without a VAD the microphone fills the server's input buffer for as long as + // the session runs, and Done speaking commits all of it. Emptying the buffer + // when a session starts and when a reply ends keeps idle silence and the + // reply's own echo out of the next turn. + const clearInputBuffer = useCallback(() => { + const dc = dcRef.current + if (!manualTurnsRef.current || !dc || dc.readyState !== 'open') return + dc.send(JSON.stringify({ type: 'input_audio_buffer.clear' })) + }, []) // ── Status helper ── const updateStatus = useCallback((state, line) => { @@ -238,7 +253,7 @@ export default function Talk() { } // Pass MCP-server-advertised tools straight through. Server-side they // get rendered into the model's prompt via the function:/argument_regex - // pair on the model config (gallery/lfm.yaml for LFM2.5-Audio). + // pair on the model config. if (tools.length > 0) session.tools = tools dc.send(JSON.stringify({ type: 'session.update', session })) @@ -293,6 +308,7 @@ export default function Talk() { switch (event.type) { case 'session.created': sendSessionUpdate() + clearInputBuffer() updateStatus('listening') break case 'session.updated': @@ -339,6 +355,19 @@ export default function Talk() { setTranscript(prev => upsertAssistant(prev, event.item_id, event.delta, 'append')) } break + case 'response.output_text.delta': + // A text-only reply (no audio) streams as text. + if (event.delta) { + inProgressIdRef.current = event.item_id + setTranscript(prev => upsertAssistant(prev, event.item_id, event.delta, 'append')) + } + break + case 'response.output_text.done': + if (event.text) { + setTranscript(prev => upsertAssistant(prev, event.item_id, event.text, 'replace')) + } + inProgressIdRef.current = null + break case 'response.output_audio_transcript.done': if (event.transcript) { setTranscript(prev => upsertAssistant(prev, event.item_id, event.transcript, 'replace')) @@ -391,6 +420,7 @@ export default function Talk() { }) setInterrupted(true) } + clearInputBuffer() updateStatus('listening') break } @@ -399,7 +429,7 @@ export default function Talk() { updateStatus('error', { key: 'server', params: { message: event.error?.message || t('detail.unknown') } }) break } - }, [sendSessionUpdate, updateStatus, handleFunctionCall, t]) + }, [sendSessionUpdate, clearInputBuffer, updateStatus, handleFunctionCall, t]) // ── Connect ── const connect = useCallback(async () => { @@ -519,6 +549,15 @@ export default function Talk() { } }, []) + // ── End of a turn for a model without VAD ── + // The commit does not answer by itself: the reply is asked for next. + const endTurn = useCallback(() => { + const dc = dcRef.current + if (!dc || dc.readyState !== 'open') return + dc.send(JSON.stringify({ type: 'input_audio_buffer.commit' })) + dc.send(JSON.stringify({ type: 'response.create' })) + }, []) + // ── Test tone ── const sendTestTone = useCallback(() => { const dc = dcRef.current @@ -695,6 +734,7 @@ export default function Talk() { const headline = showInterrupted ? t('view.interrupted.title') : t(`view.${view}.title`) let sentence = showInterrupted ? t('view.interrupted.body') : t(`view.${view}.body`) if (view === 'listening' && hearing && !showInterrupted) sentence = t('view.listening.hearing') + else if (view === 'listening' && manualTurns && !showInterrupted) sentence = t('view.listening.manual') if (detail) sentence = t(`detail.${detail.key}`, detail.params) const failed = view === 'blocked' || view === 'lost' || view === 'error' @@ -839,6 +879,11 @@ export default function Talk() { + {manualTurns && !busy && ( + + )} {view !== 'connecting' && (
- {!isConnected && view === 'idle' &&

{t('view.idle.note')}

} + {!isConnected && view === 'idle' &&

{t(manualTurns ? 'view.idle.noteManual' : 'view.idle.note')}

} {isConnected && diagVisible && (
diff --git a/backend/backend.proto b/backend/backend.proto index 79be236b2..a84147357 100644 --- a/backend/backend.proto +++ b/backend/backend.proto @@ -1439,16 +1439,16 @@ message AudioToAudioRequest { message AudioToAudioConfig { // PCM format for client→server audio. 0 => backend default - // (16 kHz for the LFM2-Audio Conformer encoder). + // (16 kHz, the rate the speech encoder expects). int32 input_sample_rate = 1; // Preferred server→client audio rate. 0 => backend default - // (24 kHz for the LFM2-Audio vocoder). + // (24 kHz, the rate the speech decoder produces). int32 output_sample_rate = 2; // Optional system prompt override. Empty => backend chooses based on // mode (e.g. "Respond with interleaved text and audio."). string system_prompt = 3; // Optional baked-voice id. Models that only ship a fixed set of - // voices (e.g. LFM2-Audio: us_male/us_female/uk_male/uk_female) match + // voices (e.g. us_male/us_female/uk_male/uk_female) match // this against their voice table; an empty string keeps the default. string voice = 4; // JSON-encoded array of tool definitions in OpenAI Chat Completions @@ -1479,6 +1479,11 @@ message AudioToAudioControl { // function_call_output as JSON in // `payload`) // "session.update" — re-configure mid-stream + // "response.create" — start a response; payload + // {"output_modalities":[...], + // "max_output_tokens":N} + // A function_call_output item is {"type":"function_call_output", + // "call_id":...,"output":...} in the payload of "conversation.item.create". string event = 1; // Event-specific JSON payload. bytes payload = 2; @@ -1487,12 +1492,18 @@ message AudioToAudioControl { message AudioToAudioResponse { // Event identifies what this frame carries. Mirrors the OpenAI Realtime // API server-event names where applicable. Initial set: + // "session.created" — meta: input_sample_rate, + // output_sample_rate, voices + // "session.updated" — meta: same as session.created + // "input_audio_buffer.committed" // "response.audio.delta" - // "response.audio_transcript.delta" + // "response.audio_transcript.delta" — meta: {"delta": text} + // "response.text.delta" — text-only responses // "response.function_call_arguments.delta" - // "response.function_call_arguments.done" - // "response.done" - // "error" + // "response.function_call_arguments.done" — meta: call_id, name, + // arguments (a JSON string) + // "response.done" — meta: status, usage, text + // "error" — meta: type, message string event = 1; // Populated when event = response.audio.delta. bytes pcm = 2; diff --git a/backend/cpp/llama-cpp/CMakeLists.txt b/backend/cpp/llama-cpp/CMakeLists.txt index 773a13b5e..a25f33877 100644 --- a/backend/cpp/llama-cpp/CMakeLists.txt +++ b/backend/cpp/llama-cpp/CMakeLists.txt @@ -128,6 +128,11 @@ if(LLAMA_GRPC_BUILD_TESTS) target_compile_features(tts_request_options_test PRIVATE cxx_std_17) add_test(NAME tts_request_options_test COMMAND tts_request_options_test) + add_executable(transcription_prompt_test transcription_prompt_test.cpp transcription_prompt.h) + target_include_directories(transcription_prompt_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) + target_compile_features(transcription_prompt_test PRIVATE cxx_std_17) + add_test(NAME transcription_prompt_test COMMAND transcription_prompt_test) + add_executable(thread_params_test thread_params_test.cpp thread_params.h) target_include_directories(thread_params_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) target_compile_features(thread_params_test PRIVATE cxx_std_17) @@ -159,4 +164,20 @@ if(LLAMA_GRPC_BUILD_TESTS) target_include_directories(stream_peer_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) target_compile_features(stream_peer_test PRIVATE cxx_std_17) add_test(NAME stream_peer_test COMMAND stream_peer_test) + + # Conversation logic of AudioToAudioStream (nlohmann/json only). + add_executable(realtime_session_test realtime_session_test.cpp realtime_session.h) + target_include_directories(realtime_session_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) + target_link_libraries(realtime_session_test PRIVATE ${_LLAMA_COMMON_TARGET}) # vendored nlohmann/json + target_compile_features(realtime_session_test PRIVATE cxx_std_17) + add_test(NAME realtime_session_test COMMAND realtime_session_test) + + # The same logic against the real LFM3 chat template from the llama.cpp tree. + if(EXISTS "${CMAKE_SOURCE_DIR}/models/templates/LFM3.jinja") + add_executable(realtime_chat_check realtime_chat_check.cpp realtime_session.h) + target_include_directories(realtime_chat_check PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) + target_link_libraries(realtime_chat_check PRIVATE ${_LLAMA_COMMON_TARGET}) + target_compile_features(realtime_chat_check PRIVATE cxx_std_17) + add_test(NAME realtime_chat_check COMMAND realtime_chat_check "${CMAKE_SOURCE_DIR}/models/templates/LFM3.jinja") + endif() endif() diff --git a/backend/cpp/llama-cpp/Makefile b/backend/cpp/llama-cpp/Makefile index 7679d1e07..189a37616 100644 --- a/backend/cpp/llama-cpp/Makefile +++ b/backend/cpp/llama-cpp/Makefile @@ -1,5 +1,5 @@ -LLAMA_VERSION?=338fce6511c744893cc0d3d958b275f40552c7a4 +LLAMA_VERSION?=0ac1a7fd35c127a18c0cbc6c57d12e256a6a385b LLAMA_REPO?=https://github.com/ggerganov/llama.cpp CMAKE_ARGS?= diff --git a/backend/cpp/llama-cpp/grpc-server.cpp b/backend/cpp/llama-cpp/grpc-server.cpp index 55f777edb..a84f35dd9 100644 --- a/backend/cpp/llama-cpp/grpc-server.cpp +++ b/backend/cpp/llama-cpp/grpc-server.cpp @@ -49,6 +49,15 @@ #include "decision_compat.h" #include "decision_images.h" #endif +// server-realtime.cpp defines server_realtime_slot, which server-context.cpp +// reaches through server-realtime.h (SERVER_TASK_TYPE_REALTIME). Upstream +// llama.cpp compiles it into the server library; this TU must carry it too or +// the link fails. Forks that predate the realtime split do not ship the file, +// so the include is guarded. +#if __has_include("server-realtime.cpp") +#define LOCALAI_HAS_SERVER_REALTIME 1 +#include "server-realtime.cpp" +#endif #include "server-context.cpp" // LocalAI @@ -67,7 +76,9 @@ #include "moe_cache_option.h" #include "passthrough_options.h" #include "stream_peer.h" +#include "realtime_session.h" #include "tts_request_options.h" +#include "transcription_prompt.h" #include #include #include @@ -1579,6 +1590,739 @@ static std::string tts_pcm_f32_to_s16(const std::string & samples) { } #endif +#if LOCALAI_HAS_REALTIME && defined(LOCALAI_HAS_SERVER_REALTIME) && !defined(LOCALAI_LLAMA_CPP_NO_TTS_TASK) +#define LOCALAI_HAS_AUDIO_TO_AUDIO 1 + +namespace rts = llama_grpc::realtime; + +// One AudioToAudioStream call: a speech-to-speech conversation on a realtime +// slot (server_realtime_slot). The slot holds the audio and text of the whole +// conversation. This class feeds it through SERVER_TASK_TYPE_CONTROL tasks and +// turns what it produces into AudioToAudioResponse events. The conversation +// logic that needs no model lives in realtime_session.h. +// +// Two threads, as upstream: a reader thread handles the client messages, the RPC +// thread consumes the results of the slot and writes the events. +class RealtimeStreamSession { + static std::string str_field(const rts::json & j, const char * key) { + return j.is_object() && j.contains(key) && j[key].is_string() ? j[key].get() : std::string(); + } + + using Stream = grpc::ServerReaderWriter; + +public: + RealtimeStreamSession(server_context & ctx_server, const common_params & params, grpc::ServerContext * gctx, Stream * stream) + : ctx_server(ctx_server), params(params), gctx(gctx), stream(stream) {} + + grpc::Status run() { + const auto & cap = ctx_server.impl->mtmd_cap; + // the same test as the websocket endpoint of upstream + if (!(cap.audio_stream && cap.chat)) { + const std::string msg = "the loaded model does not support speech-to-speech (needs an audio-stream projector and a chat template)"; + send_error("not_supported", msg); + return grpc::Status(grpc::StatusCode::UNIMPLEMENTED, msg); + } + + input_rate = mtmd_get_audio_sample_rate(ctx_server.impl->mctx); + if (cap.audio_gen) { + output_rate = mtmd_gen_audio_get_info(ctx_server.impl->mctx).sample_rate; + } + + // the first message is the config + backend::AudioToAudioRequest first; + if (!stream->Read(&first)) { + return grpc::Status::OK; + } + if (first.payload_case() != backend::AudioToAudioRequest::kConfig) { + send_error("invalid_request_error", "the first message must be a config"); + return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "the first message must be a config"); + } + std::string err; + if (!apply_config(first.config(), err)) { + send_error("invalid_request_error", err); + return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, err); + } + + // the slot answers once it holds the session + auto rd = ctx_server.get_response_reader(); + { + server_task task(SERVER_TASK_TYPE_REALTIME); + task.id = rd.get_new_id(); + task.params.sampling = sampling(); + task.params.stream = true; + task.params.oaicompat_cmpl_id = id; + task.params.oaicompat_model = ctx_server.get_meta().model_name; + rd.post_task(std::move(task)); + } + { + auto result = rd.next([this]() { return gctx->IsCancelled(); }); + if (!result) { + return grpc::Status(grpc::StatusCode::CANCELLED, "client closed the stream"); + } + if (result->is_error()) { + const std::string msg = json_value(result->to_json(), "message", std::string("server error")); + send_error("server_error", msg, true); + return grpc::Status(grpc::StatusCode::INTERNAL, msg); + } + } + send_event("session.created", rts::session_meta(input_rate, output_rate, rts::default_voices(), voice)); + configured = true; + + std::thread reader([this]() { + backend::AudioToAudioRequest req; + while (!fatal.load() && stream->Read(&req)) { + handle(req); + } + client_done = true; + }); + + // a response that is still running is allowed to finish after the client half-closed + auto should_stop = [this]() { + return fatal.load() || gctx->IsCancelled() || (client_done.load() && !response_active.load()); + }; + while (true) { + auto result = rd.next(should_stop); + if (!result) { + break; + } + if (result->is_error()) { + const json j = result->to_json(); + send_error("server_error", json_value(j, "message", std::string("server error")), true); + fatal_message = json_value(j, "message", std::string("server error")); + fatal = true; + break; + } + if (auto * res = dynamic_cast(result.get())) { + on_slot_result(*res); + } + } + + // a reader blocked in Read() only returns once the call is cancelled + if (!client_done.load()) { + gctx->TryCancel(); + } + reader.join(); + + if (!fatal_message.empty()) { + return grpc::Status(grpc::StatusCode::INTERNAL, fatal_message); + } + if (peer.gone()) { + return grpc::Status(grpc::StatusCode::CANCELLED, "client closed the stream"); + } + return grpc::Status::OK; + } + +private: + server_context & ctx_server; + const common_params & params; + grpc::ServerContext * gctx; + Stream * stream; + + const std::string id = "sess_" + random_string(); + int input_rate = 16000; + int output_rate = 24000; + + // session config, owned by the reader thread once it runs + bool configured = false; + std::string instructions; + bool instructions_default = true; // the client chose none: they follow the tools + std::string voice; + std::string tools_text; + std::vector tools; + rts::SessionParams session_params; + rts::TurnText turn; + rts::ConversationState state; + + // output + std::mutex out_mutex; + llama_grpc::StreamPeer peer; + int64_t seq = 0; + std::atomic fatal{false}; + std::string fatal_message; + std::atomic client_done{false}; + + // the response being produced, shared by the reader and the result loop + std::mutex resp_mutex; + std::unique_ptr translator; + std::atomic response_active{false}; + + // + // events + // + + void write(backend::AudioToAudioResponse & r) { + std::lock_guard lock(out_mutex); + if (peer.gone()) { + return; + } + r.set_sequence(seq++); + peer.observe_write(stream->Write(r)); + if (peer.gone()) { + fatal = true; + } + } + + void send_event(const std::string & event, const rts::json & meta) { + backend::AudioToAudioResponse r; + r.set_event(event); + r.set_meta(rts::dump_json(meta)); + write(r); + } + + void send_out(const rts::OutEvent & ev) { + backend::AudioToAudioResponse r; + r.set_event(ev.event); + if (!ev.pcm.empty()) { + r.set_pcm(ev.pcm); + r.set_sample_rate(ev.sample_rate); + } else { + r.set_meta(rts::dump_json(ev.meta)); + } + write(r); + } + + // response_scoped: the error answers a response.create (or ends the stream) + // and no response.done follows. See rts::error_meta. + void send_error(const std::string & type, const std::string & message, bool response_scoped = false) { + send_event("error", rts::error_meta(type, message, response_scoped)); + } + + // + // slot controls, run on the slot in the order they are posted + // + + bool control(server_control_action action, std::string text = {}, std::vector audio = {}, server_control_response response = {}) { + auto ctl = ctx_server.get_response_reader(); + server_task task(SERVER_TASK_TYPE_CONTROL); + task.id = ctl.get_new_id(); + task.params.control_action = action; + task.params.control_cmpl_id = id; + task.params.control_text = std::move(text); + task.params.control_audio = std::move(audio); + task.params.control_response = response; + ctl.post_task(std::move(task)); + auto result = ctl.next([this]() { return fatal.load() || gctx->IsCancelled(); }); + auto * res = result ? dynamic_cast(result.get()) : nullptr; + return res && res->success; + } + + // + // chat template + // + + common_chat_templates_inputs base_inputs() const { + const auto & chat = ctx_server.impl->chat_params; + common_chat_templates_inputs in; + in.use_jinja = chat.use_jinja; + in.enable_thinking = false; + in.chat_template_kwargs = chat.chat_template_kwargs; + in.tools = tools; + return in; + } + + static std::vector to_common(const std::vector & msgs) { + std::vector out; + for (const auto & m : msgs) { + common_chat_msg c; + c.role = m.role; + c.content = m.content; + c.tool_call_id = m.tool_call_id; + for (const auto & tc : m.tool_calls) { + c.tool_calls.push_back({tc.name, tc.arguments, tc.id}); + } + out.push_back(std::move(c)); + } + return out; + } + + rts::RenderFn renderer() const { + return [this](const std::vector & msgs, bool add_generation_prompt) { + auto in = base_inputs(); + in.messages = to_common(msgs); + in.add_generation_prompt = add_generation_prompt; + return common_chat_templates_apply(ctx_server.impl->chat_params.tmpls.get(), in).prompt.str(); + }; + } + + static common_chat_tool_choice to_common_choice(rts::ToolChoice c) { + switch (c) { + case rts::ToolChoice::None: return COMMON_CHAT_TOOL_CHOICE_NONE; + case rts::ToolChoice::Required: return COMMON_CHAT_TOOL_CHOICE_REQUIRED; + default: return COMMON_CHAT_TOOL_CHOICE_AUTO; + } + } + + // The chat session of one response, as upstream builds it for /v1/realtime: + // the template selects the grammar and the parser for the tools of the session. + common_chat_session make_chat_session(rts::ToolChoice choice) const { + auto in = base_inputs(); + in.add_generation_prompt = true; + in.parallel_tool_calls = true; + in.tool_choice = to_common_choice(choice); + // as the server does: the empty think block of the generation prompt is reasoning, not content + in.reasoning_format = ctx_server.impl->chat_params.reasoning_format; + common_chat_msg user; + user.role = "user"; + user.content = "x"; + in.messages.push_back(user); + return common_chat_session(ctx_server.impl->chat_params.tmpls.get(), ctx_server.impl->vocab, in); + } + + // Splits the text of a response into content and tool calls while it grows. + // If the parser fails, the rest of the response is content and the full text + // still goes out in response.done, so the caller can parse it another way. + static rts::ResponseTranslator::Splitter make_splitter(common_chat_session session) { + auto state = std::make_shared(std::move(session)); + auto broken = std::make_shared(false); + return [state, broken](const std::string & text, bool final) { + rts::SplitResult out; + if (*broken) { + out.content = text; + return out; + } + try { + std::vector diffs; + state->update_chat_msg(common_chat_input(text), !final, diffs, true); + for (const auto & d : diffs) { + out.content += d.content_delta; + if (d.tool_call_index != std::string::npos) { + out.calls.push_back({d.tool_call_index, d.tool_call_delta.id, d.tool_call_delta.name, d.tool_call_delta.arguments}); + } + } + } catch (const std::exception & e) { + LOG_WRN("realtime: could not parse the tool calls of a response: %s\n", e.what()); + *broken = true; + out.content = text; + } + return out; + }; + } + + // + // session config + // + + common_params_sampling sampling() const { + auto s = params.sampling; + if (session_params.temperature) { + s.temp = *session_params.temperature; + } + if (session_params.top_p) { + s.top_p = *session_params.top_p; + } + if (session_params.top_k) { + s.top_k = *session_params.top_k; + } + if (session_params.seed) { + s.seed = *session_params.seed; + } + return s; + } + + // any tool of the session, to derive the tool turn from the chat template + std::string tool_name() const { + return tools.empty() ? std::string() : tools.front().name; + } + + rts::OutMode session_out_mode() const { + return session_params.out_mode ? *session_params.out_mode : rts::default_out_mode(!tools.empty()); + } + + // tools: a JSON array in OpenAI format. Empty means none. + static bool parse_tools(const std::string & text, std::vector & out, std::string & err) { + out.clear(); + if (text.empty()) { + return true; + } + try { + const json parsed = json::parse(text); + if (!parsed.is_array()) { + err = "tools must be a JSON array"; + return false; + } + out = common_chat_tools_parse_oaicompat(parsed); + } catch (const std::exception & e) { + err = std::string("invalid tools: ") + e.what(); + return false; + } + return true; + } + + // Applies a config before the first turn: derives the text around the turns. + // Nothing changes if it fails. + bool apply_config(const backend::AudioToAudioConfig & c, std::string & err) { + if (c.input_sample_rate() != 0 && c.input_sample_rate() != input_rate) { + err = "the input audio must be PCM16 at " + std::to_string(input_rate) + " Hz"; + return false; + } + rts::SessionParams p; + if (!rts::parse_session_params(std::map(c.params().begin(), c.params().end()), p, err)) { + return false; + } + std::vector new_tools; + if (!parse_tools(c.tools(), new_tools, err)) { + return false; + } + const std::string new_instructions = rts::resolve_instructions(c.system_prompt(), !new_tools.empty()); + + // render with the new tools, keep the old state until it works + const auto old_tools = tools; + tools = new_tools; + rts::TurnText t; + if (!rts::derive_turn_text(renderer(), new_instructions, tool_name(), t, err)) { + tools = old_tools; + return false; + } + instructions = new_instructions; + instructions_default = c.system_prompt().empty() || rts::is_default_instructions(c.system_prompt()); + tools_text = c.tools(); + voice = c.voice(); + session_params = p; + turn = t; + return true; + } + + // session.update, or a later config: before the first turn everything changes, afterwards only the response options + void update_session(const std::string & new_instructions, const std::string & new_tools, const std::string & new_voice, + const rts::json & modalities, const rts::json & max_tokens, const rts::json & choice) { + const bool tools_changed = !new_tools.empty() && new_tools != tools_text; + const bool voice_changed = !new_voice.empty() && new_voice != voice; + // The default instructions follow the tools, so a client that sends none (or the + // default back) gets the text that fits the tools it ends up with. A tools update + // that is not valid keeps the old tools, and the error is sent below. + std::vector parsed_tools; + std::string err; + const bool tools_ok = !tools_changed || parse_tools(new_tools, parsed_tools, err); + const bool has_tools_after = tools_changed && tools_ok ? !parsed_tools.empty() : !tools.empty(); + std::string target_instructions = instructions; + if (!new_instructions.empty()) { + target_instructions = rts::resolve_instructions(new_instructions, has_tools_after); + } else if (instructions_default) { + target_instructions = rts::default_instructions(has_tools_after); + } + const bool instr_changed = target_instructions != instructions; + const std::string denied = state.session_update_error(instr_changed && !new_instructions.empty(), tools_changed, voice_changed); + if (!denied.empty()) { + send_error("invalid_request_error", "session.update: " + denied); + return; + } + rts::SessionParams p = session_params; + if (!modalities.is_null()) { + rts::OutMode m; + if (!rts::parse_output_modalities(modalities, m, err)) { + send_error("invalid_request_error", "session.update: " + err); + return; + } + p.out_mode = m; + } + if (!max_tokens.is_null() && !rts::parse_max_output_tokens(max_tokens, p.max_output_tokens)) { + send_error("invalid_request_error", "session.update: max_output_tokens must be an integer or \"inf\""); + return; + } + if (!choice.is_null() && !rts::parse_tool_choice(choice, p.tool_choice, err)) { + send_error("invalid_request_error", "session.update: " + err); + return; + } + if (!state.started() && (instr_changed || tools_changed)) { + if (!tools_ok) { + send_error("invalid_request_error", "session.update: " + err); + return; + } + const auto old_tools = tools; + if (tools_changed) { + tools = parsed_tools; + } + rts::TurnText tt; + if (!rts::derive_turn_text(renderer(), target_instructions, tool_name(), tt, err)) { + tools = old_tools; + send_error("server_error", err); + return; + } + turn = tt; + instructions = target_instructions; + if (!new_instructions.empty()) { + instructions_default = rts::is_default_instructions(new_instructions); + } + if (tools_changed) { + tools_text = new_tools; + } + } + if (voice_changed) { + voice = new_voice; + } + session_params = p; + send_event("session.updated", rts::session_meta(input_rate, output_rate, rts::default_voices(), voice)); + } + + // + // client messages + // + + void handle(const backend::AudioToAudioRequest & req) { + switch (req.payload_case()) { + case backend::AudioToAudioRequest::kConfig: on_config(req.config()); break; + case backend::AudioToAudioRequest::kFrame: on_frame(req.frame()); break; + case backend::AudioToAudioRequest::kControl: on_control(req.control()); break; + default: send_error("invalid_request_error", "empty message"); break; + } + } + + void on_config(const backend::AudioToAudioConfig & c) { + if (c.reset()) { + if (response_active.load()) { + send_error("invalid_request_error", "reset: a response is running, cancel it first"); + return; + } + if (!control(SERVER_CONTROL_ACTION_CLEAR)) { + send_error("server_error", "reset: the slot did not accept the control"); + return; + } + state.reset(); + std::string err; + if (!apply_config(c, err)) { + send_error("invalid_request_error", err); + return; + } + send_event("session.updated", rts::session_meta(input_rate, output_rate, rts::default_voices(), voice)); + return; + } + // the options of a later config are applied like a session.update + const auto & p = c.params(); + rts::json modalities = nullptr; + rts::json max_tokens = nullptr; + auto it = p.find("output_modalities"); + if (it != p.end()) { + rts::OutMode m; + std::string err; + if (!rts::parse_modalities_string(it->second, m, err)) { + send_error("invalid_request_error", "config: " + err); + return; + } + modalities = rts::out_mode_json(m); + } + rts::json choice = nullptr; + auto ct = p.find("tool_choice"); + if (ct != p.end()) { + choice = ct->second; + } + update_session(c.system_prompt(), c.tools(), c.voice(), modalities, max_tokens, choice); + } + + // the header of a user turn goes in before its first audio or text + bool ensure_user_turn() { + if (state.user_turn_is_open()) { + return true; + } + if (!control(SERVER_CONTROL_ACTION_TEXT_APPEND, state.user_turn_header(turn))) { + send_error("server_error", "the slot did not accept the turn"); + return false; + } + state.user_turn_opened(); + return true; + } + + // response_scoped: the commit is part of a response.create + bool commit(bool response_scoped = false) { + if (!state.has_audio()) { + send_error("input_audio_buffer_commit_empty", "input_audio_buffer.commit: the buffer is empty", response_scoped); + return false; + } + if (!control(SERVER_CONTROL_ACTION_AUDIO_END)) { + send_error("server_error", "the slot did not accept the end of the audio", response_scoped); + return false; + } + state.audio_committed(); + send_event("input_audio_buffer.committed", rts::json::object()); + return true; + } + + void on_frame(const backend::AudioToAudioFrame & f) { + if (!f.pcm().empty()) { + auto samples = rts::pcm_s16le_to_f32(f.pcm()); + if (samples.empty() || !ensure_user_turn()) { + return; + } + if (!control(SERVER_CONTROL_ACTION_AUDIO_APPEND, {}, std::move(samples))) { + send_error("server_error", "the slot did not accept the audio"); + return; + } + state.audio_appended(); + } + if (f.end_of_input()) { + commit(); + } + } + + void on_control(const backend::AudioToAudioControl & c) { + rts::json payload = rts::json::object(); + if (!c.payload().empty()) { + payload = rts::json::parse(c.payload(), nullptr, false); + if (payload.is_discarded()) { + send_error("invalid_request_error", c.event() + ": the payload is not valid JSON", c.event() == "response.create"); + return; + } + } + const std::string & ev = c.event(); + if (ev == "input_audio_buffer.commit") { + commit(); + } else if (ev == "input_audio_buffer.clear") { + // the audio already went into the conversation + send_event("input_audio_buffer.cleared", rts::json::object()); + } else if (ev == "response.create") { + create_response(payload); + } else if (ev == "response.cancel") { + if (!response_active.load()) { + send_error("response_cancel_not_active", "response.cancel: there is no response to cancel"); + } else if (!control(SERVER_CONTROL_ACTION_RESPONSE_CANCEL)) { + send_error("server_error", "the slot did not accept the cancel"); + } + } else if (ev == "conversation.item.create") { + create_item(payload); + } else if (ev == "session.update") { + const rts::json s = payload.contains("session") && payload["session"].is_object() ? payload["session"] : payload; + update_session(str_field(s, "instructions"), + s.contains("tools") ? (s["tools"].is_string() ? s["tools"].get() : s["tools"].dump()) : std::string(), + str_field(s, "voice"), + s.contains("output_modalities") ? s["output_modalities"] : rts::json(nullptr), + s.contains("max_output_tokens") ? s["max_output_tokens"] : rts::json(nullptr), + s.contains("tool_choice") ? s["tool_choice"] : rts::json(nullptr)); + } else { + send_error("invalid_request_error", "unknown event type: " + ev); + } + } + + void create_item(const rts::json & payload) { + const rts::json item = payload.contains("item") && payload["item"].is_object() ? payload["item"] : payload; + const std::string type = str_field(item, "type"); + if (type == "function_call_output") { + const rts::json out = item.value("output", rts::json("")); + std::string err; + const std::string text = state.tool_turn_text(turn, out.is_string() ? out.get() : out.dump(), err); + if (text.empty()) { + send_error("invalid_request_error", err.empty() ? "function_call_output: empty tool turn" : err); + return; + } + if (!control(SERVER_CONTROL_ACTION_TEXT_APPEND, text)) { + send_error("server_error", "the slot did not accept the tool result"); + return; + } + state.tool_turn_opened(); + return; + } + if (type != "message" || str_field(item, "role") != "user") { + send_error("invalid_request_error", "conversation.item.create: only user messages and function_call_output are supported"); + return; + } + std::string text; + for (const auto & part : item.value("content", rts::json::array())) { + const std::string t = str_field(part, "type"); + if (t != "input_text") { + send_error("invalid_request_error", "conversation.item.create: unsupported content type: " + t); + return; + } + text += str_field(part, "text"); + } + if (!ensure_user_turn()) { + return; + } + if (!control(SERVER_CONTROL_ACTION_TEXT_APPEND, text)) { + send_error("server_error", "the slot did not accept the message"); + } + } + + static mtmd_helper_gen_audio_out_mode to_mtmd(rts::OutMode m) { + switch (m) { + case rts::OutMode::Text: return MTMD_HELPER_GEN_AUDIO_OUT_TEXT; + case rts::OutMode::Audio: return MTMD_HELPER_GEN_AUDIO_OUT_AUDIO; + case rts::OutMode::Mixed: return MTMD_HELPER_GEN_AUDIO_OUT_MIXED; + default: return MTMD_HELPER_GEN_AUDIO_OUT_AUTO; + } + } + + void create_response(const rts::json & payload) { + if (response_active.load()) { + send_error("conversation_already_has_active_response", "response.create: a response is already running", true); + return; + } + const std::string not_ready = state.response_create_error(); + if (!not_ready.empty()) { + send_error("conversation_empty", not_ready, true); + return; + } + rts::ResponseOptions opts{}; + std::string err; + if (!rts::parse_response_options(payload, session_out_mode(), session_params.max_output_tokens, session_params.tool_choice, opts, err)) { + send_error("invalid_request_error", "response.create: " + err, true); + return; + } + const auto & cap = ctx_server.impl->mtmd_cap; + if (!cap.audio_gen && opts.out_mode != rts::OutMode::Text && opts.out_mode != rts::OutMode::Auto) { + send_error("invalid_request_error", "response.create: the model cannot generate audio", true); + return; + } + if (!cap.text_gen && opts.out_mode != rts::OutMode::Audio && opts.out_mode != rts::OutMode::Auto) { + send_error("invalid_request_error", "response.create: the model cannot generate text", true); + return; + } + // audio not committed yet belongs to this turn + if (state.has_audio() && !commit(true)) { + return; + } + + server_control_response response; + response.n_predict = opts.max_output_tokens; + response.out_mode = to_mtmd(opts.out_mode); + response.sampling = sampling(); + + // with tools, the grammar and the parser of the chat template take part in the response + rts::ResponseTranslator::Splitter splitter; + if (!tools.empty()) { + try { + common_chat_session chat = make_chat_session(opts.tool_choice); + chat.apply_sampling(response.sampling); + splitter = make_splitter(std::move(chat)); + } catch (const std::exception & e) { + send_error("server_error", std::string("response.create: ") + e.what(), true); + return; + } + } + { + std::lock_guard lock(resp_mutex); + translator = std::make_unique(opts.out_mode, output_rate, std::move(splitter)); + } + response_active = true; + if (!control(SERVER_CONTROL_ACTION_RESPONSE_CREATE, state.response_prompt(turn), {}, response)) { + response_active = false; + send_error("server_error", "the slot did not accept the response", true); + return; + } + state.response_created(turn); + } + + // + // results of the slot + // + + void on_slot_result(const server_task_result_realtime & res) { + std::vector events; + { + std::lock_guard lock(resp_mutex); + if (!translator) { + return; + } + events = translator->on_result(res.text, res.audio, res.response_done, res.cancelled, res.n_decoded); + if (translator->done()) { + translator.reset(); + response_active = false; + } + } + for (const auto & ev : events) { + send_out(ev); + } + } +}; +#endif + // GRPC Server start class BackendServiceImpl final : public backend::Backend::Service { private: @@ -3193,22 +3937,26 @@ public: "the loaded model does not support audio generation (no gen-audio mmproj)"); } + // Qwen3-TTS Base takes a speaker reference file; LFM3-Audio takes a + // named voice through its "tts" chat template. Forks without the voice + // probe predate named voices, so they keep the reference rule. +#if LOCALAI_HAS_TTS_VOICE + const bool needs_reference = ctx_server.impl->mtmd_cap.audio_gen_spk_ref; +#else + const bool needs_reference = true; +#endif + std::map params(request->params().begin(), request->params().end()); const auto opts = llama_grpc::parse_tts_request_options( request->text(), request->voice(), request->has_language() ? request->language() : std::string(), - params); + params, + needs_reference); if (!opts.ok) { return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, opts.error); } - auto wrapper = mtmd_helper_bitmap_init_from_file(ctx_server.impl->mctx, opts.voice_path.c_str(), false LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)); - if (!wrapper.bitmap) { - return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, - "failed to read speaker reference audio: " + opts.voice_path); - } - task.tts_inp.set_prompt(opts.text); // core/backend/tts.go always sets TTSRequest.language, so has_language() // is true even when the caller named no language and the string is empty. @@ -3219,7 +3967,28 @@ public: if (!opts.language.empty()) { task.tts_inp.set_lang(opts.language); } - task.tts_inp.set_speaker_ref(mtmd::bitmap_ptr(wrapper.bitmap)); + + if (needs_reference) { + auto wrapper = mtmd_helper_bitmap_init_from_file(ctx_server.impl->mctx, opts.voice_path.c_str(), false LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)); + if (!wrapper.bitmap) { + return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, + "failed to read speaker reference audio: " + opts.voice_path); + } + task.tts_inp.set_speaker_ref(mtmd::bitmap_ptr(wrapper.bitmap)); + } +#if LOCALAI_HAS_TTS_VOICE + else { + // Same as upstream POST /tts: the model's "tts" template renders the + // prompt with the language and voice, and an empty voice takes the + // template default. Fall back to the raw text when there is no template. + const std::string tts_prompt = common_chat_get_tts_prompt( + ctx_server.impl->chat_params.tmpls.get(), opts.text, opts.language, opts.voice); + if (!tts_prompt.empty()) { + task.tts_inp.set_prompt(tts_prompt); + } + task.tts_inp.set_voice(opts.voice); + } +#endif task.tts_inp.data.top_k = opts.top_k; task.tts_inp.data.top_p = opts.top_p; task.tts_inp.data.stream = stream; @@ -3406,6 +4175,23 @@ public: } #endif +#ifdef LOCALAI_HAS_AUDIO_TO_AUDIO + // Speech-to-speech on llama.cpp's realtime slot. The wire contract (events, + // controls, tool calls) is described in RealtimeStreamSession and in the + // comments of AudioToAudioRequest/Response in backend.proto. Forks without + // the realtime slot keep the base class answer, UNIMPLEMENTED. + grpc::Status AudioToAudioStream(ServerContext* context, + grpc::ServerReaderWriter* stream) override { + auto auth = checkAuth(context); + if (!auth.ok()) return auth; + if (params_base.model.path.empty()) { + return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded"); + } + RealtimeStreamSession session(ctx_server, params_base, context, stream); + return session.run(); + } +#endif + // Score returns the model's joint log-probability of each candidate // continuation given a shared prompt. // @@ -4205,24 +4991,37 @@ public: std::string b64 = base64_encode_bytes(bytes.data(), bytes.size()); - // Build the same prompt upstream uses in convert_transcriptions_to_chatcmpl. - std::string user_prompt = "Transcribe audio to text"; - if (!request->language().empty()) { - user_prompt += " (language: " + request->language() + ")"; - } - if (!request->prompt().empty()) { - // Optional context hint from the caller. - user_prompt += "\n" + request->prompt(); - } + // The model's own ASR preset, as upstream convert_transcriptions_to_chatcmpl + // builds it. LFM3 needs a system turn naming the language and an empty user + // turn; Qwen3-ASR gets a fixed user text and no system turn. Forks without + // the preset API keep the Qwen3-ASR text. +#if LOCALAI_HAS_ASR_PRESET + const common_chat_prompt_preset preset = + common_chat_get_asr_prompt(ctx_server.impl->chat_params.tmpls.get(), request->language()); + const std::string preset_system = preset.system; + const std::string preset_user = preset.user; + const bool preset_has_language = preset.has_language; +#else + const std::string preset_system; + const std::string preset_user = "Transcribe audio to text"; + const bool preset_has_language = false; +#endif + const auto messages = llama_grpc::build_transcription_messages( + preset_system, preset_user, preset_has_language, request->language(), request->prompt()); backend::PredictOptions synthetic; synthetic.set_usetokenizertemplate(true); synthetic.set_temperature(request->temperature()); // Generation length: leave at 0 so parse_options uses -1 (model default). // The model's stop tokens / EOS handle termination naturally for ASR. + if (!messages.system.empty()) { + backend::Message* sys = synthetic.add_messages(); + sys->set_role("system"); + sys->set_content(messages.system); + } backend::Message* msg = synthetic.add_messages(); msg->set_role("user"); - msg->set_content(user_prompt); + msg->set_content(messages.user); synthetic.add_audios(b64); return Predict(context, &synthetic, out_reply); diff --git a/backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch b/backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch index 47f46788f..a29fa0cc5 100644 --- a/backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch +++ b/backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch @@ -1,8 +1,8 @@ diff --git a/common/common.cpp b/common/common.cpp -index 6ef3b78..037a00b 100644 +index de140a2..d39eaa8 100644 --- a/common/common.cpp +++ b/common/common.cpp -@@ -1716,8 +1716,10 @@ struct llama_context_params common_context_params_to_llama(const common_params & +@@ -1757,8 +1757,10 @@ struct llama_context_params common_context_params_to_llama(const common_params & auto cparams = llama_context_default_params(); cparams.n_ctx = params.n_ctx; @@ -16,10 +16,10 @@ index 6ef3b78..037a00b 100644 cparams.n_outputs_max_per_seq = std::max(params.n_outputs_max_per_seq, 0); cparams.n_batch = params.n_batch; diff --git a/common/common.h b/common/common.h -index 70d6a67..93e2269 100644 +index ab08d15..414a76a 100644 --- a/common/common.h +++ b/common/common.h -@@ -460,6 +460,9 @@ struct common_params { +@@ -462,6 +462,9 @@ struct common_params { int32_t n_keep = 0; // number of tokens to keep from initial prompt int32_t n_chunks = -1; // max number of chunks to process (-1 = unlimited) int32_t n_parallel = 1; // number of parallel sequences to decode @@ -30,10 +30,10 @@ index 70d6a67..93e2269 100644 int32_t n_outputs_max = 0; // max outputs in a batch (0 = n_batch) int32_t n_outputs_max_per_seq = 1; // max outputs per sequence diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp -index 296893f..ef38f90 100644 +index 9377e3c..b0a0bf9 100644 --- a/tools/server/server-context.cpp +++ b/tools/server/server-context.cpp -@@ -60,6 +60,13 @@ static common_speculative_output_limits server_output_limits(const common_params +@@ -62,6 +62,13 @@ static common_speculative_output_limits server_output_limits(const common_params auto result = common_speculative_get_output_limits( params.n_batch, params.n_parallel, common_speculative_n_max(¶ms.speculative)); @@ -47,7 +47,7 @@ index 296893f..ef38f90 100644 result.total = std::max(1, result.total); result.per_seq = std::max(1, result.per_seq); return result; -@@ -325,6 +332,26 @@ struct server_slot { +@@ -327,6 +334,26 @@ struct server_slot { std::vector generated_token_probs; @@ -74,7 +74,7 @@ index 296893f..ef38f90 100644 bool has_next_token = true; bool has_new_line = false; bool truncated = false; -@@ -433,6 +460,10 @@ struct server_slot { +@@ -439,6 +466,10 @@ struct server_slot { } generated_tokens.clear(); generated_token_probs.clear(); @@ -85,7 +85,7 @@ index 296893f..ef38f90 100644 json_schema = json(); task_prev = std::move(task); -@@ -2571,6 +2602,227 @@ private: +@@ -2627,6 +2658,227 @@ private: queue_results.send(std::move(res)); } @@ -313,15 +313,15 @@ index 296893f..ef38f90 100644 // // Functions to process the task // -@@ -2957,6 +3209,7 @@ private: +@@ -3014,6 +3266,7 @@ private: case SERVER_TASK_TYPE_RERANK: case SERVER_TASK_TYPE_TTS: case SERVER_TASK_TYPE_DECISION: + case SERVER_TASK_TYPE_SCORE: { - // special case: if input is provided via CLI, tokenize it first - // otherwise, no need to tokenize as it's already done inside the HTTP thread -@@ -3549,6 +3802,13 @@ private: + if (task.need_sampling() && !decision.can_generate()) { + send_error(task, "This model is a decision model without text generation, use /v1/systemone", ERROR_TYPE_NOT_SUPPORTED); +@@ -3666,6 +3919,13 @@ private: break; // stop any further processing } } @@ -334,8 +334,8 @@ index 296893f..ef38f90 100644 + } } - void process_tts_slots() { -@@ -3987,6 +4247,16 @@ private: + // realtime conversations (SERVER_TASK_TYPE_REALTIME) drive their own decode loop, see server_realtime_slot +@@ -4120,6 +4380,16 @@ private: n_past = std::min(n_past, slot.alora_invocation_start - 1); } @@ -352,7 +352,7 @@ index 296893f..ef38f90 100644 const auto n_cache_reuse = slot.task->params.n_cache_reuse; const bool can_cache_reuse = -@@ -4257,8 +4527,12 @@ private: +@@ -4390,8 +4660,12 @@ private: bool do_checkpoint = params_base.n_ctx_checkpoints > 0; @@ -367,7 +367,7 @@ index 296893f..ef38f90 100644 // make a checkpoint of the parts of the memory that cannot be rolled back. // checkpoints are created only if: -@@ -4356,16 +4630,49 @@ private: +@@ -4489,16 +4763,49 @@ private: // embedding requires all tokens in the batch to be output; // MTP also wants logits at every prompt position so the // streaming hook can mirror t_h_nextn into ctx_dft. @@ -418,7 +418,7 @@ index 296893f..ef38f90 100644 // break at the last user message, or at user messages at least min step past the last checkpoint if (do_checkpoint && spans.is_user_start(slot.prompt.n_tokens())) { const auto pos = slot.prompt.n_tokens(); -@@ -4408,6 +4715,15 @@ private: +@@ -4541,6 +4848,15 @@ private: const bool is_user_start = spans.is_user_start(n_tokens_start); const bool is_last_user_message = n_tokens_start == last_user_pos; @@ -434,7 +434,7 @@ index 296893f..ef38f90 100644 // entire prompt has been processed if (slot.prompt.n_tokens() == slot.task->n_tokens()) { slot.state = SLOT_STATE_DONE_PROMPT; -@@ -4423,8 +4739,8 @@ private: +@@ -4556,8 +4872,8 @@ private: slot.init_sampler(); } else { // skip ordinary mid-prompt checkpoints, unless the batch starts a user @@ -445,7 +445,7 @@ index 296893f..ef38f90 100644 do_checkpoint = false; } } -@@ -4441,10 +4757,10 @@ private: +@@ -4574,10 +4890,10 @@ private: // do not checkpoint after mtmd chunks do_checkpoint = do_checkpoint && !has_mtmd; @@ -458,7 +458,7 @@ index 296893f..ef38f90 100644 n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step); SLT_DBG(slot, "main/do_checkpoint = %s, pos_min = %d, pos_max = %d\n", do_checkpoint ? "yes" : "no", pos_min, pos_max); -@@ -4690,6 +5006,13 @@ private: +@@ -4823,6 +5139,13 @@ private: } } @@ -472,7 +472,7 @@ index 296893f..ef38f90 100644 if (!is_inside_view(slot.i_batch)) { // the required token not in this sub-batch, skip return; -@@ -4718,6 +5041,25 @@ private: +@@ -4851,6 +5174,25 @@ private: return; } @@ -499,7 +499,7 @@ index 296893f..ef38f90 100644 // prompt evaluated for next-token prediction diff --git a/tools/server/server-task.h b/tools/server/server-task.h -index 810cefb..3e1ab8d 100644 +index 21b649a..eb857c9 100644 --- a/tools/server/server-task.h +++ b/tools/server/server-task.h @@ -13,11 +13,26 @@ @@ -528,8 +528,8 @@ index 810cefb..3e1ab8d 100644 + SERVER_TASK_TYPE_SCORE, SERVER_TASK_TYPE_INFILL, SERVER_TASK_TYPE_TRANSCRIBE_RT, - SERVER_TASK_TYPE_CANCEL, -@@ -168,6 +183,18 @@ struct server_task { + SERVER_TASK_TYPE_REALTIME, // a conversation over /v1/realtime, see server-realtime.cpp +@@ -183,6 +198,18 @@ struct server_task { task_params params; server_tokens tokens; @@ -548,7 +548,7 @@ index 810cefb..3e1ab8d 100644 // only used by CLI, this allow tokenizing CLI inputs on server side // we need this because mtmd_context and vocab are not accessible outside of server_context bool cli = false; -@@ -249,6 +276,7 @@ struct server_task { +@@ -269,6 +296,7 @@ struct server_task { case SERVER_TASK_TYPE_COMPLETION: case SERVER_TASK_TYPE_INFILL: case SERVER_TASK_TYPE_TRANSCRIBE_RT: @@ -556,7 +556,7 @@ index 810cefb..3e1ab8d 100644 return true; case SERVER_TASK_TYPE_DECISION: return !decision.labels.empty(); -@@ -558,6 +586,25 @@ struct server_task_result_decision : server_task_result { +@@ -594,6 +622,25 @@ struct server_task_result_decision : server_task_result { virtual json to_json() override; }; diff --git a/backend/cpp/llama-cpp/prepare.sh b/backend/cpp/llama-cpp/prepare.sh index 3d70f5447..d3659fe87 100644 --- a/backend/cpp/llama-cpp/prepare.sh +++ b/backend/cpp/llama-cpp/prepare.sh @@ -62,6 +62,10 @@ cp -r passthrough_options_test.cpp llama.cpp/tools/grpc-server/ # regression test. cp -r tts_request_options.h llama.cpp/tools/grpc-server/ cp -r tts_request_options_test.cpp llama.cpp/tools/grpc-server/ +# Transcription prompt construction (included by grpc-server.cpp) and its +# standalone regression test. +cp -r transcription_prompt.h llama.cpp/tools/grpc-server/ +cp -r transcription_prompt_test.cpp llama.cpp/tools/grpc-server/ # Thread-count default normalization and its standalone regression test. cp -r thread_params.h llama.cpp/tools/grpc-server/ cp -r thread_params_test.cpp llama.cpp/tools/grpc-server/ @@ -80,6 +84,12 @@ cp -r parent_watch_test.cpp llama.cpp/tools/grpc-server/ # test (run via backend/cpp/run-unit-tests.sh; also buildable under ctest). cp -r stream_peer.h llama.cpp/tools/grpc-server/ cp -r stream_peer_test.cpp llama.cpp/tools/grpc-server/ +# Conversation logic of AudioToAudioStream (included by grpc-server.cpp), its +# standalone unit test, and the check against the real LFM3 chat template +# (ctest only, see CMakeLists.txt). +cp -r realtime_session.h llama.cpp/tools/grpc-server/ +cp -r realtime_session_test.cpp llama.cpp/tools/grpc-server/ +cp -r realtime_chat_check.cpp llama.cpp/tools/grpc-server/ # Slot cache save/restore helpers (included by grpc-server.cpp) and their # unit test (compiled only when -DLLAMA_GRPC_BUILD_TESTS=ON). cp -r slot_cache.h llama.cpp/tools/grpc-server/ @@ -128,6 +138,27 @@ if grep -q "llama_lazy_mode lazy_mode" llama.cpp/common/common.h; then else HAS_LAZY_MODE=0 fi +# Realtime conversations (SERVER_TASK_TYPE_REALTIME, server-realtime.cpp). +# Forks that branched before the upstream realtime split lack the task type. +if grep -q "SERVER_TASK_TYPE_REALTIME" llama.cpp/tools/server/server-task.h; then + HAS_REALTIME=1 +else + HAS_REALTIME=0 +fi +# Per-session TTS voice (mtmd_helper_gen_audio_inp::set_voice). Older pins have +# no voice selection for gen-audio models. +if grep -q "set_voice" llama.cpp/tools/mtmd/mtmd-helper.h; then + HAS_TTS_VOICE=1 +else + HAS_TTS_VOICE=0 +fi +# Per-task ASR preset (common_chat_get_asr_prompt takes the requested language, +# ggml-org/llama.cpp#31257). Older pins have the one-argument form, or none. +if grep -qF "common_chat_get_asr_prompt(const common_chat_templates * chat_templates, const std::string & language" llama.cpp/common/chat.h; then + HAS_ASR_PRESET=1 +else + HAS_ASR_PRESET=0 +fi # Newer servers return template state separately from the request JSON. if grep -q "common_chat_session & out_session" llama.cpp/tools/server/server-common.h; then HAS_CHAT_SESSION=1 @@ -152,6 +183,9 @@ cat > llama.cpp/tools/grpc-server/llama_compat.h < +#include +#include +#include + +namespace rts = llama_grpc::realtime; + +namespace { + +int failures = 0; + +void check(bool ok, const char * what) { + if (!ok) { + std::fprintf(stderr, "FAIL: %s\n", what); + ++failures; + } +} + +void check_eq(const std::string & got, const std::string & want, const char * what) { + if (got != want) { + std::fprintf(stderr, "FAIL: %s\n--- got\n%s\n--- want\n%s\n", what, got.c_str(), want.c_str()); + ++failures; + } +} + +const char * kTools = R"([{"type":"function","function":{"name":"get_time","description":"Get the time", + "parameters":{"type":"object","properties":{"tz":{"type":"string"}},"required":["tz"]}}}])"; + +std::vector to_common(const std::vector & msgs) { + std::vector out; + for (const auto & m : msgs) { + common_chat_msg c; + c.role = m.role; + c.content = m.content; + c.tool_call_id = m.tool_call_id; + for (const auto & tc : m.tool_calls) { + c.tool_calls.push_back({tc.name, tc.arguments, tc.id}); + } + out.push_back(c); + } + return out; +} + +} // namespace + +int main(int argc, char ** argv) { + if (argc < 2) { + std::fprintf(stderr, "usage: %s \n", argv[0]); + return 2; + } + std::ifstream f(argv[1]); + if (!f) { + std::fprintf(stderr, "cannot open %s\n", argv[1]); + return 2; + } + std::stringstream ss; + ss << f.rdbuf(); + common_chat_templates_ptr tmpls(common_chat_templates_init(nullptr, ss.str())); + + for (const bool with_tools : {false, true}) { + std::vector tools; + if (with_tools) { + tools = common_chat_tools_parse_oaicompat(common_json::parse(kTools)); + } + auto base = [&]() { + common_chat_templates_inputs in; + in.use_jinja = true; + in.enable_thinking = false; + in.tools = tools; + return in; + }; + rts::RenderFn render = [&](const std::vector & msgs, bool gen) { + auto in = base(); + in.messages = to_common(msgs); + in.add_generation_prompt = gen; + return common_chat_templates_apply(tmpls.get(), in).prompt.str(); + }; + + rts::TurnText t; + std::string err; + check(rts::derive_turn_text(render, rts::default_instructions(with_tools), with_tools ? "get_time" : "", t, err), "LFM3 template fits the turn format"); + if (!err.empty()) { + std::fprintf(stderr, " %s\n", err.c_str()); + } + check(t.has_tool_turn == with_tools, "LFM3 renders a tool turn when the session has tools"); + if (with_tools) { + check(t.first_header.find("") != std::string::npos || t.first_header.find("get_time") != std::string::npos, + "the first header lists the tools"); + check(t.first_header.find("Respond with function calls") != std::string::npos, "the first header asks for function calls"); + } + + // what the slot sees must equal what the template renders for the whole conversation + const auto sys = rts::Msg{"system", rts::default_instructions(with_tools), {}, ""}; + auto m = [](const char * role, const char * content) { return rts::Msg{role, content, {}, ""}; }; + const std::string slot_two_turns = t.first_header + "hi" + t.user_footer + t.assistant_header + "hello" + t.assistant_footer + + t.user_header + "bye" + t.user_footer + t.assistant_header; + check_eq(slot_two_turns, render({sys, m("user", "hi"), m("assistant", "hello"), m("user", "bye")}, true), "two user turns"); + + if (with_tools) { + // the response is the call the model wrote, the result of the tool follows + const std::string call = "\n\n\n"; + const std::string slot_tool = t.first_header + "hi" + t.user_footer + t.assistant_header + call + t.assistant_footer + + t.tool_header + "12:00" + t.tool_footer + t.assistant_header_after_tool; + rts::Msg assistant{"assistant", "", {{"get_time", "{}", "call_0"}}, ""}; + rts::Msg result{"tool", "12:00", {}, "call_0"}; + check_eq(slot_tool, render({sys, m("user", "hi"), assistant, result}, true), "tool turn"); + } + std::printf("tools=%d\n first_header=%s\n tool_header=%s\n tool_footer=%s\n assistant_footer=%s\n", with_tools, + rts::json(t.first_header).dump().c_str(), rts::json(t.tool_header).dump().c_str(), + rts::json(t.tool_footer).dump().c_str(), rts::json(t.assistant_footer).dump().c_str()); + } + + // the tool-call block the model writes goes through the grammar and the incremental parser of the template + { + common_chat_templates_inputs in; + in.use_jinja = true; + in.enable_thinking = false; + in.add_generation_prompt = true; + in.parallel_tool_calls = true; + in.reasoning_format = COMMON_REASONING_FORMAT_AUTO; // as the server does + in.tools = common_chat_tools_parse_oaicompat(common_json::parse(kTools)); + common_chat_msg user; + user.role = "user"; + user.content = "x"; + in.messages.push_back(user); + + { + common_chat_session session(tmpls.get(), nullptr, in); + common_params_sampling sampling; + session.apply_sampling(sampling); + check(!sampling.grammar.grammar.empty(), "the template gives a grammar for the tools"); + check(!sampling.preserved_tokens.empty() || !sampling.grammar_triggers.empty(), "the template names the tokens of a call"); + } + + const std::string call = "\n\n\nUTC\n\n\n"; + const std::string text = "One moment.\n" + call; + + // the splitter of grpc-server.cpp, on the session API only: the same diffs, fed piece by piece + auto make_splitter = [&]() -> rts::ResponseTranslator::Splitter { + auto session = std::make_shared(tmpls.get(), nullptr, in); + auto prev = std::make_shared(); + return [session, prev](const std::string & piece, bool final) { + rts::SplitResult out; + const auto & msg = final ? session->finish(common_chat_input(piece)) : session->feed(common_chat_input(piece)); + for (const auto & d : common_chat_msg_diff::compute_diffs(*prev, msg)) { + out.content += d.content_delta; + if (d.tool_call_index != std::string::npos) { + out.calls.push_back({d.tool_call_index, d.tool_call_delta.id, d.tool_call_delta.name, d.tool_call_delta.arguments}); + } + } + *prev = msg; + return out; + }; + }; + + for (const size_t step : {size_t(7), size_t(1), text.size()}) { + rts::ResponseTranslator tr(rts::OutMode::Auto, 24000, make_splitter()); + std::vector all; + for (size_t i = 0; i < text.size(); i += step) { + auto ev = tr.on_result(text.substr(i, step), {0.1f}, false, false, 1); + all.insert(all.end(), ev.begin(), ev.end()); + } + auto ev = tr.on_result("", {}, true, false, 9); + all.insert(all.end(), ev.begin(), ev.end()); + + std::string transcript, name, args; + size_t calls = 0; + for (const auto & e : all) { + if (e.event == "response.audio_transcript.delta") { + transcript += e.meta["delta"].get(); + } else if (e.event == "response.function_call_arguments.done") { + calls++; + name = e.meta["name"].get(); + args = e.meta["arguments"].get(); + } + } + const std::string label = "step " + std::to_string(step) + ": "; + check(calls == 1, (label + "one tool call").c_str()); + check_eq(name, "get_time", (label + "tool name").c_str()); + const auto parsed = rts::json::parse(args, nullptr, false); + check(parsed.is_object() && parsed.value("tz", "") == "UTC", (label + "tool arguments are a JSON object").c_str()); + check_eq(transcript, "One moment.", (label + "the call is not in the transcript").c_str()); + size_t audio_frames = 0; + for (const auto & e : all) { + if (e.event == "response.audio.delta") { + audio_frames++; + } + } + check(audio_frames > 0, (label + "audio is sent while the call is written").c_str()); + check(all.back().event == "response.done", (label + "response.done is last").c_str()); + check(all.back().meta["raw_text"].get() == text, (label + "response.done carries the raw text").c_str()); + } + + // a reply without a call is only content + { + rts::ResponseTranslator tr(rts::OutMode::Auto, 24000, make_splitter()); + auto ev = tr.on_result("It is noon.", {}, false, false, 1); + auto fin = tr.on_result("", {}, true, false, 2); + ev.insert(ev.end(), fin.begin(), fin.end()); + std::string transcript; + for (const auto & e : ev) { + check(e.event != "response.function_call_arguments.done", "no call in a plain reply"); + if (e.event == "response.audio_transcript.delta") { + transcript += e.meta["delta"].get(); + } + } + check_eq(transcript, "It is noon.", "a plain reply is the transcript"); + } + } + + if (failures) { + std::fprintf(stderr, "%d check(s) failed\n", failures); + return 1; + } + std::puts("realtime_chat_check: all checks passed"); + return 0; +} diff --git a/backend/cpp/llama-cpp/realtime_session.h b/backend/cpp/llama-cpp/realtime_session.h new file mode 100644 index 000000000..c99123f87 --- /dev/null +++ b/backend/cpp/llama-cpp/realtime_session.h @@ -0,0 +1,707 @@ +// SPDX-License-Identifier: MIT +#pragma once + +// Conversation logic of AudioToAudioStream for speech-to-speech models that run +// on llama.cpp's realtime slot (server_realtime_slot, LFM3-Audio). +// +// Everything here is header-only and free of llama.cpp and gRPC types, so it can +// be tested on its own (see realtime_session_test.cpp). grpc-server.cpp supplies +// the pieces that need the model: a chat-template renderer (RenderFn) and the +// parser that splits the text of a response into content and tool calls +// (ResponseTranslator::Splitter). +// +// The functions mirror the websocket conversation upstream keeps in +// tools/server/server-realtime.cpp (derive_turn_text, ensure_turn, commit, +// on_item_create, on_response_create, on_result), including the tool turn that +// is derived from the chat template and the incremental tool-call parsing. + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace llama_grpc { +namespace realtime { + +using json = nlohmann::json; + +// System prompts of the LFM3-Audio model card. Without tools the model answers in +// interleaved text and audio. With tools the card asks for "function calls, and +// interleaved text and audio": under the plain interleaved prompt the model +// answers in words ("I'll check the weather ...") and never writes the call. +// Used when the client sends no instructions. The tools prompt ends with the +// label of the tool list the chat template adds. +inline const char * default_instructions(bool has_tools = false) { + return has_tools ? "Respond with function calls, and interleaved text and audio.\n\nList of tools:" + : "Respond with interleaved text and audio."; +} + +// True for either default. A client that sends one back (the LocalAI server +// does, as the default of the session) has not chosen it, so it follows the +// tools of the session. +inline bool is_default_instructions(const std::string & s) { + return s == default_instructions(false) || s == default_instructions(true); +} + +// The instructions of a session: what the client asked for, or the default for +// the tools it has. Empty requested means the client sent none. +inline std::string resolve_instructions(const std::string & requested, bool has_tools) { + return requested.empty() || is_default_instructions(requested) ? std::string(default_instructions(has_tools)) : requested; +} + +// Voices of the LFM3 "tts" chat template. +inline std::vector default_voices() { + return {"female", "male"}; +} + +// UTF-8 safe JSON text. Token pieces can end in the middle of a character, and +// nlohmann::json::dump() throws on invalid UTF-8 unless told to replace it. +inline std::string dump_json(const json & j) { + return j.dump(-1, ' ', false, json::error_handler_t::replace); +} + +// +// chat template: the text around a turn +// + +struct ToolCallMsg { + std::string name; + std::string arguments; // JSON object as text + std::string id; +}; + +struct Msg { + std::string role; // system, user, assistant, tool + std::string content; + std::vector tool_calls; + std::string tool_call_id; // of a tool message: the call it answers +}; + +// Renders messages with the chat template of the model. The tools of the session +// are captured by the renderer, not passed here. +using RenderFn = std::function & messages, bool add_generation_prompt)>; + +struct TurnText { + std::string first_header; // system prompt (with the tools) + the header of the first user turn + std::string user_header; // header of a later user turn + std::string user_footer; // closes a user turn + std::string assistant_header; // opens the response + std::string assistant_footer; // closes the response, goes in before the next turn + // tool turn: the result of a tool call. Absent if the template does not render one. + bool has_tool_turn = false; + std::string tool_header; + std::string tool_footer; + std::string assistant_header_after_tool; // opens the response that follows a tool turn +}; + +// Derives the text around the turns from placeholder messages, the same method +// as upstream derive_turn_text(): render a few conversations and cut the +// placeholders out. Returns false, with err set, if the template does not have +// the shape of a turn-based chat. tool_name is any tool of the session: the tool +// turn is cut out of a conversation where the assistant called it. Empty means +// the session has no tools and so no tool turn. +inline bool derive_turn_text(const RenderFn & render, const std::string & instructions, const std::string & tool_name, TurnText & out, + std::string & err) { + const std::string user = "@@USER@@"; + const std::string reply = "@@REPLY@@"; + const std::string tool = "@@RESULT@@"; + auto msg = [](const char * role, const std::string & content) { + Msg m; + m.role = role; + m.content = content; + return m; + }; + std::vector sys; + if (!instructions.empty()) { + sys.push_back(msg("system", instructions)); + } + try { + auto with = [&](const std::vector & rest, bool add_generation_prompt = false) { + std::vector all = sys; + all.insert(all.end(), rest.begin(), rest.end()); + return render(all, add_generation_prompt); + }; + const std::string a = with({msg("user", user)}); + const std::string g = with({msg("user", user)}, true); + const std::string c = with({msg("user", user), msg("assistant", reply)}); + const std::string d = with({msg("user", user), msg("assistant", reply), msg("user", user)}); + + const size_t pa = a.find(user); + const size_t pc = c.find(reply); + const size_t pd = d.find(user, c.size()); + if (pa == std::string::npos || pc == std::string::npos || pd == std::string::npos || + c.compare(0, a.size(), a) != 0 || d.compare(0, c.size(), c) != 0 || g.compare(0, a.size(), a) != 0 || pc < a.size()) { + err = "the chat template does not fit the realtime turn format"; + return false; + } + TurnText t; + // the response opens with the generation prompt, a past turn may render differently (e.g. its think block) + t.first_header = a.substr(0, pa); + t.user_footer = a.substr(pa + user.size()); + t.assistant_header = g.substr(a.size()); + t.assistant_footer = c.substr(pc + reply.size()); + t.user_header = d.substr(c.size(), pd - c.size()); + + // tool turn: a tool message after a response that called a tool + if (!tool_name.empty()) { + Msg call = msg("assistant", ""); + call.tool_calls.push_back({tool_name, "{}", "call_0"}); + Msg res = msg("tool", tool); + res.tool_call_id = "call_0"; + const std::string e = with({msg("user", user), call}); + const std::string f = with({msg("user", user), call, res}); + const std::string fg = with({msg("user", user), call, res}, true); + const size_t pt = f.find(tool, e.size()); + if (pt != std::string::npos && f.compare(0, e.size(), e) == 0 && fg.compare(0, f.size(), f) == 0) { + t.tool_header = f.substr(e.size(), pt - e.size()); + t.tool_footer = f.substr(pt + tool.size()); + t.assistant_header_after_tool = fg.substr(f.size()); + t.has_tool_turn = true; + } + } + out = std::move(t); + return true; + } catch (const std::exception & ex) { + err = std::string("the chat template failed: ") + ex.what(); + return false; + } +} + +// +// session parameters +// + +enum class OutMode { Auto, Text, Audio, Mixed }; + +// output_modalities of a response. null: the model switches between text and +// audio itself. +inline bool parse_output_modalities(const json & j, OutMode & mode, std::string & err) { + if (j.is_null()) { + mode = OutMode::Auto; + return true; + } + if (!j.is_array() || j.empty()) { + err = "output_modalities must be a non-empty array"; + return false; + } + bool text = false; + bool audio = false; + for (const auto & m : j) { + if (m == "text") { + text = true; + } else if (m == "audio") { + audio = true; + } else { + err = "output_modalities: unknown modality " + m.dump(); + return false; + } + } + if (text && audio) { + mode = OutMode::Mixed; + } else { + mode = audio ? OutMode::Audio : OutMode::Text; + } + return true; +} + +inline json out_mode_json(OutMode mode) { + switch (mode) { + case OutMode::Text: return json::array({"text"}); + case OutMode::Audio: return json::array({"audio"}); + case OutMode::Mixed: return json::array({"text", "audio"}); + default: return nullptr; + } +} + +// -1 is "until the context is full", like upstream "inf" +inline bool parse_max_output_tokens(const json & j, int32_t & n) { + if (j.is_string() && j.get() == "inf") { + n = -1; + return true; + } + if (!j.is_number_integer() || j.get() < 1) { + return false; + } + n = (int32_t) std::min(j.get(), INT32_MAX); + return true; +} + +// tool_choice of a session or a response: the model may call a tool (auto), +// must not (none) or must (required). +enum class ToolChoice { Auto, None, Required }; + +inline bool parse_tool_choice(const json & j, ToolChoice & choice, std::string & err) { + const std::string v = j.is_string() ? j.get() : std::string(); + if (v == "auto") { + choice = ToolChoice::Auto; + } else if (v == "none") { + choice = ToolChoice::None; + } else if (v == "required") { + choice = ToolChoice::Required; + } else { + err = "tool_choice must be \"auto\", \"none\" or \"required\""; + return false; + } + return true; +} + +// Mixed so that a transcript exists. With tools the model chooses when to speak: +// forcing interleaved audio makes it read out the tool call. +inline OutMode default_out_mode(bool has_tools) { + return has_tools ? OutMode::Auto : OutMode::Mixed; +} + +// AudioToAudioConfig.params: free-form strings. +struct SessionParams { + std::optional out_mode; + ToolChoice tool_choice = ToolChoice::Auto; + int32_t max_output_tokens = -1; + std::optional temperature; + std::optional top_p; + std::optional top_k; + std::optional seed; +}; + +// output_modalities is "text", "audio", "text,audio" or a JSON array. +inline bool parse_modalities_string(const std::string & s, OutMode & mode, std::string & err) { + json arr = json::array(); + if (!s.empty() && s[0] == '[') { + arr = json::parse(s, nullptr, false); + if (arr.is_discarded()) { + err = "output_modalities is not valid JSON"; + return false; + } + } else { + size_t start = 0; + while (start <= s.size()) { + size_t end = s.find(',', start); + if (end == std::string::npos) { + end = s.size(); + } + std::string part = s.substr(start, end - start); + part.erase(std::remove_if(part.begin(), part.end(), [](unsigned char ch) { return std::isspace(ch); }), part.end()); + if (!part.empty()) { + arr.push_back(part); + } + start = end + 1; + } + } + return parse_output_modalities(arr, mode, err); +} + +inline bool parse_session_params(const std::map & params, SessionParams & out, std::string & err) { + SessionParams p; + try { + for (const auto & kv : params) { + const std::string & k = kv.first; + const std::string & v = kv.second; + if (k == "output_modalities") { + OutMode m; + if (!parse_modalities_string(v, m, err)) { + return false; + } + p.out_mode = m; + } else if (k == "tool_choice") { + if (!parse_tool_choice(json(v), p.tool_choice, err)) { + return false; + } + } else if (k == "max_output_tokens" || k == "max_new_tokens") { + json j = v == "inf" ? json(v) : json::parse(v, nullptr, false); + if (j.is_discarded() || !parse_max_output_tokens(j, p.max_output_tokens)) { + err = k + " must be a positive integer or \"inf\""; + return false; + } + } else if (k == "temperature") { + p.temperature = std::stof(v); + } else if (k == "top_p") { + p.top_p = std::stof(v); + } else if (k == "top_k") { + p.top_k = (int32_t) std::stol(v); + } else if (k == "seed") { + p.seed = (uint32_t) std::stoul(v); + } + // other keys are for other backends + } + } catch (const std::exception &) { + err = "a sampling parameter is not a number"; + return false; + } + out = p; + return true; +} + +// Options of one response.create. They may sit at the top of the payload or under +// "response", the way the OpenAI event nests them. +struct ResponseOptions { + OutMode out_mode; + int32_t max_output_tokens; + ToolChoice tool_choice = ToolChoice::Auto; +}; + +inline bool parse_response_options(const json & payload, OutMode default_mode, int32_t default_max, ToolChoice default_choice, + ResponseOptions & out, std::string & err) { + ResponseOptions o{default_mode, default_max, default_choice}; + json src = payload.is_object() ? payload : json::object(); + if (src.contains("response") && src["response"].is_object()) { + src = src["response"]; + } + if (src.contains("output_modalities") && !parse_output_modalities(src["output_modalities"], o.out_mode, err)) { + return false; + } + if (src.contains("max_output_tokens") && !parse_max_output_tokens(src["max_output_tokens"], o.max_output_tokens)) { + err = "max_output_tokens must be an integer or \"inf\""; + return false; + } + if (src.contains("tool_choice") && !parse_tool_choice(src["tool_choice"], o.tool_choice, err)) { + return false; + } + out = o; + return true; +} + +// +// conversation state +// + +// Where the conversation is, and the text that has to go into the slot next. +// The caller applies a step to the slot first and records it here after the slot +// accepted it, so a failed control leaves the state as it was. +class ConversationState { +public: + enum class Turn { None, User, Tool }; + + bool started() const { return started_; } + Turn turn() const { return turn_; } + bool has_audio() const { return has_audio_; } + int n_turns() const { return n_turns_; } + + // the header a user turn needs before its first audio or text; empty if the turn is open + std::string user_turn_header(const TurnText & t) const { + if (turn_ == Turn::User) { + return {}; + } + return n_turns_ == 0 ? t.first_header : next_header_ + t.user_header; + } + bool user_turn_is_open() const { return turn_ == Turn::User; } + void user_turn_opened() { + started_ = true; + turn_ = Turn::User; + next_header_.clear(); + } + + void audio_appended() { has_audio_ = true; } + void audio_committed() { has_audio_ = false; } + + // The result of a tool call: closes the response, then the tool turn. + // Empty err and the text to append, or the reason it cannot be appended now. + std::string tool_turn_text(const TurnText & t, const std::string & output, std::string & err) const { + if (!t.has_tool_turn) { + err = "the chat template has no tool turn"; + return {}; + } + if (n_turns_ == 0) { + err = "function_call_output: there is no function call to answer"; + return {}; + } + if (turn_ == Turn::User) { + err = "function_call_output: a user turn is open"; + return {}; + } + // a second result of the same call round continues the open tool turn + const std::string lead = turn_ == Turn::Tool ? std::string() : next_header_; + return lead + t.tool_header + output + t.tool_footer; + } + void tool_turn_opened() { + started_ = true; + turn_ = Turn::Tool; + next_header_.clear(); + } + + // "" if a response can be created now, or why not + std::string response_create_error() const { + if (turn_ == Turn::None) { + return "response.create: commit audio or create a message first"; + } + return {}; + } + + // the text that opens the response: close the turn, then the assistant header + std::string response_prompt(const TurnText & t) const { + if (turn_ == Turn::Tool) { + return t.assistant_header_after_tool; + } + return t.user_footer + t.assistant_header; + } + + // the response ends the turn; its footer goes in before the next one + void response_created(const TurnText & t) { + next_header_ = t.assistant_footer; + turn_ = Turn::None; + n_turns_++; + } + + // "" if the session can change now. Before the first audio or message + // everything can change; after it only the response options. + std::string session_update_error(bool instructions, bool tools, bool voice) const { + if (!started_) { + return {}; + } + if (instructions) { + return "instructions cannot change after the first audio or message"; + } + if (tools) { + return "tools cannot change after the first audio or message"; + } + if (voice) { + return "voice cannot change after the first audio or message"; + } + return {}; + } + + void reset() { *this = ConversationState(); } + +private: + bool started_ = false; // something went into the slot, the session config is fixed + Turn turn_ = Turn::None; + bool has_audio_ = false; // the open turn has audio not committed yet + int n_turns_ = 0; // responses created + std::string next_header_; // text that goes in before the next turn +}; + +// +// audio +// + +// s16le mono to f32, an odd trailing byte is ignored +inline std::vector pcm_s16le_to_f32(const std::string & bytes) { + std::vector out(bytes.size() / 2); + for (size_t i = 0; i < out.size(); i++) { + int16_t v; + std::memcpy(&v, bytes.data() + 2 * i, sizeof(v)); + out[i] = v / 32768.0f; + } + return out; +} + +inline std::string f32_to_pcm_s16le(const std::vector & samples) { + std::string bytes(samples.size() * sizeof(int16_t), '\0'); + for (size_t i = 0; i < samples.size(); i++) { + const int16_t v = (int16_t) std::lround(std::max(-1.0f, std::min(1.0f, samples[i])) * 32767.0f); + std::memcpy(&bytes[2 * i], &v, sizeof(v)); + } + return bytes; +} + +// +// text: UTF-8 +// + +// Number of leading bytes of s that end on a character boundary. A token piece +// can stop inside a multi-byte character; the rest comes with the next piece. +inline size_t utf8_complete_prefix(const std::string & s) { + size_t i = s.size(); + // look back over at most 3 continuation bytes for the lead byte + size_t back = 0; + while (i > 0 && back < 4) { + const unsigned char ch = (unsigned char) s[i - 1]; + if ((ch & 0xC0) == 0x80) { + i--; + back++; + continue; + } + size_t need = 1; + if ((ch & 0xE0) == 0xC0) { + need = 2; + } else if ((ch & 0xF0) == 0xE0) { + need = 3; + } else if ((ch & 0xF8) == 0xF0) { + need = 4; + } + // lead byte at i - 1 with `back` continuation bytes after it + return (back + 1 >= need) ? s.size() : i - 1; + } + return s.size(); +} + +// +// events to the client +// + +struct OutEvent { + std::string event; + std::string pcm; + int sample_rate = 0; + json meta = json::object(); +}; + +// response_scoped marks an error that ends the request for a response: no +// response.done follows it. Other errors are about something else (a refused +// session.update, an empty commit) and a running response goes on. +inline json error_meta(const std::string & type, const std::string & message, bool response_scoped = false) { + json meta = {{"type", type}, {"message", message}}; + if (response_scoped) { + meta["response_scoped"] = true; + } + return meta; +} + +inline json session_meta(int input_rate, int output_rate, const std::vector & voices, const std::string & requested_voice) { + json meta = {{"input_sample_rate", input_rate}, {"output_sample_rate", output_rate}, {"voices", voices}}; + if (!requested_voice.empty()) { + // the realtime slot of llama.cpp has no voice selection + meta["voice"] = requested_voice; + meta["voice_applied"] = false; + } + return meta; +} + +inline std::string make_call_id() { + static thread_local std::mt19937_64 rng{std::random_device{}()}; + static const char hex[] = "0123456789abcdef"; + std::string id = "call_"; + for (int i = 0; i < 24; i++) { + id += hex[rng() & 15]; + } + return id; +} + +// A change in one tool call, as the incremental parser reports it. The name and +// the id come with the first delta of a call, arguments is the text that was +// added to the JSON arguments. +struct CallDelta { + size_t index = 0; + std::string id; + std::string name; + std::string arguments; +}; + +// What the parser made of a piece of the response: text for the transcript and +// changes in the tool calls. The text of a call is in neither. +struct SplitResult { + std::string content; + std::vector calls; +}; + +// Turns the results of the slot into the events of the stream, in order: +// text deltas and audio frames as they come, then one function_call_arguments.done +// per call, then response.done. +class ResponseTranslator { +public: + // Parses the text of a response as it grows. final is set with the last piece + // of an uncancelled response. Without a splitter every piece is content. + using Splitter = std::function; + + ResponseTranslator(OutMode mode, int sample_rate, Splitter splitter = nullptr, std::function id_gen = make_call_id) + : mode_(mode), sample_rate_(sample_rate), splitter_(std::move(splitter)), id_gen_(std::move(id_gen)) {} + + std::vector on_result(const std::string & text, const std::vector & audio, bool response_done, bool cancelled, int n_decoded) { + std::vector events; + if (done_) { + return events; + } + raw_ += text; + pending_ += text; + // a token piece can stop inside a character: the rest comes with the next piece + const size_t n = response_done ? pending_.size() : utf8_complete_prefix(pending_); + std::string chunk = pending_.substr(0, n); + pending_.erase(0, n); + + // Also on a cancelled response, as upstream: the parser flushes the text it held back. + // A cancelled response still makes no tool call (see below). + const bool final = response_done; + if (!chunk.empty() || (final && splitter_)) { + SplitResult r = splitter_ ? splitter_(chunk, final) : SplitResult{chunk, {}}; + if (!r.content.empty()) { + shown_ += r.content; + events.push_back(delta_event(r.content)); + } + for (const auto & d : r.calls) { + merge_call(d); + } + } + if (!audio.empty()) { + // Audio is never dropped because a call started, as upstream: in mixed + // mode the speech for text written before the call can still be pending. + OutEvent ev; + ev.event = "response.audio.delta"; + ev.pcm = f32_to_pcm_s16le(audio); + ev.sample_rate = sample_rate_; + events.push_back(std::move(ev)); + } + if (!response_done) { + return events; + } + done_ = true; + if (!cancelled) { + for (const auto & kv : calls_) { + const ToolCallMsg & call = kv.second; + if (call.name.empty()) { + continue; + } + OutEvent ev; + ev.event = "response.function_call_arguments.done"; + ev.meta = {{"call_id", call.id.empty() ? id_gen_() : call.id}, + {"name", call.name}, + {"arguments", call.arguments.empty() ? std::string("{}") : call.arguments}}; + events.push_back(std::move(ev)); + n_calls_++; + } + } + OutEvent ev; + ev.event = "response.done"; + ev.meta = {{"status", cancelled ? "cancelled" : "completed"}, {"usage", {{"output_tokens", n_decoded}}}, {"text", shown_}}; + if (splitter_) { + // what the model wrote, tool-call markup included, for a caller that parses it again + ev.meta["raw_text"] = raw_; + } + events.push_back(std::move(ev)); + return events; + } + + bool done() const { return done_; } + int n_calls() const { return n_calls_; } + const std::string & shown_text() const { return shown_; } + +private: + OutMode mode_; + int sample_rate_; + Splitter splitter_; + std::function id_gen_; + std::map calls_; + std::string raw_; // everything the model wrote, tool-call markup included + std::string shown_; // what went to the transcript + std::string pending_; // bytes of a character that is not complete yet + bool done_ = false; + int n_calls_ = 0; + + OutEvent delta_event(const std::string & delta) const { + OutEvent ev; + // a text-only response is plain text; otherwise the text is the transcript of the audio + ev.event = mode_ == OutMode::Text ? "response.text.delta" : "response.audio_transcript.delta"; + ev.meta = {{"delta", delta}}; + return ev; + } + + void merge_call(const CallDelta & d) { + ToolCallMsg & call = calls_[d.index]; + if (!d.id.empty()) { + call.id = d.id; + } + if (!d.name.empty()) { + call.name = d.name; + } + call.arguments += d.arguments; + } +}; + +} // namespace realtime +} // namespace llama_grpc diff --git a/backend/cpp/llama-cpp/realtime_session_test.cpp b/backend/cpp/llama-cpp/realtime_session_test.cpp new file mode 100644 index 000000000..d74fedc4b --- /dev/null +++ b/backend/cpp/llama-cpp/realtime_session_test.cpp @@ -0,0 +1,593 @@ +// SPDX-License-Identifier: MIT +#include "realtime_session.h" + +#include +#include +#include +#include + +namespace rt = llama_grpc::realtime; + +namespace { + +int failures = 0; + +void check(bool condition, const char * what) { + if (!condition) { + std::fprintf(stderr, "FAIL: %s\n", what); + ++failures; + } +} + +template +void check_eq(const A & got, const B & want, const char * what) { + if (!(got == want)) { + std::fprintf(stderr, "FAIL: %s\n got: %s\n want: %s\n", what, nlohmann::json(got).dump().c_str(), nlohmann::json(want).dump().c_str()); + ++failures; + } +} + +// A ChatML-like template. With tools, the system turn lists them in . +rt::RenderFn fake_renderer(const std::string & tools) { + return [tools](const std::vector & msgs, bool gen) { + std::string out; + bool has_sys = !msgs.empty() && msgs[0].role == "system"; + if (!has_sys && !tools.empty()) { + out += "<|im_start|>system\n" + tools + "<|im_end|>\n"; + } + for (const auto & m : msgs) { + if (m.role == "system") { + out += "<|im_start|>system\n" + m.content; + if (!tools.empty()) { + out += "\n" + tools + ""; + } + out += "<|im_end|>\n"; + } else if (m.role == "assistant" && !m.tool_calls.empty()) { + out += "<|im_start|>assistant\n"; + for (const auto & tc : m.tool_calls) { + out += "" + tc.name + ""; + } + out += "<|im_end|>\n"; + } else if (m.role == "tool") { + out += "<|im_start|>tool\n\n" + m.content + "\n<|im_end|>\n"; + } else { + out += "<|im_start|>" + m.role + "\n" + m.content + "<|im_end|>\n"; + } + } + if (gen) { + out += "<|im_start|>assistant\n"; + } + return out; + }; +} + +rt::TurnText derive(const std::string & instructions, const std::string & tools, const std::string & tool_name = "get_time") { + rt::TurnText t; + std::string err; + check(rt::derive_turn_text(fake_renderer(tools), instructions, tool_name, t, err), "derive_turn_text succeeds"); + return t; +} + +void test_turn_text() { + const auto t = derive("Be brief.", ""); + check_eq(t.first_header, std::string("<|im_start|>system\nBe brief.<|im_end|>\n<|im_start|>user\n"), "first header"); + check_eq(t.user_header, std::string("<|im_start|>user\n"), "user header"); + check_eq(t.user_footer, std::string("<|im_end|>\n"), "user footer"); + check_eq(t.assistant_header, std::string("<|im_start|>assistant\n"), "assistant header"); + check_eq(t.assistant_footer, std::string("<|im_end|>\n"), "assistant footer"); + check(t.has_tool_turn, "a tool turn is derived"); + check_eq(t.tool_header, std::string("<|im_start|>tool\n\n"), "tool header"); + check_eq(t.tool_footer, std::string("\n<|im_end|>\n"), "tool footer"); + check_eq(t.assistant_header_after_tool, std::string("<|im_start|>assistant\n"), "assistant header after a tool turn"); +} + +void test_tool_turn_needs_a_tool() { + // the tool turn follows an assistant turn that called a tool, so a session without tools has none + check(!derive("Be brief.", "", "").has_tool_turn, "no tool, no tool turn"); +} + +void test_turn_text_with_tools() { + const std::string tools = R"([{"name":"get_time"}])"; + const auto t = derive("Be brief.", tools); + check(t.first_header.find("" + tools + "") != std::string::npos, "the system turn lists the tools"); + check_eq(t.first_header.compare(t.first_header.size() - 17, 17, "<|im_start|>user\n"), 0, "the first header ends with the user header"); + // tools only change the system turn + check_eq(t.user_header, derive("Be brief.", "").user_header, "tools do not change the user header"); +} + +void test_turn_text_without_instructions() { + const auto t = derive("", R"([{"name":"f"}])"); + check(t.first_header.find("") != std::string::npos, "the template adds the tools turn itself"); +} + +void test_turn_text_rejects_bad_template() { + rt::RenderFn broken = [](const std::vector &, bool) { return std::string("no placeholders"); }; + rt::TurnText t; + std::string err; + check(!rt::derive_turn_text(broken, "x", "f", t, err), "a template that drops the content is rejected"); + check(!err.empty(), "the reason is reported"); + + rt::RenderFn throwing = [](const std::vector &, bool) -> std::string { throw std::runtime_error("boom"); }; + err.clear(); + check(!rt::derive_turn_text(throwing, "x", "f", t, err), "a throwing template is rejected"); + check(err.find("boom") != std::string::npos, "the exception text is reported"); +} + +void test_no_tool_turn() { + // a template that cannot render a tool message after a plain reply + rt::RenderFn base = fake_renderer(""); + rt::RenderFn no_tool = [base](const std::vector & msgs, bool gen) { + std::vector kept; + for (const auto & m : msgs) { + if (m.role != "tool") { + kept.push_back(m); + } + } + return base(kept, gen); + }; + rt::TurnText t; + std::string err; + check(rt::derive_turn_text(no_tool, "x", "f", t, err), "derivation still works"); + check(!t.has_tool_turn, "no tool turn is reported"); +} + +void test_modalities() { + rt::OutMode m = rt::OutMode::Text; + std::string err; + check(rt::parse_output_modalities(nullptr, m, err) && m == rt::OutMode::Auto, "null is auto"); + check(rt::parse_output_modalities(nlohmann::json::array({"text", "audio"}), m, err) && m == rt::OutMode::Mixed, "text and audio is mixed"); + check(rt::parse_output_modalities(nlohmann::json::array({"audio"}), m, err) && m == rt::OutMode::Audio, "audio"); + check(rt::parse_output_modalities(nlohmann::json::array({"text"}), m, err) && m == rt::OutMode::Text, "text"); + check(!rt::parse_output_modalities(nlohmann::json::array(), m, err), "an empty array is rejected"); + check(!rt::parse_output_modalities(nlohmann::json::array({"video"}), m, err), "an unknown modality is rejected"); + check(!rt::parse_output_modalities("text", m, err), "a bare string is rejected"); + + check_eq(rt::default_out_mode(false) == rt::OutMode::Mixed, true, "no tools: interleaved"); + check_eq(rt::default_out_mode(true) == rt::OutMode::Auto, true, "tools: the model chooses"); + + int32_t n = 0; + check(rt::parse_max_output_tokens(200, n) && n == 200, "max_output_tokens integer"); + check(rt::parse_max_output_tokens("inf", n) && n == -1, "max_output_tokens inf"); + check(!rt::parse_max_output_tokens(0, n), "max_output_tokens 0 is rejected"); + check(!rt::parse_max_output_tokens("many", n), "max_output_tokens string is rejected"); +} + +void test_session_params() { + rt::SessionParams p; + std::string err; + check(rt::parse_session_params({{"output_modalities", "text, audio"}, {"max_new_tokens", "64"}, {"temperature", "0.5"}, + {"top_k", "40"}, {"seed", "7"}, {"unrelated", "x"}}, p, err), "params parse"); + check(p.out_mode && *p.out_mode == rt::OutMode::Mixed, "comma list of modalities"); + check_eq(p.max_output_tokens, 64, "max_new_tokens alias"); + check(p.temperature && *p.temperature == 0.5f, "temperature"); + check(p.top_k && *p.top_k == 40, "top_k"); + check(p.seed && *p.seed == 7u, "seed"); + check(!p.top_p, "top_p stays unset"); + + check(rt::parse_session_params({{"output_modalities", "[\"audio\"]"}}, p, err) && *p.out_mode == rt::OutMode::Audio, "JSON array of modalities"); + check(!rt::parse_session_params({{"output_modalities", "video"}}, p, err), "bad modality"); + check(!rt::parse_session_params({{"temperature", "hot"}}, p, err), "bad number"); + check(rt::parse_session_params({{"tool_choice", "required"}}, p, err) && p.tool_choice == rt::ToolChoice::Required, "tool_choice"); + check(rt::parse_session_params({}, p, err) && p.tool_choice == rt::ToolChoice::Auto, "tool_choice defaults to auto"); + check(!rt::parse_session_params({{"tool_choice", "x"}}, p, err), "bad tool_choice"); + check(!rt::parse_session_params({{"max_output_tokens", "-3"}}, p, err), "bad token cap"); +} + +void test_response_options() { + rt::ResponseOptions o{}; + std::string err; + check(rt::parse_response_options(nlohmann::json::parse(R"({"output_modalities":["text"],"max_output_tokens":200})"), rt::OutMode::Mixed, -1, rt::ToolChoice::Auto, o, err), "flat payload"); + check(o.out_mode == rt::OutMode::Text && o.max_output_tokens == 200, "flat payload values"); + check(rt::parse_response_options(nlohmann::json::parse(R"({"response":{"max_output_tokens":9}})"), rt::OutMode::Mixed, -1, rt::ToolChoice::Auto, o, err), "nested payload"); + check(o.out_mode == rt::OutMode::Mixed && o.max_output_tokens == 9, "nested payload keeps the session mode"); + check(rt::parse_response_options(nlohmann::json(), rt::OutMode::Auto, 5, rt::ToolChoice::Auto, o, err) && o.max_output_tokens == 5, "empty payload keeps the defaults"); + check(!rt::parse_response_options(nlohmann::json::parse(R"({"output_modalities":["x"]})"), rt::OutMode::Mixed, -1, rt::ToolChoice::Auto, o, err), "bad modality"); + check(!rt::parse_response_options(nlohmann::json::parse(R"({"max_output_tokens":0})"), rt::OutMode::Mixed, -1, rt::ToolChoice::Auto, o, err), "bad cap"); + + check(rt::parse_response_options(nlohmann::json::parse(R"({"response":{"tool_choice":"required"}})"), rt::OutMode::Mixed, -1, rt::ToolChoice::None, o, err) && + o.tool_choice == rt::ToolChoice::Required, "a response overrides the tool choice of the session"); + check(rt::parse_response_options(nlohmann::json::parse("{}"), rt::OutMode::Mixed, -1, rt::ToolChoice::None, o, err) && o.tool_choice == rt::ToolChoice::None, + "the tool choice of the session is the default"); + check(!rt::parse_response_options(nlohmann::json::parse(R"({"tool_choice":"sometimes"})"), rt::OutMode::Mixed, -1, rt::ToolChoice::Auto, o, err), "bad tool choice"); + check(!rt::parse_response_options(nlohmann::json::parse(R"({"tool_choice":{"type":"function"}})"), rt::OutMode::Mixed, -1, rt::ToolChoice::Auto, o, err), + "a named function is not a tool choice mode"); +} + +void test_state_machine() { + const auto t = derive("sys", ""); + rt::ConversationState s; + std::string err; + + check(!s.response_create_error().empty(), "response.create without an open turn is rejected"); + check_eq(s.user_turn_header(t), t.first_header, "the first turn opens with the first header"); + s.user_turn_opened(); + check(s.started(), "a turn fixes the session"); + check_eq(s.user_turn_header(t), std::string(), "an open turn needs no header"); + check(s.response_create_error().empty(), "response.create is allowed with an open turn"); + check_eq(s.response_prompt(t), t.user_footer + t.assistant_header, "prompt after a user turn closes it"); + + s.response_created(t); + check_eq(s.n_turns(), 1, "one response"); + check(!s.response_create_error().empty(), "a second response.create without new input is rejected"); + check_eq(s.user_turn_header(t), t.assistant_footer + t.user_header, "the next turn closes the response first"); + + // tool round + const std::string text = s.tool_turn_text(t, "12:00", err); + check_eq(text, t.assistant_footer + t.tool_header + "12:00" + t.tool_footer, "tool turn text"); + s.tool_turn_opened(); + check(s.turn() == rt::ConversationState::Turn::Tool, "tool turn is open"); + check_eq(s.tool_turn_text(t, "second", err), t.tool_header + "second" + t.tool_footer, "a second result continues the tool turn"); + check_eq(s.response_prompt(t), t.assistant_header_after_tool, "prompt after a tool turn is the assistant header only"); + check(s.response_create_error().empty(), "response.create is allowed after a tool result"); + s.response_created(t); + check_eq(s.n_turns(), 2, "two responses"); + + // audio after a tool turn re-opens a user turn with the user header only + rt::ConversationState s2; + s2.user_turn_opened(); + s2.response_created(t); + (void) s2.tool_turn_text(t, "x", err); + s2.tool_turn_opened(); + check_eq(s2.user_turn_header(t), t.user_header, "a user turn after a tool turn needs only the user header"); +} + +void test_state_tool_errors() { + const auto t = derive("sys", ""); + rt::ConversationState s; + std::string err; + check(s.tool_turn_text(t, "x", err).empty() && !err.empty(), "no function call to answer before a response"); + s.user_turn_opened(); + err.clear(); + check(s.tool_turn_text(t, "x", err).empty() && !err.empty(), "no tool result inside a user turn"); + rt::TurnText none = t; + none.has_tool_turn = false; + rt::ConversationState s2; + s2.user_turn_opened(); + s2.response_created(none); + err.clear(); + check(s2.tool_turn_text(none, "x", err).empty() && !err.empty(), "no tool result without a tool turn in the template"); +} + +void test_session_update_rules() { + rt::ConversationState s; + check(s.session_update_error(true, true, true).empty(), "everything can change before the first turn"); + s.user_turn_opened(); + check(!s.session_update_error(true, false, false).empty(), "instructions are fixed after the first turn"); + check(!s.session_update_error(false, true, false).empty(), "tools are fixed after the first turn"); + check(!s.session_update_error(false, false, true).empty(), "voice is fixed after the first turn"); + check(s.session_update_error(false, false, false).empty(), "response options can still change"); + s.reset(); + check(!s.started() && s.n_turns() == 0, "reset forgets the conversation"); + check(s.session_update_error(true, true, true).empty(), "reset unlocks the session"); +} + +void test_pcm() { + std::string bytes(6, '\0'); + int16_t vals[3] = {0, 16384, -32768}; + std::memcpy(&bytes[0], vals, 6); + auto f = rt::pcm_s16le_to_f32(bytes); + check_eq(f.size(), (size_t) 3, "three samples"); + check(f[1] == 0.5f && f[2] == -1.0f, "scaled to [-1, 1)"); + check_eq(rt::pcm_s16le_to_f32(bytes + "x").size(), (size_t) 3, "an odd byte is ignored"); + + const std::string back = rt::f32_to_pcm_s16le({0.0f, 2.0f, -2.0f}); + int16_t out[3]; + std::memcpy(out, back.data(), 6); + check(out[0] == 0 && out[1] == 32767 && out[2] == -32767, "clipping"); +} + +void test_utf8() { + check_eq(rt::utf8_complete_prefix("abc"), (size_t) 3, "ascii"); + check_eq(rt::utf8_complete_prefix("a\xE2\x82"), (size_t) 1, "euro sign cut after 2 bytes"); + check_eq(rt::utf8_complete_prefix("a\xE2\x82\xAC"), (size_t) 4, "euro sign whole"); + check_eq(rt::utf8_complete_prefix("\xF0\x9F\x98"), (size_t) 0, "emoji cut after 3 bytes"); + check_eq(rt::utf8_complete_prefix("\xC3"), (size_t) 0, "lone lead byte"); + check_eq(rt::utf8_complete_prefix(""), (size_t) 0, "empty"); +} + +// the full text a fake slot produces, as the results of the slot +struct Step { + std::string text; + std::vector audio; +}; + +std::vector run(rt::ResponseTranslator & tr, const std::vector & steps, bool cancelled = false, int n_decoded = 7) { + std::vector all; + for (const auto & s : steps) { + for (auto & e : tr.on_result(s.text, s.audio, false, false, n_decoded)) { + all.push_back(std::move(e)); + } + } + for (auto & e : tr.on_result("", {}, true, cancelled, n_decoded)) { + all.push_back(std::move(e)); + } + return all; +} + +std::vector names(const std::vector & ev) { + std::vector n; + for (const auto & e : ev) { + n.push_back(e.event); + } + return n; +} + +void test_translator_audio_and_text() { + rt::ResponseTranslator tr(rt::OutMode::Mixed, 24000); + const auto ev = run(tr, {{"Hi", {}}, {"", {0.5f, -0.5f}}, {" there", {0.25f}}}); + const std::vector want = {"response.audio_transcript.delta", "response.audio.delta", "response.audio_transcript.delta", + "response.audio.delta", "response.done"}; + check_eq(names(ev), want, "events follow the order of the results, response.done last"); + check_eq(ev[0].meta["delta"].get(), std::string("Hi"), "first transcript delta"); + check_eq(ev[1].sample_rate, 24000, "audio carries its rate"); + check_eq(ev[1].pcm.size(), (size_t) 4, "two s16 samples"); + const auto & done = ev.back().meta; + check_eq(done["status"].get(), std::string("completed"), "completed"); + check_eq(done["text"].get(), std::string("Hi there"), "full text"); + check_eq(done["usage"]["output_tokens"].get(), 7, "usage"); +} + +void test_translator_text_only() { + rt::ResponseTranslator tr(rt::OutMode::Text, 24000); + const auto ev = run(tr, {{"Yes", {}}}); + check_eq(ev[0].event, std::string("response.text.delta"), "text-only responses use text deltas"); +} + +// A fake parser for text of the form "name|args": it reports the content +// before a call and the arguments of a call as they grow. +rt::ResponseTranslator::Splitter fake_splitter(std::vector * seen = nullptr) { + struct State { + std::string buf; + size_t emitted_content = 0; + size_t emitted_args = 0; + bool named = false; + }; + auto st = std::make_shared(); + return [st, seen](const std::string & text, bool final) { + if (seen) { + seen->push_back(text + (final ? "|final" : "")); + } + st->buf += text; + rt::SplitResult r; + const size_t open = st->buf.find(""); + if (open == std::string::npos) { + // hold back a possible start of the tag + size_t keep = 0; + for (size_t n = std::min(2, st->buf.size()); n > 0; n--) { + if (std::string("").compare(0, n, st->buf, st->buf.size() - n, n) == 0) { + keep = n; + break; + } + } + r.content = st->buf.substr(st->emitted_content, st->buf.size() - keep - st->emitted_content); + st->emitted_content += r.content.size(); + return r; + } + r.content = st->buf.substr(st->emitted_content, open - st->emitted_content); + st->emitted_content = open; + const size_t bar = st->buf.find('|', open); + if (bar == std::string::npos) { + return r; + } + rt::CallDelta d; + d.index = 0; + if (!st->named) { + d.name = st->buf.substr(open + 3, bar - open - 3); + d.id = "call_x"; + st->named = true; + } + size_t end = st->buf.find("", bar); + end = end == std::string::npos ? st->buf.size() : end; + d.arguments = st->buf.substr(bar + 1 + st->emitted_args, end - bar - 1 - st->emitted_args); + st->emitted_args += d.arguments.size(); + r.calls.push_back(d); + return r; + }; +} + +void test_translator_cancel() { + rt::ResponseTranslator tr(rt::OutMode::Mixed, 24000, fake_splitter()); + const auto ev = run(tr, {{"Hel", {0.1f}}, {"f|{}", {}}}, /*cancelled=*/true, 3); + check_eq(ev.back().event, std::string("response.done"), "a cancelled response ends with response.done"); + check_eq(ev.back().meta["status"].get(), std::string("cancelled"), "status cancelled"); + check_eq(ev.back().meta["text"].get(), std::string("Hel"), "the decoded part is reported"); + for (const auto & e : ev) { + check(e.event != "response.function_call_arguments.done", "a cancelled response makes no tool call"); + } + check(tr.on_result("late", {0.1f}, false, false, 9).empty(), "nothing after response.done"); +} + +void test_translator_cancel_flushes_held_back_text() { + std::vector seen; + rt::ResponseTranslator tr(rt::OutMode::Mixed, 24000, fake_splitter(&seen)); + // "<" could start a tag: the parser holds it back until it knows + const auto ev = run(tr, {{"Hel", {}}, {"lo <", {}}}, /*cancelled=*/true, 3); + check_eq(seen.back(), std::string("|final"), "the parser is told the cancelled response ended"); + std::string transcript; + for (const auto & e : ev) { + if (e.event == "response.audio_transcript.delta") { + transcript += e.meta["delta"].get(); + } + } + check(transcript.rfind("Hello", 0) == 0, "the transcript has the decoded text"); +} + +void test_translator_tool_call() { + std::vector seen; + rt::ResponseTranslator tr(rt::OutMode::Auto, 24000, fake_splitter(&seen), [] { return std::string("call_gen"); }); + const auto ev = run(tr, {{"One moment.", {0.1f}}, {"<", {}}, {"c>get_time|{\"tz\"", {0.2f}}, {":\"UTC\"}", {0.3f}}, {"", {0.4f}}}); + // the transcript never shows the call; audio is not dropped because a call started + std::string transcript; + for (const auto & e : ev) { + if (e.event == "response.audio_transcript.delta") { + transcript += e.meta["delta"].get(); + } + } + check_eq(transcript, std::string("One moment."), "the call is not in the transcript"); + size_t audio_frames = 0; + for (const auto & e : ev) { + if (e.event == "response.audio.delta") { + audio_frames++; + } + } + check_eq(audio_frames, (size_t) 4, "every audio frame is sent, also the ones after the start of a call"); + check_eq(seen.back(), std::string("|final"), "the parser is told when the response ends"); + + const auto n = names(ev); + const auto done_pos = std::find(n.begin(), n.end(), "response.done") - n.begin(); + const auto call_pos = std::find(n.begin(), n.end(), "response.function_call_arguments.done") - n.begin(); + check(call_pos < done_pos, "tool calls come before response.done"); + check_eq((size_t) std::count(n.begin(), n.end(), "response.function_call_arguments.done"), (size_t) 1, "one event per call"); + const auto & a = ev[call_pos].meta; + check_eq(a["name"].get(), std::string("get_time"), "call name"); + check_eq(a["arguments"].get(), std::string(R"({"tz":"UTC"})"), "the arguments of the deltas are joined into a JSON string"); + check_eq(a["call_id"].get(), std::string("call_x"), "a call id from the parser is kept"); + const auto & d = ev.back().meta; + check_eq(d["text"].get(), std::string("One moment."), "response.done text has no tool call"); + check_eq(d["raw_text"].get(), std::string("One moment.get_time|{\"tz\":\"UTC\"}"), "response.done carries what the model wrote"); +} + +void test_translator_call_ids_and_empty_arguments() { + rt::ResponseTranslator tr(rt::OutMode::Auto, 24000, [](const std::string &, bool final) { + rt::SplitResult r; + if (final) { + r.calls.push_back({0, "", "get_time", ""}); + r.calls.push_back({1, "call_fixed", "get_date", "{}"}); + } + return r; + }, [] { return std::string("call_gen"); }); + const auto ev = run(tr, {{"x", {}}}); + std::vector calls; + for (const auto & e : ev) { + if (e.event == "response.function_call_arguments.done") { + calls.push_back(e.meta); + } + } + check_eq(calls.size(), (size_t) 2, "two calls"); + check_eq(calls[0]["call_id"].get(), std::string("call_gen"), "a missing call id is generated"); + check_eq(calls[0]["arguments"].get(), std::string("{}"), "empty arguments become an empty object"); + check_eq(calls[1]["call_id"].get(), std::string("call_fixed"), "a call id from the parser is kept"); +} + +void test_translator_without_splitter() { + rt::ResponseTranslator tr(rt::OutMode::Mixed, 24000); + const auto ev = run(tr, {{"a b|c", {0.1f}}}); + check_eq(ev.back().meta["text"].get(), std::string("a b|c"), "without tools nothing is a call"); + check(!ev.back().meta.contains("raw_text"), "without tools the raw text is not repeated"); + check_eq(tr.n_calls(), 0, "no calls"); +} + +void test_translator_final_without_text() { + // the last result of a response has no text, the parser still has to see the end + std::vector seen; + rt::ResponseTranslator tr(rt::OutMode::Mixed, 24000, fake_splitter(&seen)); + (void) run(tr, {{"hi", {}}}); + check_eq(seen.size(), (size_t) 2, "one piece and the end"); + check_eq(seen.back(), std::string("|final"), "the end is marked"); +} + +void test_translator_parts_split_in_a_character() { + // the parser gets whole characters only + std::vector seen; + rt::ResponseTranslator tr(rt::OutMode::Mixed, 24000, fake_splitter(&seen)); + (void) run(tr, {{"5 \xE2\x82", {}}, {"\xAC", {}}}); + for (const auto & s : seen) { + std::string text = s.substr(0, s.find("|final")); + check_eq(rt::utf8_complete_prefix(text), text.size(), "the parser gets whole characters"); + } +} + +void test_translator_utf8_split() { + rt::ResponseTranslator tr(rt::OutMode::Mixed, 24000); + const auto ev = run(tr, {{"5 \xE2\x82", {}}, {"\xAC", {}}}); + std::string joined; + for (const auto & e : ev) { + if (e.event == "response.audio_transcript.delta") { + joined += e.meta["delta"].get(); + // every delta must be valid UTF-8 on its own + check_eq(rt::utf8_complete_prefix(e.meta["delta"].get()), e.meta["delta"].get().size(), "delta ends on a boundary"); + } + } + check_eq(joined, std::string("5 \xE2\x82\xAC"), "the split character is rejoined"); + // the JSON of every event can be dumped + for (const auto & e : ev) { + (void) rt::dump_json(e.meta); + } +} + +void test_translator_utf8_dangling() { + rt::ResponseTranslator tr(rt::OutMode::Mixed, 24000); + const auto ev = run(tr, {{"x\xE2", {}}}); + check_eq(ev.back().event, std::string("response.done"), "ends"); + // must not throw + (void) rt::dump_json(ev.back().meta); + (void) rt::dump_json(ev[ev.size() - 2].meta); +} + +void test_meta() { + const auto m = rt::session_meta(16000, 24000, rt::default_voices(), ""); + check_eq(m["input_sample_rate"].get(), 16000, "input rate"); + check_eq(m["output_sample_rate"].get(), 24000, "output rate"); + check_eq(m["voices"], nlohmann::json::array({"female", "male"}), "voices"); + check(!m.contains("voice"), "no voice requested"); + const auto v = rt::session_meta(16000, 24000, rt::default_voices(), "male"); + check(v["voice"] == "male" && v["voice_applied"] == false, "a requested voice is reported as not applied"); + const auto e = rt::error_meta("conversation_empty", "msg"); + check(e["type"] == "conversation_empty" && e["message"] == "msg", "error meta"); + check(!e.contains("response_scoped"), "an error is not response scoped by default"); + check(rt::error_meta("conversation_empty", "msg", true)["response_scoped"] == true, "a response scoped error says so"); + check(rt::make_call_id().rfind("call_", 0) == 0, "call id prefix"); + check(rt::make_call_id() != rt::make_call_id(), "call ids differ"); +} + +} // namespace + +void test_default_instructions_follow_the_tools() { + check(std::string(rt::default_instructions(false)) == "Respond with interleaved text and audio.", "default without tools"); + // the model card prompt for tool calls: under the plain one the model answers in words and never calls + check(std::string(rt::default_instructions(true)).rfind("Respond with function calls", 0) == 0, "default with tools asks for function calls"); + check(rt::is_default_instructions(rt::default_instructions(false)) && rt::is_default_instructions(rt::default_instructions(true)), "both defaults are known"); + check(!rt::is_default_instructions("Be brief."), "a client prompt is not a default"); + + check_eq(rt::resolve_instructions("", true), std::string(rt::default_instructions(true)), "none + tools"); + check_eq(rt::resolve_instructions("", false), std::string(rt::default_instructions(false)), "none, no tools"); + // the server sends the plain default back as the default of the session + check_eq(rt::resolve_instructions(rt::default_instructions(false), true), std::string(rt::default_instructions(true)), "plain default + tools"); + check_eq(rt::resolve_instructions("Be brief.", true), std::string("Be brief."), "a client prompt wins"); +} + +int main() { + test_default_instructions_follow_the_tools(); + test_turn_text(); + test_tool_turn_needs_a_tool(); + test_turn_text_with_tools(); + test_turn_text_without_instructions(); + test_turn_text_rejects_bad_template(); + test_no_tool_turn(); + test_modalities(); + test_session_params(); + test_response_options(); + test_state_machine(); + test_state_tool_errors(); + test_session_update_rules(); + test_pcm(); + test_utf8(); + test_translator_audio_and_text(); + test_translator_text_only(); + test_translator_cancel(); + test_translator_cancel_flushes_held_back_text(); + test_translator_tool_call(); + test_translator_call_ids_and_empty_arguments(); + test_translator_without_splitter(); + test_translator_final_without_text(); + test_translator_parts_split_in_a_character(); + test_translator_utf8_split(); + test_translator_utf8_dangling(); + test_meta(); + if (failures != 0) { + std::fprintf(stderr, "%d check(s) failed\n", failures); + return 1; + } + std::puts("realtime_session_test: all checks passed"); + return 0; +} diff --git a/backend/cpp/llama-cpp/transcription_prompt.h b/backend/cpp/llama-cpp/transcription_prompt.h new file mode 100644 index 000000000..0031de645 --- /dev/null +++ b/backend/cpp/llama-cpp/transcription_prompt.h @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: MIT + +#pragma once + +#include + +namespace llama_grpc { + +// The chat messages of one transcription request, before the audio marker. +// An empty system string means "no system turn". +struct transcription_messages { + std::string system; + std::string user; +}; + +// Builds the text of an ASR request from the model's ASR preset +// (common_chat_get_asr_prompt). Mirrors convert_transcriptions_to_chatcmpl +// upstream, with one deliberate difference: a caller prompt is appended to the +// preset's user text, not substituted for it. That is LocalAI's historical +// behaviour, so Qwen3-ASR requests keep their exact prompt. +// +// preset_has_language is true when the preset already names the language (LFM3 +// writes it into the system turn), in which case no "(language: ...)" suffix is +// added to the user text. +inline transcription_messages build_transcription_messages( + const std::string & preset_system, + const std::string & preset_user, + bool preset_has_language, + const std::string & language, + const std::string & prompt) { + transcription_messages out; + out.system = preset_system; + out.user = preset_user; + + if (!language.empty() && !preset_has_language) { + out.user += " (language: " + language + ")"; + } + if (!prompt.empty()) { + // Only join with a newline when there is preset text to join to, so a + // model with an empty user turn (LFM3) gets the caller prompt as is. + if (!out.user.empty()) { + out.user += "\n"; + } + out.user += prompt; + } + return out; +} + +} // namespace llama_grpc diff --git a/backend/cpp/llama-cpp/transcription_prompt_test.cpp b/backend/cpp/llama-cpp/transcription_prompt_test.cpp new file mode 100644 index 000000000..466274bc7 --- /dev/null +++ b/backend/cpp/llama-cpp/transcription_prompt_test.cpp @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: MIT + +#include +#include + +#include "transcription_prompt.h" + +static int failures = 0; + +static void check(bool ok, const char * name) { + if (!ok) { + ++failures; + std::fprintf(stderr, "FAIL: %s\n", name); + } +} + +// Qwen3-ASR keeps the default preset: no system turn, the fixed user text. +static const char * kQwenUser = "Transcribe audio to text"; + +static void test_qwen_preset_without_options_is_unchanged() { + const auto m = llama_grpc::build_transcription_messages( + "", kQwenUser, false, "", ""); + + check(m.system.empty(), "qwen preset adds no system turn"); + check(m.user == kQwenUser, "qwen preset keeps the fixed user text"); +} + +static void test_qwen_preset_appends_language_hint() { + const auto m = llama_grpc::build_transcription_messages( + "", kQwenUser, false, "fr", ""); + + check(m.user == "Transcribe audio to text (language: fr)", + "qwen preset appends the language hint"); +} + +static void test_qwen_preset_appends_caller_prompt_after_newline() { + // LocalAI has always appended the caller prompt to the preset text, so + // Qwen3-ASR requests keep exactly the prompt they had before. + const auto m = llama_grpc::build_transcription_messages( + "", kQwenUser, false, "fr", "Names: Kennedy"); + + check(m.user == "Transcribe audio to text (language: fr)\nNames: Kennedy", + "qwen preset appends the caller prompt after a newline"); +} + +static void test_lfm3_preset_puts_language_in_system_turn() { + // LFM3 names the language in the system turn and sends an empty user turn, + // so no "(language: ...)" suffix may appear in the user text. + const auto m = llama_grpc::build_transcription_messages( + "Perform ASR in French.", "", true, "fr", ""); + + check(m.system == "Perform ASR in French.", "lfm3 preset keeps its system turn"); + check(m.user.empty(), "lfm3 user turn is empty (media only)"); +} + +static void test_lfm3_preset_with_caller_prompt_has_no_leading_newline() { + const auto m = llama_grpc::build_transcription_messages( + "Perform ASR.", "", true, "", "Speaker is a nurse"); + + check(m.user == "Speaker is a nurse", "empty preset user takes the caller prompt as is"); +} + +static void test_lfm3_preset_without_language_or_prompt() { + const auto m = llama_grpc::build_transcription_messages( + "Perform ASR.", "", true, "", ""); + + check(m.system == "Perform ASR.", "lfm3 auto-detect keeps the generic system turn"); + check(m.user.empty(), "lfm3 auto-detect has an empty user turn"); +} + +int main() { + test_qwen_preset_without_options_is_unchanged(); + test_qwen_preset_appends_language_hint(); + test_qwen_preset_appends_caller_prompt_after_newline(); + test_lfm3_preset_puts_language_in_system_turn(); + test_lfm3_preset_with_caller_prompt_has_no_leading_newline(); + test_lfm3_preset_without_language_or_prompt(); + + if (failures == 0) { + std::printf("transcription_prompt_test: all checks passed\n"); + } + return failures; +} diff --git a/backend/cpp/llama-cpp/tts_request_options.h b/backend/cpp/llama-cpp/tts_request_options.h index 0fdc092f2..177705bf1 100644 --- a/backend/cpp/llama-cpp/tts_request_options.h +++ b/backend/cpp/llama-cpp/tts_request_options.h @@ -18,7 +18,11 @@ struct tts_request_options { std::string error; std::string text; + // Speaker reference file, set only when the model needs one (Qwen3-TTS Base). std::string voice_path; + // Named voice, set only when the model takes a name (LFM3-Audio "tts" + // template: "female", "male"). Empty means the template default. + std::string voice; std::string language; // 0 / 0.0f mean "unset": upstream only overrides the sampler defaults when @@ -83,26 +87,35 @@ inline tts_request_options reject(const std::string & message) { } // namespace detail +// needs_reference is the model's mtmd_helper_cap.audio_gen_spk_ref. When true +// (Qwen3-TTS Base), voice is a speaker reference file and is required. When +// false (LFM3-Audio), voice is a named voice for the "tts" chat template, and +// an empty voice lets the template pick its default. inline tts_request_options parse_tts_request_options( const std::string & text, const std::string & voice, const std::string & language, - const std::map & params) { + const std::map & params, + bool needs_reference) { if (text.empty()) { return detail::reject("text must be a non-empty string"); } - // The Qwen3-TTS Base checkpoints have no built-in speaker. Without a - // reference clip the model picks an arbitrary voice, so an unset voice is - // a request error rather than a defaulted one. - if (voice.empty()) { - return detail::reject("voice must name a speaker reference audio file"); - } - tts_request_options opts; - opts.text = text; - opts.voice_path = voice; - opts.language = language; + opts.text = text; + opts.language = language; + + if (needs_reference) { + // The Qwen3-TTS Base checkpoints have no built-in speaker. Without a + // reference clip the model picks an arbitrary voice, so an unset voice + // is a request error rather than a defaulted one. + if (voice.empty()) { + return detail::reject("voice must name a speaker reference audio file"); + } + opts.voice_path = voice; + } else { + opts.voice = voice; + } // Both values are range-checked here rather than left to the caller: the // consumer copies them straight into mtmd_helper::gen_audio::inp, and only diff --git a/backend/cpp/llama-cpp/tts_request_options_test.cpp b/backend/cpp/llama-cpp/tts_request_options_test.cpp index d4804d916..abd786f44 100644 --- a/backend/cpp/llama-cpp/tts_request_options_test.cpp +++ b/backend/cpp/llama-cpp/tts_request_options_test.cpp @@ -17,12 +17,13 @@ static void check(bool ok, const char * name) { static void test_accepts_a_minimal_valid_request() { const auto opts = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "en", {}); + "Hello world", "/models/voices/ref.wav", "en", {}, true); check(opts.ok, "minimal request is accepted"); check(opts.error.empty(), "minimal request has no error"); check(opts.text == "Hello world", "text passes through"); check(opts.voice_path == "/models/voices/ref.wav", "voice path passes through"); + check(opts.voice.empty(), "reference mode leaves the named voice empty"); check(opts.language == "en", "language passes through"); check(opts.top_k == 0, "top_k defaults to the unset sentinel"); check(opts.top_p == 0.0f, "top_p defaults to the unset sentinel"); @@ -31,29 +32,56 @@ static void test_accepts_a_minimal_valid_request() { static void test_rejects_empty_text() { const auto opts = llama_grpc::parse_tts_request_options( - "", "/models/voices/ref.wav", "en", {}); + "", "/models/voices/ref.wav", "en", {}, true); check(!opts.ok, "empty text is rejected"); check(opts.error.find("text") != std::string::npos, "empty-text error names the field"); + + const auto no_ref = llama_grpc::parse_tts_request_options( + "", "Female", "en", {}, false); + check(!no_ref.ok, "empty text is rejected without a speaker reference too"); } static void test_rejects_missing_speaker_reference() { // Qwen3-TTS Base has no built-in speaker; without a reference it produces // an arbitrary voice, so this must be a hard error rather than a surprise. const auto opts = llama_grpc::parse_tts_request_options( - "Hello world", "", "en", {}); + "Hello world", "", "en", {}, true); check(!opts.ok, "missing voice is rejected"); check(opts.error.find("voice") != std::string::npos, "missing-voice error names the field"); } +static void test_accepts_named_voice_without_reference() { + // LFM3-Audio has no speaker reference. Its "tts" chat template takes a + // voice name ("female" or "male"), so the value is a name, not a path. + const auto opts = llama_grpc::parse_tts_request_options( + "Hello world", "male", "en", {}, false); + + check(opts.ok, "named voice is accepted when no reference is needed"); + check(opts.error.empty(), "named voice has no error"); + check(opts.voice == "male", "named voice passes through"); + check(opts.voice_path.empty(), "named voice is not treated as a file path"); + check(opts.language == "en", "language passes through without a reference"); +} + +static void test_accepts_empty_voice_without_reference() { + // An empty voice lets the template pick its own default. + const auto opts = llama_grpc::parse_tts_request_options( + "Hello world", "", "", {}, false); + + check(opts.ok, "empty voice is accepted when no reference is needed"); + check(opts.voice.empty(), "empty voice stays empty for the template default"); + check(opts.voice_path.empty(), "empty voice without reference has no path"); +} + static void test_parses_sampling_params() { const std::map params{ {"top_k", "40"}, {"top_p", "0.85"}, }; const auto opts = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", params); + "Hello world", "/models/voices/ref.wav", "", params, true); check(opts.ok, "sampling params are accepted"); check(opts.top_k == 40, "top_k is parsed"); @@ -63,26 +91,26 @@ static void test_parses_sampling_params() { static void test_rejects_malformed_sampling_params() { const auto bad_top_k = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_k", "forty"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_k", "forty"}}, true); check(!bad_top_k.ok, "non-numeric top_k is rejected"); check(bad_top_k.error.find("top_k") != std::string::npos, "top_k error names the field"); const auto bad_top_p = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_p", ""}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_p", ""}}, true); check(!bad_top_p.ok, "empty top_p is rejected"); const auto trailing = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_k", "40abc"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_k", "40abc"}}, true); check(!trailing.ok, "top_k with trailing garbage is rejected"); const auto trailing_float = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_p", "0.8abc"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_p", "0.8abc"}}, true); check(!trailing_float.ok, "top_p with trailing garbage is rejected"); // std::stol returns a long, which is wider than int32_t on 64-bit hosts, so // an in-range-for-long value still has to be caught before the narrowing. const auto overflow_top_k = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_k", "99999999999"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_k", "99999999999"}}, true); check(!overflow_top_k.ok, "top_k beyond int32 range is rejected"); check(overflow_top_k.error.find("top_k") != std::string::npos, "top_k overflow error names the field"); @@ -92,45 +120,61 @@ static void test_rejects_out_of_range_sampling_params() { // These reach mtmd_helper::gen_audio::inp unconditionally downstream, where // the "> 0" sampler guard does not screen them, so they must die here. const auto negative_top_k = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_k", "-5"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_k", "-5"}}, true); check(!negative_top_k.ok, "negative top_k is rejected"); check(negative_top_k.error.find("top_k") != std::string::npos, "negative top_k error names the field"); const auto negative_top_p = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_p", "-0.1"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_p", "-0.1"}}, true); check(!negative_top_p.ok, "negative top_p is rejected"); check(negative_top_p.error.find("top_p") != std::string::npos, "negative top_p error names the field"); const auto large_top_p = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_p", "1.5"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_p", "1.5"}}, true); check(!large_top_p.ok, "top_p above 1.0 is rejected"); // NaN survives a naive "p < 0.0f || p > 1.0f" range test because every // comparison against NaN is false. This case pins the correct form. const auto nan_top_p = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_p", "nan"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_p", "nan"}}, true); check(!nan_top_p.ok, "NaN top_p is rejected"); const auto inf_top_p = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_p", "inf"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_p", "inf"}}, true); check(!inf_top_p.ok, "infinite top_p is rejected"); } +static void test_sampling_validation_applies_without_reference() { + // The range checks do not depend on the speaker-reference mode. + const auto negative_top_k = llama_grpc::parse_tts_request_options( + "Hello world", "female", "", {{"top_k", "-5"}}, false); + check(!negative_top_k.ok, "negative top_k is rejected without a reference"); + + const auto nan_top_p = llama_grpc::parse_tts_request_options( + "Hello world", "female", "", {{"top_p", "nan"}}, false); + check(!nan_top_p.ok, "NaN top_p is rejected without a reference"); + + const auto max_frames = llama_grpc::parse_tts_request_options( + "Hello world", "female", "", {{"max_frames", "120"}}, false); + check(max_frames.ok && max_frames.max_frames == 120, + "max_frames is parsed without a reference"); +} + static void test_accepts_sampling_param_boundaries() { const auto zero_top_p = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_p", "0.0"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_p", "0.0"}}, true); check(zero_top_p.ok, "top_p of 0.0 is accepted"); check(zero_top_p.top_p == 0.0f, "top_p of 0.0 round-trips"); const auto one_top_p = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_p", "1.0"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_p", "1.0"}}, true); check(one_top_p.ok, "top_p of 1.0 is accepted"); check(one_top_p.top_p == 1.0f, "top_p of 1.0 round-trips"); const auto zero_top_k = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_k", "0"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_k", "0"}}, true); check(zero_top_k.ok, "top_k of 0 is accepted"); } @@ -139,45 +183,45 @@ static void test_parses_max_frames() { // 512-frame default in place when it is unset, so the sentinel matters as // much as the parsed value. const auto opts = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "120"}}); + "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "120"}}, true); check(opts.ok, "max_frames is accepted"); check(opts.max_frames == 120, "max_frames is parsed"); const auto absent = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"top_k", "40"}}); + "Hello world", "/models/voices/ref.wav", "", {{"top_k", "40"}}, true); check(absent.ok, "a request without max_frames is accepted"); check(absent.max_frames == 0, "absent max_frames leaves the unset sentinel"); const auto zero = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "0"}}); + "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "0"}}, true); check(zero.ok, "max_frames of 0 is accepted"); check(zero.max_frames == 0, "max_frames of 0 means unset"); } static void test_rejects_malformed_max_frames() { const auto negative = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "-1"}}); + "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "-1"}}, true); check(!negative.ok, "negative max_frames is rejected"); check(negative.error.find("max_frames") != std::string::npos, "negative max_frames error names the field"); const auto non_numeric = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "many"}}); + "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "many"}}, true); check(!non_numeric.ok, "non-numeric max_frames is rejected"); check(non_numeric.error.find("max_frames") != std::string::npos, "non-numeric max_frames error names the field"); const auto trailing = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "120abc"}}); + "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "120abc"}}, true); check(!trailing.ok, "max_frames with trailing garbage is rejected"); const auto empty = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"max_frames", ""}}); + "Hello world", "/models/voices/ref.wav", "", {{"max_frames", ""}}, true); check(!empty.ok, "empty max_frames is rejected"); const auto overflow = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "99999999999"}}); + "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "99999999999"}}, true); check(!overflow.ok, "max_frames beyond int32 range is rejected"); } @@ -185,7 +229,7 @@ static void test_ignores_unknown_params() { // Unknown keys are backend-specific knobs meant for other TTS engines. A // request routed here must not fail just because it carries them. const auto opts = llama_grpc::parse_tts_request_options( - "Hello world", "/models/voices/ref.wav", "", {{"exaggeration", "0.7"}}); + "Hello world", "/models/voices/ref.wav", "", {{"exaggeration", "0.7"}}, true); check(opts.ok, "unknown params are ignored, not rejected"); } @@ -194,9 +238,12 @@ int main() { test_accepts_a_minimal_valid_request(); test_rejects_empty_text(); test_rejects_missing_speaker_reference(); + test_accepts_named_voice_without_reference(); + test_accepts_empty_voice_without_reference(); test_parses_sampling_params(); test_rejects_malformed_sampling_params(); test_rejects_out_of_range_sampling_params(); + test_sampling_validation_applies_without_reference(); test_accepts_sampling_param_boundaries(); test_parses_max_frames(); test_rejects_malformed_max_frames(); diff --git a/backend/index.yaml b/backend/index.yaml index 8291f8f0b..139129ec9 100644 --- a/backend/index.yaml +++ b/backend/index.yaml @@ -2715,35 +2715,6 @@ nvidia-l4t: nvidia-l4t-vibevoice nvidia-l4t-cuda-12: nvidia-l4t-vibevoice nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-vibevoice -- name: liquid-audio - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - name: qwen-tts alias: qwen-tts license: apache-2.0 @@ -24240,483 +24211,6 @@ uri: quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-vibevoice mirrors: - localai/localai-backends:master-metal-darwin-arm64-vibevoice -- name: liquid-audio-development - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio-development - default: cpu-liquid-audio-development - intel: intel-liquid-audio-development - metal: metal-liquid-audio-development - nvidia: cuda12-liquid-audio-development - nvidia-cuda-12: cuda12-liquid-audio-development - nvidia-cuda-13: cuda13-liquid-audio-development - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio-development -- name: cpu-liquid-audio - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:latest-cpu-liquid-audio - mirrors: - - localai/localai-backends:latest-cpu-liquid-audio -- name: cpu-liquid-audio-development - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:master-cpu-liquid-audio - mirrors: - - localai/localai-backends:master-cpu-liquid-audio -- name: metal-liquid-audio - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-liquid-audio - mirrors: - - localai/localai-backends:latest-metal-darwin-arm64-liquid-audio -- name: metal-liquid-audio-development - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-liquid-audio - mirrors: - - localai/localai-backends:master-metal-darwin-arm64-liquid-audio -- name: cuda12-liquid-audio - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-liquid-audio - mirrors: - - localai/localai-backends:latest-gpu-nvidia-cuda-12-liquid-audio -- name: cuda12-liquid-audio-development - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-liquid-audio - mirrors: - - localai/localai-backends:master-gpu-nvidia-cuda-12-liquid-audio -- name: cuda13-liquid-audio - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-liquid-audio - mirrors: - - localai/localai-backends:latest-gpu-nvidia-cuda-13-liquid-audio -- name: cuda13-liquid-audio-development - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-liquid-audio - mirrors: - - localai/localai-backends:master-gpu-nvidia-cuda-13-liquid-audio -- name: intel-liquid-audio - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:latest-gpu-intel-liquid-audio - mirrors: - - localai/localai-backends:latest-gpu-intel-liquid-audio -- name: intel-liquid-audio-development - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:master-gpu-intel-liquid-audio - mirrors: - - localai/localai-backends:master-gpu-intel-liquid-audio -- name: rocm-liquid-audio - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-liquid-audio - mirrors: - - localai/localai-backends:latest-gpu-rocm-hipblas-liquid-audio -- name: rocm-liquid-audio-development - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-liquid-audio - mirrors: - - localai/localai-backends:master-gpu-rocm-hipblas-liquid-audio -- name: cuda13-nvidia-l4t-arm64-liquid-audio - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-liquid-audio - mirrors: - - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-liquid-audio -- name: cuda13-nvidia-l4t-arm64-liquid-audio-development - alias: liquid-audio - license: LFM-Open-License-v1.0 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - uri: quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-liquid-audio - mirrors: - - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-liquid-audio - name: qwen-tts-development alias: qwen-tts license: apache-2.0 diff --git a/backend/python/common/model_utils_test.py b/backend/python/common/model_utils_test.py index 844eb972d..82ea9f1f7 100644 --- a/backend/python/common/model_utils_test.py +++ b/backend/python/common/model_utils_test.py @@ -52,7 +52,6 @@ class MigratedBackendSourceTest(unittest.TestCase): "nemo", "voxcpm", "qwen-tts", - "liquid-audio", "vllm", "vllm-omni", "sglang", diff --git a/backend/python/liquid-audio/Makefile b/backend/python/liquid-audio/Makefile deleted file mode 100644 index 59b71c8c9..000000000 --- a/backend/python/liquid-audio/Makefile +++ /dev/null @@ -1,23 +0,0 @@ -.PHONY: liquid-audio -liquid-audio: - bash install.sh - -.PHONY: run -run: liquid-audio - @echo "Running liquid-audio..." - bash run.sh - @echo "liquid-audio run." - -.PHONY: test -test: liquid-audio - @echo "Testing liquid-audio..." - bash test.sh - @echo "liquid-audio tested." - -.PHONY: protogen-clean -protogen-clean: - $(RM) backend_pb2_grpc.py backend_pb2.py - -.PHONY: clean -clean: protogen-clean - rm -rf venv __pycache__ diff --git a/backend/python/liquid-audio/backend.py b/backend/python/liquid-audio/backend.py deleted file mode 100644 index bcc9165c1..000000000 --- a/backend/python/liquid-audio/backend.py +++ /dev/null @@ -1,867 +0,0 @@ -#!/usr/bin/env python3 -""" -Liquid Audio backend for LocalAI. - -Wraps LiquidAI's `liquid-audio` Python package (https://github.com/Liquid4All/liquid-audio). -The same model serves four roles, selected by the `mode` option at load time: -chat, asr, tts, s2s. Fine-tuning is exposed via StartFineTune. -""" -from concurrent import futures -import argparse -import json -import os -import queue -import signal -import sys -import threading -import time -import traceback -import uuid - -import grpc - -sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common')) -sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common')) -from grpc_auth import get_auth_interceptors # noqa: E402 -from model_utils import resolve_model_reference # noqa: E402 -from python_utils import parse_options # noqa: E402 - -import backend_pb2 # noqa: E402 -import backend_pb2_grpc # noqa: E402 - -_ONE_DAY_IN_SECONDS = 60 * 60 * 24 -MAX_WORKERS = int(os.environ.get('PYTHON_GRPC_MAX_WORKERS', '1')) - -# Voice id → system-prompt suffix. The model only ships these four voices. -VOICE_PROMPTS = { - "us_male": "Perform TTS. Use the US male voice.", - "us_female": "Perform TTS. Use the US female voice.", - "uk_male": "Perform TTS. Use the UK male voice.", - "uk_female": "Perform TTS. Use the UK female voice.", -} -DEFAULT_VOICE = "us_female" - -# Special-token IDs that LFM2-Audio emits to delimit modality boundaries. -# Sourced from liquid_audio/model/lfm2_audio.py (see generate_sequential/_sample_*). -TEXT_END_TOKEN = 130 # <|text_end|> -AUDIO_START_TOKEN = 128 # <|audio_start|> -IM_END_TOKEN = 7 # <|im_end|> -AUDIO_EOS_CODE = 2048 # signals end-of-audio in any codebook position - -_PATCHED_LOCAL_PATHS = False - - -def _patch_liquid_audio_local_paths(): - """Make liquid_audio.utils.get_model_dir() tolerate local directories. - - Upstream always passes its argument to huggingface_hub.snapshot_download, - which only accepts `owner/repo` ids. LocalAI's gallery hands us absolute - paths under //, so we intercept snapshot_download - in the liquid_audio.utils namespace and return the directory as-is when - it already exists on disk. Idempotent. - """ - global _PATCHED_LOCAL_PATHS - if _PATCHED_LOCAL_PATHS: - return - import liquid_audio.utils as _la_utils - _orig_snapshot_download = _la_utils.snapshot_download - - def _local_first_snapshot_download(repo_id, revision=None, **kwargs): - if isinstance(repo_id, (str, os.PathLike)) and os.path.isdir(str(repo_id)): - return str(repo_id) - return _orig_snapshot_download(repo_id, revision=revision, **kwargs) - - _la_utils.snapshot_download = _local_first_snapshot_download - _PATCHED_LOCAL_PATHS = True - - -def _select_device(): - import torch - if torch.cuda.is_available(): - return "cuda" - if hasattr(torch.backends, "mps") and torch.backends.mps.is_available(): - return "mps" - return "cpu" - - -class ActiveJob: - """Tracks an in-flight fine-tune so FineTuneProgress can stream from its queue.""" - - def __init__(self, job_id): - self.job_id = job_id - self.progress_queue = queue.Queue() - self.thread = None - self.stopped = False - self.completed = False - self.error = None - - -class BackendServicer(backend_pb2_grpc.BackendServicer): - def __init__(self): - self.processor = None - self.model = None - self.device = "cpu" - self.dtype = None - self.options = {} - self.model_id = None - self.active_job = None - - @property - def mode(self): - return str(self.options.get("mode", "chat")).lower() - - @property - def voice(self): - v = str(self.options.get("voice", DEFAULT_VOICE)).lower() - return v if v in VOICE_PROMPTS else DEFAULT_VOICE - - - def Free(self, request, context): - # Called by LocalAI when unloading the model. Drop GPU tensors so the - # next load starts from a clean state instead of bumping into OOM. - try: - for attr in ("model", "processor", "tokenizer"): - if hasattr(self, attr): - try: - delattr(self, attr) - except Exception: - pass - import gc - gc.collect() - try: - import torch - if torch.cuda.is_available(): - torch.cuda.empty_cache() - except Exception: - pass - return backend_pb2.Result(success=True, message="OK") - except Exception as exc: - print(f"Free failed: {exc}", file=sys.stderr) - return backend_pb2.Result(success=False, message=str(exc)) - - - def Health(self, request, context): - return backend_pb2.Reply(message=bytes("OK", 'utf-8')) - - - def LoadModel(self, request, context): - try: - import torch - - self.options = parse_options(request.Options) - if self.options.get("voice") and self.options["voice"] not in VOICE_PROMPTS: - print(f"Warning: unknown voice '{self.options['voice']}'; defaulting to '{DEFAULT_VOICE}'", - file=sys.stderr) - - requested_device = self.options.get("device") - self.device = requested_device or _select_device() - if self.device == "cuda" and not torch.cuda.is_available(): - return backend_pb2.Result(success=False, message="CUDA requested but not available") - if self.device == "mps" and not (hasattr(torch.backends, "mps") and - torch.backends.mps.is_available()): - print("MPS not available; falling back to CPU", file=sys.stderr) - self.device = "cpu" - - dtype_name = str(self.options.get("dtype", "bfloat16")).lower() - self.dtype = { - "bfloat16": torch.bfloat16, - "bf16": torch.bfloat16, - "float16": torch.float16, - "fp16": torch.float16, - "half": torch.float16, - "float32": torch.float32, - "fp32": torch.float32, - }.get(dtype_name, torch.bfloat16) - - # `_local_only` distinguishes a staged snapshot from the legacy - # repository fallback. liquid_audio has no matching Hub keyword, - # so the compatibility shim below enforces local loading instead. - model_id, _local_only = resolve_model_reference( - request, "LiquidAI/LFM2.5-Audio-1.5B" - ) - if not model_id: - return backend_pb2.Result(success=False, message="No model identifier provided") - self.model_id = model_id - - # Pure fine-tune jobs don't need an in-memory inference model — the - # Trainer instantiates its own copy at StartFineTune time. - if self.mode == "finetune": - print(f"Loaded liquid-audio backend in fine-tune mode (model id: {model_id})", - file=sys.stderr) - return backend_pb2.Result(success=True, message="OK") - - from liquid_audio import LFM2AudioModel, LFM2AudioProcessor - - # liquid_audio's from_pretrained unconditionally routes through - # huggingface_hub.snapshot_download, which rejects local paths - # (HFValidationError on `/models/LiquidAI/LFM2.5-Audio-1.5B`). - # When LocalAI's gallery has already staged the weights on disk, - # short-circuit the download to return the local directory. - _patch_liquid_audio_local_paths() - - print(f"Loading liquid-audio model '{model_id}' on {self.device} ({self.dtype})", - file=sys.stderr) - self.processor = LFM2AudioProcessor.from_pretrained(model_id, device=self.device).eval() - self.model = LFM2AudioModel.from_pretrained( - model_id, device=self.device, dtype=self.dtype - ).eval() - - print(f"Liquid-audio mode={self.mode}, voice={self.voice}", file=sys.stderr) - return backend_pb2.Result(success=True, message="OK") - - except Exception as exc: - print(f"LoadModel failed: {exc}", file=sys.stderr) - print(traceback.format_exc(), file=sys.stderr) - return backend_pb2.Result(success=False, message=str(exc)) - - - def Predict(self, request, context): - try: - text = "".join(self._generate_text_stream(request)) - return backend_pb2.Reply(message=text.encode("utf-8")) - except Exception as exc: - print(f"Predict failed: {exc}", file=sys.stderr) - print(traceback.format_exc(), file=sys.stderr) - context.set_code(grpc.StatusCode.INTERNAL) - context.set_details(str(exc)) - return backend_pb2.Reply() - - def PredictStream(self, request, context): - try: - for delta in self._generate_text_stream(request): - yield backend_pb2.Reply(message=delta.encode("utf-8")) - except Exception as exc: - print(f"PredictStream failed: {exc}", file=sys.stderr) - print(traceback.format_exc(), file=sys.stderr) - context.set_code(grpc.StatusCode.INTERNAL) - context.set_details(str(exc)) - - - def VAD(self, request, context): - # Stub voice-activity detector: RMS-energy threshold over 30ms frames at - # 16 kHz. Good enough for the realtime endpoint's handleVAD loop, which - # only inspects segment presence + last segment end. The proper signal - # would come from the model's audio encoder, but that ride-along is a - # PR-D scope item — until then this keeps the legacy pipeline path - # working without forcing the operator to install a separate VAD model. - import numpy as np - try: - audio = np.asarray(request.audio, dtype=np.float32) - if audio.size == 0: - return backend_pb2.VADResponse(segments=[]) - - sample_rate = 16000 - frame_size = sample_rate * 30 // 1000 # 30ms → 480 samples - threshold = float(self.options.get("vad_rms_threshold", 0.01)) - min_speech_frames = int(self.options.get("vad_min_speech_frames", 2)) # ≥60ms - # handleVAD ticks every 300 ms and only inspects segment presence - # + last segment end relative to silence_threshold (~500 ms). Cap - # the analysed window to the tail of the buffer so we don't redo - # the entire growing utterance every tick. - window_s = float(self.options.get("vad_window_s", 5.0)) - window_samples = int(window_s * sample_rate) - time_offset_s = 0.0 - if audio.size > window_samples: - time_offset_s = (audio.size - window_samples) / sample_rate - audio = audio[-window_samples:] - - n_frames = audio.size // frame_size - if n_frames == 0: - return backend_pb2.VADResponse(segments=[]) - frames = audio[: n_frames * frame_size].reshape(n_frames, frame_size) - rms = np.sqrt(np.mean(frames ** 2, axis=1)) - speech = rms > threshold - - def _emit(start_idx, end_idx, out): - if end_idx - start_idx >= min_speech_frames: - out.append(backend_pb2.VADSegment( - start=time_offset_s + start_idx * frame_size / sample_rate, - end=time_offset_s + end_idx * frame_size / sample_rate, - )) - - segments = [] - start_idx = None - for i, is_speech in enumerate(speech): - if is_speech and start_idx is None: - start_idx = i - elif not is_speech and start_idx is not None: - _emit(start_idx, i, segments) - start_idx = None - if start_idx is not None: - _emit(start_idx, n_frames, segments) - return backend_pb2.VADResponse(segments=segments) - except Exception as exc: - print(f"VAD failed: {exc}", file=sys.stderr) - print(traceback.format_exc(), file=sys.stderr) - context.set_code(grpc.StatusCode.INTERNAL) - context.set_details(str(exc)) - return backend_pb2.VADResponse(segments=[]) - - - def TTS(self, request, context): - try: - if self.model is None or self.processor is None: - return backend_pb2.Result(success=False, message="Model not loaded") - - import torch - import torchaudio - from liquid_audio import ChatState - - voice = request.voice.lower() if request.voice else self.voice - voice = voice.removeprefix("lfm2:").removeprefix("lfm:") - if voice not in VOICE_PROMPTS: - voice = self.voice - system_prompt = VOICE_PROMPTS[voice] - - chat = ChatState(self.processor) - chat.new_turn("system") - chat.add_text(system_prompt) - chat.end_turn() - chat.new_turn("user") - chat.add_text(request.text or "") - chat.end_turn() - chat.new_turn("assistant") - - audio_top_k = int(self.options.get("audio_top_k", 64)) - audio_temp = float(self.options.get("audio_temperature", 0.8)) - max_new = int(self.options.get("max_new_tokens", 2048)) - - audio_out = [] - for tok in self.model.generate_sequential( - **chat, - max_new_tokens=max_new, - audio_temperature=audio_temp, - audio_top_k=audio_top_k, - ): - if tok.numel() > 1: - audio_out.append(tok) - - if len(audio_out) <= 1: - return backend_pb2.Result(success=False, message="No audio frames generated") - - # Drop the trailing end-of-audio frame, matching the package's examples. - audio_codes = torch.stack(audio_out[:-1], 1).unsqueeze(0) - waveform = self.processor.decode(audio_codes) - - out_path = request.dst - if not out_path: - return backend_pb2.Result(success=False, message="dst path is required") - os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True) - # soundfile in preference to torchaudio.save — the latter routes - # through torchcodec, whose native libs need NVIDIA NPP that we - # don't bundle in the cuda13 image. - import soundfile as _sf - _sf.write(out_path, waveform.cpu().numpy().squeeze(0).T, 24_000) - - return backend_pb2.Result(success=True) - except Exception as exc: - print(f"TTS failed: {exc}", file=sys.stderr) - print(traceback.format_exc(), file=sys.stderr) - return backend_pb2.Result(success=False, message=str(exc)) - - - def AudioToAudioStream(self, request_iterator, context): - """Bidirectional any-to-any speech-to-speech stream. - - See `backend.proto` AudioToAudioStream for the wire protocol. Audio - is decoded once per turn here; chunked detokenization for sub-second - TTFB is left to a future iteration once the LFM2AudioDetokenizer - gains a streaming entry point. - """ - try: - yield from self._audio_to_audio_stream(request_iterator, context) - except Exception as exc: - print(f"AudioToAudioStream failed: {exc}", file=sys.stderr) - print(traceback.format_exc(), file=sys.stderr) - yield backend_pb2.AudioToAudioResponse( - event="error", - meta=json.dumps({"message": str(exc)}).encode("utf-8"), - ) - - def _audio_to_audio_stream(self, request_iterator, context): - if self.model is None or self.processor is None: - raise RuntimeError("Model not loaded") - - import torch - import torchaudio - from liquid_audio import ChatState - - cfg = None - chat = None - input_sample_rate = 16000 - output_sample_rate = 24000 - sequence = 0 - - def _new_event(event, **kwargs): - nonlocal sequence - sequence += 1 - kwargs.setdefault("sequence", sequence) - return backend_pb2.AudioToAudioResponse(event=event, **kwargs) - - def _ensure_chat(): - """Build a fresh ChatState seeded with the system prompt.""" - nonlocal chat - chat = ChatState(self.processor) - system_prompt = (cfg.system_prompt if cfg and cfg.system_prompt - else "Respond with interleaved text and audio.") - chat.new_turn("system") - chat.add_text(system_prompt) - chat.end_turn() - - # Buffers for the in-flight user turn - pcm_buffer = bytearray() - - def _consume_user_turn(): - nonlocal pcm_buffer - if not pcm_buffer: - return - # Avoid the bytes(pcm_buffer) copy and let the float widen happen - # in-place: numpy view → torch view → in-place divide. - import numpy as np - arr = np.frombuffer(memoryview(pcm_buffer), dtype=np.int16) - wav = torch.from_numpy(arr).to(torch.float32).div_(32768.0).unsqueeze(0) - chat.new_turn("user") - chat.add_audio(wav, input_sample_rate) - chat.end_turn() - pcm_buffer = bytearray() - - def _run_generation(): - """Run generate_interleaved; yield response events as we go.""" - chat.new_turn("assistant") - audio_top_k = int(self.options.get("audio_top_k", 4)) - audio_temp = float(self.options.get("audio_temperature", 1.0)) - text_top_k = int(self.options.get("text_top_k", 0)) or None - text_temp = float(self.options.get("text_temperature", 0)) or None - max_new = int(self.options.get("max_new_tokens", 512)) - - audio_tokens = [] - for tok in self.model.generate_interleaved( - **chat, - max_new_tokens=max_new, - text_temperature=text_temp, - text_top_k=text_top_k, - audio_temperature=audio_temp, - audio_top_k=audio_top_k, - ): - if tok.numel() == 1: - if tok.item() == IM_END_TOKEN: - break - text = self.processor.text.decode(tok) - if not text: - continue - yield _new_event( - "response.audio_transcript.delta", - meta=json.dumps({"delta": text}).encode("utf-8"), - ) - else: - audio_tokens.append(tok) - - # Detokenize the accumulated audio at end-of-turn — the - # LFM2AudioDetokenizer is non-streaming today. - if len(audio_tokens) > 1: - audio_codes = torch.stack(audio_tokens[:-1], 1).unsqueeze(0) - waveform = self.processor.decode(audio_codes) - # Convert to s16le PCM bytes at output_sample_rate - if output_sample_rate != 24000: - waveform = torchaudio.functional.resample( - waveform.cpu(), 24000, output_sample_rate - ) - pcm = (waveform.cpu().squeeze(0).clamp(-1, 1) * 32767.0).to( - torch.int16 - ).numpy().tobytes() - yield _new_event( - "response.audio.delta", - pcm=pcm, - sample_rate=output_sample_rate, - ) - - yield _new_event("response.done", meta=b"{}") - - for req in request_iterator: - if not context.is_active(): - return - payload = req.WhichOneof("payload") - if payload == "config": - cfg = req.config - if cfg.input_sample_rate > 0: - input_sample_rate = cfg.input_sample_rate - if cfg.output_sample_rate > 0: - output_sample_rate = cfg.output_sample_rate - # The first config implicitly resets state. - _ensure_chat() - pcm_buffer = bytearray() - elif payload == "frame": - if chat is None: - _ensure_chat() - if req.frame.pcm: - pcm_buffer.extend(req.frame.pcm) - if req.frame.end_of_input: - _consume_user_turn() - yield from _run_generation() - elif payload == "control": - event = req.control.event - if event == "input_audio_buffer.commit": - _consume_user_turn() - yield from _run_generation() - elif event == "response.cancel": - # Synchronous generation here means cancel can only - # take effect between turns; we ack so the client unblocks. - yield _new_event("response.done", meta=b'{"cancelled":true}') - elif event == "session.update": - # Free-form session re-config; treat as a soft reset. - _ensure_chat() - pcm_buffer = bytearray() - # Unknown events are ignored — forward-compatible. - - - def AudioTranscription(self, request, context): - try: - if self.model is None or self.processor is None: - return backend_pb2.TranscriptResult(segments=[], text="") - - import torchaudio - from liquid_audio import ChatState - - audio_path = request.dst - if not audio_path: - return backend_pb2.TranscriptResult(segments=[], text="") - - chat = ChatState(self.processor) - chat.new_turn("system") - chat.add_text("Perform ASR.") - chat.end_turn() - chat.new_turn("user") - # soundfile in preference to torchaudio.load — the latter routes - # through torchcodec which needs NVIDIA NPP libs we don't bundle. - import soundfile as _sf - import torch - audio_np, sr = _sf.read(audio_path, dtype="float32", always_2d=True) - wav = torch.from_numpy(audio_np.T) # (channels, samples) - if wav.shape[0] > 1: - # Down-mix to mono — the processor expects a single channel - wav = wav.mean(dim=0, keepdim=True) - chat.add_audio(wav, sr) - chat.end_turn() - chat.new_turn("assistant") - - max_new = int(self.options.get("max_new_tokens", 1024)) - - pieces = [] - for tok in self.model.generate_sequential(**chat, max_new_tokens=max_new): - if tok.numel() == 1: - if tok.item() == IM_END_TOKEN: - break - pieces.append(self.processor.text.decode(tok)) - - text = "".join(pieces).strip() - duration_ms = int((wav.shape[1] / sr) * 1000) - segment = backend_pb2.TranscriptSegment( - id=0, start=0, end=duration_ms, text=text, tokens=[], - ) - return backend_pb2.TranscriptResult(segments=[segment], text=text) - except Exception as exc: - print(f"AudioTranscription failed: {exc}", file=sys.stderr) - print(traceback.format_exc(), file=sys.stderr) - return backend_pb2.TranscriptResult(segments=[], text="") - - - def StartFineTune(self, request, context): - if self.active_job is not None and not self.active_job.completed: - return backend_pb2.FineTuneJobResult( - job_id="", success=False, - message="A fine-tuning job is already running", - ) - - job_id = request.job_id or str(uuid.uuid4()) - job = ActiveJob(job_id) - self.active_job = job - - thread = threading.Thread(target=self._run_training, args=(request, job), daemon=True) - job.thread = thread - thread.start() - - return backend_pb2.FineTuneJobResult( - job_id=job_id, success=True, message="Training started", - ) - - def FineTuneProgress(self, request, context): - if self.active_job is None or self.active_job.job_id != request.job_id: - context.set_code(grpc.StatusCode.NOT_FOUND) - context.set_details(f"Job {request.job_id} not found") - return - - job = self.active_job - while True: - try: - update = job.progress_queue.get(timeout=1.0) - except queue.Empty: - if job.completed or job.stopped: - break - if not context.is_active(): - break - continue - if update is None: - break - yield update - if update.status in ("completed", "failed", "stopped"): - break - - def StopFineTune(self, request, context): - # We can't kill the Accelerate training loop mid-step cleanly from here; - # LocalAI's job manager kills the backend process on stop. The flag below - # at least lets the progress stream terminate quickly. - if self.active_job is not None and self.active_job.job_id == request.job_id: - self.active_job.stopped = True - self.active_job.progress_queue.put(None) - return backend_pb2.Result(success=True, message="OK") - - def _run_training(self, request, job): - try: - self._do_train(request, job) - job.completed = True - job.progress_queue.put(backend_pb2.FineTuneProgressUpdate( - job_id=job.job_id, status="completed", message="Training completed", - progress_percent=100.0, - )) - except Exception as exc: - job.error = str(exc) - job.completed = True - print(f"Training failed: {exc}", file=sys.stderr) - print(traceback.format_exc(), file=sys.stderr) - job.progress_queue.put(backend_pb2.FineTuneProgressUpdate( - job_id=job.job_id, status="failed", message=str(exc), - )) - finally: - job.progress_queue.put(None) - - def _do_train(self, request, job): - from liquid_audio import LFM2AudioModel # noqa: F401 (sanity import) - from liquid_audio.data.dataloader import LFM2DataLoader - from liquid_audio.trainer import Trainer - - model_id = request.model or self.model_id or "LiquidAI/LFM2.5-Audio-1.5B" - - dataset_path = request.dataset_source - if not dataset_path: - raise ValueError("dataset_source is required (path to a preprocessed dataset)") - - extras = dict(request.extra_options) if request.extra_options else {} - val_path = extras.get("val_dataset") - - # Map FineTuneRequest hyperparameters to liquid_audio.Trainer constructor args - lr = request.learning_rate or 3e-5 - max_steps = request.max_steps or 1000 - warmup_steps = request.warmup_steps or min(100, max_steps // 10) - batch_size = request.batch_size or 16 - save_interval = request.save_steps or max(1, max_steps // 4) - - output_dir = request.output_dir or os.path.join( - os.environ.get("LIQUID_AUDIO_OUTPUT_DIR", "/tmp"), - f"liquid-audio-{job.job_id}", - ) - os.makedirs(output_dir, exist_ok=True) - - job.progress_queue.put(backend_pb2.FineTuneProgressUpdate( - job_id=job.job_id, status="loading_dataset", - message=f"Loading preprocessed dataset from {dataset_path}", - )) - train_data = LFM2DataLoader(dataset_path) - val_data = LFM2DataLoader(val_path) if val_path else None - - job.progress_queue.put(backend_pb2.FineTuneProgressUpdate( - job_id=job.job_id, status="loading_model", - message=f"Loading base model {model_id}", - )) - - # The Liquid Trainer logs via self.accelerator.print; we subclass it to - # also push progress events onto the queue every logging_interval steps. - progress_q = job.progress_queue - - class QueuedTrainer(Trainer): - def log(self_, model_output): - if self_.step > 0 and self_.step % self_.logging_interval == 0: - try: - loss = self_.accelerator.reduce( - model_output.loss.detach(), reduction="mean" - ).item() - except Exception: - loss = float("nan") - lr_now = self_.optimizer.param_groups[0]["lr"] - pct = (self_.step / self_.max_steps * 100.0) if self_.max_steps else 0.0 - progress_q.put(backend_pb2.FineTuneProgressUpdate( - job_id=job.job_id, - current_step=int(self_.step), - total_steps=int(self_.max_steps), - current_epoch=float(self_.epoch), - loss=float(loss), - learning_rate=float(lr_now), - progress_percent=float(pct), - status="training", - )) - # Honour stop requests: raising here terminates the loop cleanly - if job.stopped: - raise KeyboardInterrupt("stop requested") - return super().log(model_output) - - def validate(self_): - progress_q.put(backend_pb2.FineTuneProgressUpdate( - job_id=job.job_id, current_step=int(self_.step), - total_steps=int(self_.max_steps), status="training", - message=f"Running validation at step {self_.step}", - )) - return super().validate() - - trainer = QueuedTrainer( - model_id=model_id, - train_data=train_data, - val_data=val_data, - lr=lr, - max_steps=max_steps, - warmup_steps=warmup_steps, - batch_size=batch_size, - save_interval=save_interval, - output_dir=output_dir, - weight_decay=request.weight_decay or 0.1, - ) - - job.progress_queue.put(backend_pb2.FineTuneProgressUpdate( - job_id=job.job_id, status="training", message="Training started", - total_steps=int(max_steps), - )) - trainer.train() - - job.progress_queue.put(backend_pb2.FineTuneProgressUpdate( - job_id=job.job_id, status="saving", - message=f"Saved final model to {output_dir}", - checkpoint_path=os.path.join(output_dir, "final"), - )) - - - def _build_chat_state(self, messages, user_prompt, tools_prelude=None): - """Build a ChatState from a list of (role, content) tuples plus an optional final user turn. - - tools_prelude, when non-empty, is prepended as an extra system turn carrying - the LFM2 tool-list block — mirrors gallery/lfm.yaml's `function:` template - so the model sees the same prompt shape whether served via llama-cpp or here. - """ - from liquid_audio import ChatState - chat = ChatState(self.processor) - if tools_prelude: - chat.new_turn("system") - chat.add_text(tools_prelude) - chat.end_turn() - for role, content in messages: - chat.new_turn(role) - chat.add_text(content) - chat.end_turn() - if user_prompt: - chat.new_turn("user") - chat.add_text(user_prompt) - chat.end_turn() - chat.new_turn("assistant") - return chat - - def _collect_messages(self, request): - """Translate PredictOptions.Messages into (role, content) tuples.""" - out = [] - for m in request.Messages: - role = (m.role or "user").lower() - if role not in ("system", "user", "assistant"): - role = "user" - out.append((role, m.content or "")) - return out - - def _render_tools_prelude(self, request): - """Build the LFM2 `<|tool_list_start|>…<|tool_list_end|>` system prelude - from request.Tools (OpenAI Chat-Completions tool JSON). Returns "" when - no tools are attached. Output mirrors gallery/lfm.yaml's `function:` - template so the model sees the same prompt whether routed via llama-cpp - or this backend.""" - tools_raw = getattr(request, "Tools", "") or "" - if not tools_raw: - return "" - try: - tools = json.loads(tools_raw) - except json.JSONDecodeError: - print(f"liquid-audio: ignoring malformed Tools JSON: {tools_raw[:200]!r}", - file=sys.stderr) - return "" - if not isinstance(tools, list) or not tools: - return "" - # The LFM2 chat template uses single-quoted Python-dict-ish syntax in - # examples, but the tokenizer treats this whole block as opaque text; - # JSON works fine and is what other backends emit. - return ( - "You are a function calling AI model. You are provided with functions to " - "execute. You may call one or more functions to assist with the user query. " - "Don't make assumptions about what values to plug into functions.\n" - "List of tools: <|tool_list_start|>" - + json.dumps(tools, separators=(",", ":")) - + "<|tool_list_end|>" - ) - - def _generate_text_stream(self, request): - """Yield text-only deltas from generate_sequential. Caller joins for unary Predict.""" - if self.model is None or self.processor is None: - raise RuntimeError("Model not loaded") - messages = self._collect_messages(request) - user_prompt = request.Prompt or None - tools_prelude = self._render_tools_prelude(request) - # If the request already carries Messages, Prompt is the templated form - # of the same content — don't append a duplicate user turn. - chat = self._build_chat_state( - messages, - user_prompt if not messages else None, - tools_prelude=tools_prelude, - ) - - max_new = request.Tokens if request.Tokens > 0 else int(self.options.get("max_new_tokens", 512)) - temperature = request.Temperature if request.Temperature > 0 else None - top_k = request.TopK if request.TopK > 0 else None - - for tok in self.model.generate_sequential( - **chat, - max_new_tokens=max_new, - text_temperature=temperature, - text_top_k=top_k, - ): - if tok.numel() == 1: - if tok.item() == IM_END_TOKEN: - break - yield self.processor.text.decode(tok) - - -def serve(address): - server = grpc.server( - futures.ThreadPoolExecutor(max_workers=MAX_WORKERS), - options=[ - ('grpc.max_message_length', 50 * 1024 * 1024), - ('grpc.max_send_message_length', 50 * 1024 * 1024), - ('grpc.max_receive_message_length', 50 * 1024 * 1024), - ], - interceptors=get_auth_interceptors(), - ) - backend_pb2_grpc.add_BackendServicer_to_server(BackendServicer(), server) - server.add_insecure_port(address) - server.start() - print(f"Liquid-audio backend listening on {address}", file=sys.stderr, flush=True) - - def stop(_signum, _frame): - server.stop(0) - sys.exit(0) - - signal.signal(signal.SIGTERM, stop) - signal.signal(signal.SIGINT, stop) - - try: - while True: - time.sleep(_ONE_DAY_IN_SECONDS) - except KeyboardInterrupt: - server.stop(0) - - -if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Liquid Audio gRPC backend") - parser.add_argument("--addr", default="localhost:50051", help="gRPC server address") - args = parser.parse_args() - serve(args.addr) diff --git a/backend/python/liquid-audio/install.sh b/backend/python/liquid-audio/install.sh deleted file mode 100755 index fe0f9caad..000000000 --- a/backend/python/liquid-audio/install.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/bin/bash -set -e - -# liquid-audio requires Python ≥ 3.12 (per its pyproject.toml); the default -# portable Python in libbackend.sh is 3.10. Override before sourcing. -export PYTHON_VERSION="${PYTHON_VERSION:-3.12}" -export PYTHON_PATCH="${PYTHON_PATCH:-11}" - -backend_dir=$(dirname $0) -if [ -d $backend_dir/common ]; then - source $backend_dir/common/libbackend.sh -else - source $backend_dir/../common/libbackend.sh -fi - -# liquid-audio's torch wheels are large; allow upgrades to satisfy transitive pins -EXTRA_PIP_INSTALL_FLAGS+=" --upgrade" -# --index-strategy is a uv-only flag. The darwin/MPS build installs with pip -# (USE_PIP=true in scripts/build/python-darwin.sh), which rejects it. Only add -# it on the uv path; Linux/CUDA resolution is unchanged. -if [ "x${USE_PIP:-}" != "xtrue" ]; then - EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-first-match" -fi -installRequirements diff --git a/backend/python/liquid-audio/protogen.sh b/backend/python/liquid-audio/protogen.sh deleted file mode 100755 index df3325c6f..000000000 --- a/backend/python/liquid-audio/protogen.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash -set -e - -backend_dir=$(dirname $0) -if [ -d $backend_dir/common ]; then - source $backend_dir/common/libbackend.sh -else - source $backend_dir/../common/libbackend.sh -fi - -runProtogen diff --git a/backend/python/liquid-audio/requirements-cpu.txt b/backend/python/liquid-audio/requirements-cpu.txt deleted file mode 100644 index c2fee25e3..000000000 --- a/backend/python/liquid-audio/requirements-cpu.txt +++ /dev/null @@ -1,13 +0,0 @@ ---extra-index-url https://download.pytorch.org/whl/cpu -torch>=2.8.0 -torchaudio>=2.8.0 -torchcodec>=0.9.1 -transformers>=4.55.4 -accelerate>=1.10.1 -datasets>=4.8.4 -einops>=0.8.1 -librosa>=0.11.0 -soundfile>=0.12.1 -sentencepiece>=0.2.1 -huggingface-hub>=1.3.0 -liquid-audio>=1.2.0 diff --git a/backend/python/liquid-audio/requirements-cublas12.txt b/backend/python/liquid-audio/requirements-cublas12.txt deleted file mode 100644 index 8f1a965bb..000000000 --- a/backend/python/liquid-audio/requirements-cublas12.txt +++ /dev/null @@ -1,13 +0,0 @@ ---extra-index-url https://download.pytorch.org/whl/cu121 -torch>=2.8.0 -torchaudio>=2.8.0 -torchcodec>=0.9.1 -transformers>=4.55.4 -accelerate>=1.10.1 -datasets>=4.8.4 -einops>=0.8.1 -librosa>=0.11.0 -soundfile>=0.12.1 -sentencepiece>=0.2.1 -huggingface-hub>=1.3.0 -liquid-audio>=1.2.0 diff --git a/backend/python/liquid-audio/requirements-cublas13.txt b/backend/python/liquid-audio/requirements-cublas13.txt deleted file mode 100644 index 4ab768158..000000000 --- a/backend/python/liquid-audio/requirements-cublas13.txt +++ /dev/null @@ -1,13 +0,0 @@ ---extra-index-url https://download.pytorch.org/whl/cu130 -torch>=2.8.0 -torchaudio>=2.8.0 -torchcodec>=0.9.1 -transformers>=4.55.4 -accelerate>=1.10.1 -datasets>=4.8.4 -einops>=0.8.1 -librosa>=0.11.0 -soundfile>=0.12.1 -sentencepiece>=0.2.1 -huggingface-hub>=1.3.0 -liquid-audio>=1.2.0 diff --git a/backend/python/liquid-audio/requirements-hipblas.txt b/backend/python/liquid-audio/requirements-hipblas.txt deleted file mode 100644 index 304f934b1..000000000 --- a/backend/python/liquid-audio/requirements-hipblas.txt +++ /dev/null @@ -1,13 +0,0 @@ ---extra-index-url https://download.pytorch.org/whl/rocm7.0 -torch>=2.8.0 -torchaudio>=2.8.0 -torchcodec>=0.9.1 -transformers>=4.55.4 -accelerate>=1.10.1 -datasets>=4.8.4 -einops>=0.8.1 -librosa>=0.11.0 -soundfile>=0.12.1 -sentencepiece>=0.2.1 -huggingface-hub>=1.3.0 -liquid-audio>=1.2.0 diff --git a/backend/python/liquid-audio/requirements-l4t13.txt b/backend/python/liquid-audio/requirements-l4t13.txt deleted file mode 100644 index 7ecc7203d..000000000 --- a/backend/python/liquid-audio/requirements-l4t13.txt +++ /dev/null @@ -1,13 +0,0 @@ ---extra-index-url https://pypi.jetson-ai-lab.io/jp7/cu130 -torch>=2.8.0 -torchaudio>=2.8.0 -torchcodec>=0.9.1 -transformers>=4.55.4 -accelerate>=1.10.1 -datasets>=4.8.4 -einops>=0.8.1 -librosa>=0.11.0 -soundfile>=0.12.1 -sentencepiece>=0.2.1 -huggingface-hub>=1.3.0 -liquid-audio>=1.2.0 diff --git a/backend/python/liquid-audio/requirements-mps.txt b/backend/python/liquid-audio/requirements-mps.txt deleted file mode 100644 index 3c9c36cca..000000000 --- a/backend/python/liquid-audio/requirements-mps.txt +++ /dev/null @@ -1,13 +0,0 @@ -# MPS (Apple Silicon / Metal) build profile - installed by the darwin CI job. -torch>=2.8.0 -torchaudio>=2.8.0 -torchcodec>=0.9.1 -transformers>=4.55.4 -accelerate>=1.10.1 -datasets>=4.8.4 -einops>=0.8.1 -librosa>=0.11.0 -soundfile>=0.12.1 -sentencepiece>=0.2.1 -huggingface-hub>=1.3.0 -liquid-audio>=1.2.0 diff --git a/backend/python/liquid-audio/requirements.txt b/backend/python/liquid-audio/requirements.txt deleted file mode 100644 index 0834a8fcd..000000000 --- a/backend/python/liquid-audio/requirements.txt +++ /dev/null @@ -1,3 +0,0 @@ -grpcio==1.78.1 -protobuf -certifi diff --git a/backend/python/liquid-audio/run.sh b/backend/python/liquid-audio/run.sh deleted file mode 100755 index bd17c6e1d..000000000 --- a/backend/python/liquid-audio/run.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash - -backend_dir=$(dirname $0) -if [ -d $backend_dir/common ]; then - source $backend_dir/common/libbackend.sh -else - source $backend_dir/../common/libbackend.sh -fi - -startBackend $@ diff --git a/backend/python/liquid-audio/test.py b/backend/python/liquid-audio/test.py deleted file mode 100644 index 450ce3156..000000000 --- a/backend/python/liquid-audio/test.py +++ /dev/null @@ -1,89 +0,0 @@ -"""Smoke tests for the liquid-audio backend. - -These run without contacting HuggingFace or loading model weights: -they only verify that the gRPC service starts and Health() responds. - -To run an end-to-end inference test, set LIQUID_AUDIO_MODEL_ID -(e.g. "LiquidAI/LFM2.5-Audio-1.5B") in the environment — see test_inference(). -""" -import os -import subprocess -import sys -import time -import unittest - -import grpc - -# Ensure generated protobuf stubs are importable -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) - -import backend_pb2 -import backend_pb2_grpc - - -class TestBackend(unittest.TestCase): - @classmethod - def setUpClass(cls): - addr = os.environ.get("LIQUID_AUDIO_TEST_ADDR", "localhost:50053") - cls.addr = addr - cls.server = subprocess.Popen( - [sys.executable, os.path.join(os.path.dirname(__file__), "backend.py"), "--addr", addr], - ) - time.sleep(2) # Give the server a moment to bind - - @classmethod - def tearDownClass(cls): - cls.server.terminate() - try: - cls.server.wait(timeout=5) - except subprocess.TimeoutExpired: - cls.server.kill() - - def _stub(self): - channel = grpc.insecure_channel(self.addr) - return backend_pb2_grpc.BackendStub(channel) - - def test_health(self): - stub = self._stub() - reply = stub.Health(backend_pb2.HealthMessage(), timeout=5) - self.assertEqual(reply.message, b"OK") - - def test_load_finetune_mode_without_weights(self): - """Loading in fine-tune mode should succeed without pulling model weights.""" - stub = self._stub() - result = stub.LoadModel( - backend_pb2.ModelOptions( - Model="LiquidAI/LFM2.5-Audio-1.5B", - Options=["mode:finetune"], - ), - timeout=10, - ) - self.assertTrue(result.success, msg=result.message) - - @unittest.skipUnless(os.environ.get("LIQUID_AUDIO_MODEL_ID"), - "Set LIQUID_AUDIO_MODEL_ID to run an end-to-end inference smoke test") - def test_inference(self): - """End-to-end: load a real LFM2-Audio model and run one short prediction.""" - stub = self._stub() - model_id = os.environ["LIQUID_AUDIO_MODEL_ID"] - result = stub.LoadModel( - backend_pb2.ModelOptions( - Model=model_id, - Options=["mode:chat"], - ), - timeout=600, - ) - self.assertTrue(result.success, msg=result.message) - reply = stub.Predict( - backend_pb2.PredictOptions( - Prompt="Hello!", - Tokens=8, - Temperature=0.0, - ), - timeout=120, - ) - self.assertGreater(len(reply.message), 0) - - -if __name__ == "__main__": - unittest.main() diff --git a/backend/python/liquid-audio/test.sh b/backend/python/liquid-audio/test.sh deleted file mode 100755 index eb59f2aaf..000000000 --- a/backend/python/liquid-audio/test.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash -set -e - -backend_dir=$(dirname $0) -if [ -d $backend_dir/common ]; then - source $backend_dir/common/libbackend.sh -else - source $backend_dir/../common/libbackend.sh -fi - -runUnittests diff --git a/core/backend/audio_to_audio.go b/core/backend/audio_to_audio.go new file mode 100644 index 000000000..2ec628743 --- /dev/null +++ b/core/backend/audio_to_audio.go @@ -0,0 +1,60 @@ +package backend + +import ( + "context" + "fmt" + "sync" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/pkg/grpc" + "github.com/mudler/LocalAI/pkg/model" +) + +// AudioToAudioStream is the duplex stream to a speech-to-speech backend. It +// holds the global backend slot until Close, so a long realtime session counts +// as one in-flight backend call, like any other streaming RPC. +type AudioToAudioStream struct { + grpc.AudioToAudioStreamClient + release func() + releaseOnce sync.Once +} + +// Close ends the send side and releases the backend slot. It is safe to call +// more than once. The caller should also cancel the context given to +// ModelAudioToAudioStream so a blocked Recv returns. +func (s *AudioToAudioStream) Close() error { + err := s.CloseSend() + s.releaseOnce.Do(s.release) + return err +} + +// ModelAudioToAudioStream loads the model with its standard options (so +// options such as parallel and the multimodal projector apply) and opens the +// bidirectional AudioToAudioStream RPC. The caller sends the config message +// first and owns the stream until Close. +func ModelAudioToAudioStream( + ctx context.Context, + loader *model.ModelLoader, + appConfig *config.ApplicationConfig, + modelConfig config.ModelConfig, +) (*AudioToAudioStream, error) { + mopts := ModelOptions(modelConfig, appConfig) + s2sModel, err := loader.Load(mopts...) + if err != nil { + recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil) + return nil, err + } + if s2sModel == nil { + return nil, fmt.Errorf("could not load speech-to-speech model %q", modelConfig.Model) + } + release, err := AcquireGlobalBackendSlot() + if err != nil { + return nil, err + } + stream, err := s2sModel.AudioToAudioStream(ctx) + if err != nil { + release() + return nil, err + } + return &AudioToAudioStream{AudioToAudioStreamClient: stream, release: release}, nil +} diff --git a/core/config/backend_capabilities.go b/core/config/backend_capabilities.go index cd728d863..6a85e697d 100644 --- a/core/config/backend_capabilities.go +++ b/core/config/backend_capabilities.go @@ -324,12 +324,14 @@ var BackendCapabilities = map[string]BackendCapability{ // narrowed per model in VoiceCloningForModel, since the vast majority of // llama-cpp models in the gallery are text LLMs that clone nothing. "llama-cpp": { - GRPCMethods: []GRPCMethod{MethodPredict, MethodPredictStream, MethodEmbedding, MethodTokenizeString, MethodScore, MethodTTS, MethodTTSStream}, - PossibleUsecases: []string{UsecaseChat, UsecaseCompletion, UsecaseEdit, UsecaseEmbeddings, UsecaseTokenize, UsecaseVision, UsecaseScore, UsecaseDecisions, UsecaseTTS}, - DefaultUsecases: []string{UsecaseChat}, - AcceptsImages: true, // requires mmproj - VoiceCloning: referenceVoiceCloning(), - Description: "llama.cpp GGUF models: LLM inference with optional vision via mmproj, and Qwen3-TTS speech with reference-audio cloning", + GRPCMethods: []GRPCMethod{MethodPredict, MethodPredictStream, MethodEmbedding, MethodTokenizeString, MethodScore, MethodTTS, MethodTTSStream, + MethodAudioTranscription, MethodAudioToAudioStream}, + PossibleUsecases: []string{UsecaseChat, UsecaseCompletion, UsecaseEdit, UsecaseEmbeddings, UsecaseTokenize, UsecaseVision, UsecaseScore, UsecaseDecisions, UsecaseTTS, + UsecaseTranscript, UsecaseRealtimeAudio}, + DefaultUsecases: []string{UsecaseChat}, + AcceptsImages: true, // requires mmproj + VoiceCloning: referenceVoiceCloning(), + Description: "llama.cpp GGUF models: LLM inference with optional vision via mmproj, Qwen3-TTS speech with reference-audio cloning, and LFM3-Audio speech recognition, text-to-speech and speech-to-speech", }, // privacy-filter is the standalone GGML engine (backend/cpp/privacy-filter, // wrapping privacy-filter.cpp) for the openai-privacy-filter PII/NER token @@ -774,15 +776,6 @@ var BackendCapabilities = map[string]BackendCapability{ Description: "Meta MusicGen via transformers — music generation from text", }, - // --- Any-to-any audio backends --- - "liquid-audio": { - GRPCMethods: []GRPCMethod{MethodPredict, MethodPredictStream, MethodAudioTranscription, MethodTTS, MethodAudioToAudioStream, MethodVAD}, - PossibleUsecases: []string{UsecaseChat, UsecaseCompletion, UsecaseTranscript, UsecaseTTS, UsecaseRealtimeAudio, UsecaseVAD}, - DefaultUsecases: []string{UsecaseRealtimeAudio, UsecaseChat, UsecaseTranscript, UsecaseTTS, UsecaseVAD}, - AcceptsAudios: true, - Description: "LFM2 / LFM2.5-Audio — self-contained any-to-any audio model for the Realtime API; also exposes chat, transcription, TTS and a stub energy-based VAD endpoint", - }, - // --- Audio transform backends --- "localvqe": { GRPCMethods: []GRPCMethod{MethodAudioTransform}, diff --git a/core/config/backend_capabilities_test.go b/core/config/backend_capabilities_test.go index 73005e8b3..3a4617907 100644 --- a/core/config/backend_capabilities_test.go +++ b/core/config/backend_capabilities_test.go @@ -502,3 +502,23 @@ var _ = Describe("timestral forecast capability", func() { Expect(*flags).To(Equal(FLAG_FORECAST)) }) }) + +// LFM3-Audio runs through llama.cpp for speech recognition and for the +// speech-to-speech loop, so the llama-cpp entry advertises both RPCs. Its +// default stays chat: a bare GGUF is still a chat model. +var _ = Describe("llama-cpp speech and realtime capabilities", func() { + It("advertises the transcription and audio-to-audio RPCs", func() { + capability := GetBackendCapability("llama-cpp") + Expect(capability).NotTo(BeNil()) + Expect(capability.GRPCMethods).To(ContainElements(MethodAudioTranscription, MethodAudioToAudioStream)) + }) + + It("accepts the transcript and realtime_audio usecases", func() { + Expect(IsValidUsecaseForBackend("llama-cpp", UsecaseTranscript)).To(BeTrue()) + Expect(IsValidUsecaseForBackend("llama-cpp", UsecaseRealtimeAudio)).To(BeTrue()) + }) + + It("keeps chat as its only default usecase", func() { + Expect(GetBackendCapability("llama-cpp").DefaultUsecases).To(Equal([]string{UsecaseChat})) + }) +}) diff --git a/core/config/gguf.go b/core/config/gguf.go index f61cd0fe8..42d200387 100644 --- a/core/config/gguf.go +++ b/core/config/gguf.go @@ -30,6 +30,18 @@ func reservedNonChatModel(cfg *ModelConfig) bool { // projector carrying the speech-synthesis pipeline (ggml-org/llama.cpp#26254). const genAudioEncoderKey = "clip.has_gen_audio_encoder" +// audioEncoderKey is the mmproj metadata flag for the audio input side of a +// projector (a speech encoder). +const audioEncoderKey = "clip.has_audio_encoder" + +// audioProjectorTypeKey names the audio input encoder's family. +const audioProjectorTypeKey = "clip.audio.projector_type" + +// lfm3AudioProjectorType is the audio encoder family llama.cpp writes for +// LFM3-Audio. Qwen3-TTS sets both encoder flags as well, with its own speaker +// encoder (qwen3tts_spkenc), so the flags cannot tell the two families apart. +const lfm3AudioProjectorType = "lfm3a" + // HasGenAudioProjector reports whether a parsed mmproj GGUF holds a gen-audio // pipeline (Qwen3-TTS) rather than a vision tower. This is the same flag // mtmd_helper_gen_audio itself checks before building the pipeline, so it is @@ -37,10 +49,37 @@ const genAudioEncoderKey = "clip.has_gen_audio_encoder" // otherwise indistinguishable from a vision projector by name alone, and every // TTS repo names it mmproj-*.gguf exactly like a vision one. func HasGenAudioProjector(f *gguf.GGUFFile) bool { + return ggufBoolFlag(f, genAudioEncoderKey) +} + +// HasAudioEncoderProjector reports whether a parsed mmproj GGUF carries a +// speech encoder on its input side. Speaker-only encoders (Qwen3-TTS reference +// audio) set it too, so it is only meaningful together with +// HasGenAudioProjector, which identifies a speech-to-speech projector. +func HasAudioEncoderProjector(f *gguf.GGUFFile) bool { + return ggufBoolFlag(f, audioEncoderKey) +} + +// HasSpeechToSpeechProjector reports whether a parsed mmproj GGUF is the +// projector of an any-to-any speech model (LFM3-Audio): an LFM3 audio encoder +// together with the gen-audio decoder. The decoder alone is TTS, and the +// encoder alone is reference-audio input, so neither flag decides on its own. +func HasSpeechToSpeechProjector(f *gguf.GGUFFile) bool { + if !HasGenAudioProjector(f) || !HasAudioEncoderProjector(f) { + return false + } + kv, ok := f.Header.MetadataKV.Get(audioProjectorTypeKey) + return ok && kv.ValueType == gguf.GGUFMetadataValueTypeString && kv.ValueString() == lfm3AudioProjectorType +} + +// ggufBoolFlag reads a boolean metadata key. A missing key, a value of another +// type or a nil file all read as false: ValueBool panics on a type mismatch, +// and these files come from arbitrary user repos. +func ggufBoolFlag(f *gguf.GGUFFile, key string) bool { if f == nil { return false } - kv, ok := f.Header.MetadataKV.Get(genAudioEncoderKey) + kv, ok := f.Header.MetadataKV.Get(key) if !ok || kv.ValueType != gguf.GGUFMetadataValueTypeBool { return false } diff --git a/core/config/gguf_audio_encoder_test.go b/core/config/gguf_audio_encoder_test.go new file mode 100644 index 000000000..d4753c933 --- /dev/null +++ b/core/config/gguf_audio_encoder_test.go @@ -0,0 +1,94 @@ +package config_test + +import ( + . "github.com/mudler/LocalAI/core/config" + + gguf "github.com/gpustack/gguf-parser-go" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// An mmproj with clip.has_audio_encoder carries the audio input side of an +// any-to-any model (LFM3-Audio). It is not the gen-audio decoder, so the +// importer uses it to tell a speech-to-speech model from a TTS-only one. +var _ = Describe("HasAudioEncoderProjector", func() { + mmproj := func(key string, valueType gguf.GGUFMetadataValueType, value any) *gguf.GGUFFile { + return &gguf.GGUFFile{ + Header: gguf.GGUFHeader{ + MetadataKV: gguf.GGUFMetadataKVs{ + {Key: "general.architecture", ValueType: gguf.GGUFMetadataValueTypeString, Value: "clip"}, + {Key: key, ValueType: valueType, Value: value}, + }, + }, + } + } + + It("detects a projector with an audio encoder", func() { + f := mmproj("clip.has_audio_encoder", gguf.GGUFMetadataValueTypeBool, true) + Expect(HasAudioEncoderProjector(f)).To(BeTrue()) + }) + + It("is false for a vision projector", func() { + f := mmproj("clip.has_vision_encoder", gguf.GGUFMetadataValueTypeBool, true) + Expect(HasAudioEncoderProjector(f)).To(BeFalse()) + }) + + It("is false when the flag is present but off", func() { + f := mmproj("clip.has_audio_encoder", gguf.GGUFMetadataValueTypeBool, false) + Expect(HasAudioEncoderProjector(f)).To(BeFalse()) + }) + + It("is false, not a panic, when the flag carries the wrong type", func() { + f := mmproj("clip.has_audio_encoder", gguf.GGUFMetadataValueTypeString, "true") + Expect(HasAudioEncoderProjector(f)).To(BeFalse()) + }) + + It("is false for a nil file", func() { + Expect(HasAudioEncoderProjector(nil)).To(BeFalse()) + }) +}) + +// Qwen3-TTS sets both encoder flags, so the speech-to-speech check must read +// the encoder family as well as the flags. +var _ = Describe("HasSpeechToSpeechProjector", func() { + projector := func(kvs ...gguf.GGUFMetadataKV) *gguf.GGUFFile { + kvs = append([]gguf.GGUFMetadataKV{{Key: "general.architecture", ValueType: gguf.GGUFMetadataValueTypeString, Value: "clip"}}, kvs...) + return &gguf.GGUFFile{Header: gguf.GGUFHeader{MetadataKV: kvs}} + } + flag := func(key string) gguf.GGUFMetadataKV { + return gguf.GGUFMetadataKV{Key: key, ValueType: gguf.GGUFMetadataValueTypeBool, Value: true} + } + family := func(value string) gguf.GGUFMetadataKV { + return gguf.GGUFMetadataKV{Key: "clip.audio.projector_type", ValueType: gguf.GGUFMetadataValueTypeString, Value: value} + } + + It("detects an LFM3 audio encoder with the gen-audio decoder", func() { + f := projector(flag("clip.has_audio_encoder"), flag("clip.has_gen_audio_encoder"), family("lfm3a")) + Expect(HasSpeechToSpeechProjector(f)).To(BeTrue()) + }) + + It("is false for the Qwen3-TTS speaker encoder with the gen-audio decoder", func() { + f := projector(flag("clip.has_audio_encoder"), flag("clip.has_gen_audio_encoder"), family("qwen3tts_spkenc")) + Expect(HasSpeechToSpeechProjector(f)).To(BeFalse()) + }) + + It("is false for an LFM3 encoder without the gen-audio decoder", func() { + f := projector(flag("clip.has_audio_encoder"), family("lfm3a")) + Expect(HasSpeechToSpeechProjector(f)).To(BeFalse()) + }) + + It("is false for the gen-audio decoder without an audio encoder", func() { + f := projector(flag("clip.has_gen_audio_encoder"), family("lfm3a")) + Expect(HasSpeechToSpeechProjector(f)).To(BeFalse()) + }) + + It("is false, not a panic, when the encoder family has the wrong type", func() { + f := projector(flag("clip.has_audio_encoder"), flag("clip.has_gen_audio_encoder"), + gguf.GGUFMetadataKV{Key: "clip.audio.projector_type", ValueType: gguf.GGUFMetadataValueTypeBool, Value: true}) + Expect(HasSpeechToSpeechProjector(f)).To(BeFalse()) + }) + + It("is false for a nil file", func() { + Expect(HasSpeechToSpeechProjector(nil)).To(BeFalse()) + }) +}) diff --git a/core/config/hardware_defaults.go b/core/config/hardware_defaults.go index dd30fc6d5..e7ce56a14 100644 --- a/core/config/hardware_defaults.go +++ b/core/config/hardware_defaults.go @@ -280,6 +280,10 @@ func hasParallelOption(opts []string) bool { return backendOptionSet(opts, "parallel", "n_parallel") } +// HasParallelOption reports whether opts already carry a parallel or +// n_parallel backend option, whatever its value. +func HasParallelOption(opts []string) bool { return hasParallelOption(opts) } + // localGPU builds a GPU descriptor from local detection, used by SetDefaults on // a single host (the distributed router builds it from the selected node's // reported info instead). It is a package var so tests can inject a diff --git a/core/config/meta/registry.go b/core/config/meta/registry.go index 901c2693a..b5efefeca 100644 --- a/core/config/meta/registry.go +++ b/core/config/meta/registry.go @@ -569,6 +569,13 @@ func DefaultRegistry() map[string]FieldMetaOverride { Component: "toggle", Order: 67, }, + "pipeline.any_to_any": { + Section: "pipeline", + Label: "Speech-to-Speech Model", + Description: "The model serves the whole speech loop itself (AudioToAudioStream), for example LFM3-Audio on llama.cpp. The Realtime session then uses this one model with no separate VAD, transcription, LLM or TTS slots. Leave off to keep the chained pipeline.", + Component: "toggle", + Order: 69, + }, "pipeline.reasoning_effort": { Section: "pipeline", Label: "Reasoning Effort", diff --git a/core/config/model_artifact_inference.go b/core/config/model_artifact_inference.go index 48a3238f8..316d6b83c 100644 --- a/core/config/model_artifact_inference.go +++ b/core/config/model_artifact_inference.go @@ -21,7 +21,7 @@ var managedArtifactBackends = map[string]struct{}{ "transformers": {}, "huggingface-embeddings": {}, "sentencetransformers": {}, "transformers-musicgen": {}, "mamba": {}, "diffusers": {}, "qwen-asr": {}, "fish-speech": {}, "nemo": {}, "voxcpm": {}, "qwen-tts": {}, - "liquid-audio": {}, "vllm": {}, "vllm-omni": {}, "sglang": {}, + "vllm": {}, "vllm-omni": {}, "sglang": {}, "longcat-video": {}, } diff --git a/core/config/model_config.go b/core/config/model_config.go index 25dfab7ee..67ecf78f4 100644 --- a/core/config/model_config.go +++ b/core/config/model_config.go @@ -1166,6 +1166,12 @@ type Pipeline struct { // a diarize request they cannot serve. Speaker labels are per turn. Diarization bool `yaml:"diarization,omitempty" json:"diarization,omitempty"` + // AnyToAny marks a realtime_audio model that serves the whole speech loop + // itself through AudioToAudioStream (for example LFM3-Audio on llama.cpp). + // true routes the session to that single stream; unset or false keeps the + // chained VAD, transcription, LLM and TTS pipeline the four slots describe. + AnyToAny *bool `yaml:"any_to_any,omitempty" json:"any_to_any,omitempty"` + // ReasoningEffort sets the reasoning effort (none|minimal|low|medium|high) for // the pipeline's LLM without editing the LLM model config. Overrides the LLM's // own reasoning_effort. Unset leaves the LLM model config in charge. @@ -1381,6 +1387,10 @@ type PipelineStreaming struct { ClauseChunking *bool `yaml:"clause_chunking,omitempty" json:"clause_chunking,omitempty"` } +// IsAnyToAny reports whether the realtime session runs the whole speech loop +// on one model through AudioToAudioStream (pipeline.any_to_any: true). +func (p Pipeline) IsAnyToAny() bool { return p.AnyToAny != nil && *p.AnyToAny } + // StreamLLM reports whether LLM tokens should be streamed for this pipeline. func (p Pipeline) StreamLLM() bool { return p.Streaming.LLM != nil && *p.Streaming.LLM } @@ -2772,13 +2782,10 @@ func (c *ModelConfig) GuessUsecases(u ModelConfigUsecase) bool { } if (u & FLAG_REALTIME_AUDIO) == FLAG_REALTIME_AUDIO { - // Backends that own a single any-to-any loop and implement - // AudioToAudioStream — listed here so models without an explicit - // known_usecases still surface on the Talk page. - realtimeAudioBackends := []string{"liquid-audio"} - if !slices.Contains(realtimeAudioBackends, c.Backend) { - return false - } + // No heuristic: realtime_audio is a deliberate operator choice. A + // plain GGUF must not appear on the Talk page, so HasUsecases( + // FLAG_REALTIME_AUDIO) is true only when KnownUsecases declares it. + return false } if (u & FLAG_SCORE) == FLAG_SCORE { diff --git a/core/config/model_config_test.go b/core/config/model_config_test.go index 57e5caccb..9b26f36eb 100644 --- a/core/config/model_config_test.go +++ b/core/config/model_config_test.go @@ -1039,3 +1039,30 @@ var _ = Describe("decisions usecase", func() { Expect(reservedNonChatModel(&ModelConfig{Backend: "vllm-cpp", KnownUsecases: declared})).To(BeTrue()) }) }) + +var _ = Describe("realtime_audio and pipeline.any_to_any on llama-cpp", func() { + It("is not inferred from the llama-cpp backend alone", func() { + cfg := ModelConfig{Name: "plain-gguf", Backend: "llama-cpp"} + Expect(cfg.HasUsecases(FLAG_REALTIME_AUDIO)).To(BeFalse()) + }) + + It("is set when known_usecases declares it", func() { + cfg := ModelConfig{Name: "lfm3-audio-2b", Backend: "llama-cpp"} + cfg.KnownUsecaseStrings = []string{"realtime_audio"} + cfg.syncKnownUsecasesFromString() + Expect(cfg.HasUsecases(FLAG_REALTIME_AUDIO)).To(BeTrue()) + }) + + It("reads pipeline.any_to_any from YAML", func() { + var cfg ModelConfig + Expect(yaml.Unmarshal([]byte("name: lfm3-audio-2b\nbackend: llama-cpp\npipeline:\n any_to_any: true\n"), &cfg)).To(Succeed()) + Expect(cfg.Pipeline.AnyToAny).NotTo(BeNil()) + Expect(*cfg.Pipeline.AnyToAny).To(BeTrue()) + }) + + It("leaves pipeline.any_to_any unset when the key is absent", func() { + var cfg ModelConfig + Expect(yaml.Unmarshal([]byte("name: chained\npipeline:\n llm: llama\n"), &cfg)).To(Succeed()) + Expect(cfg.Pipeline.AnyToAny).To(BeNil()) + }) +}) diff --git a/core/gallery/backend_library_test.go b/core/gallery/backend_library_test.go index e72a68cd5..145f1a9e9 100644 --- a/core/gallery/backend_library_test.go +++ b/core/gallery/backend_library_test.go @@ -17,9 +17,9 @@ const ( realBackendLibrary = "../../gallery/backends/index.yaml" backendLibraryGolds = "testdata/backend_library" // The gallery before the library became the source, kept as the golden - // input: 1093 entries, five of them twins of another entry's name. - goldenGalleryEntries = 1093 - goldenGalleryNames = 1088 + // input: 1077 entries, five of them twins of another entry's name. + goldenGalleryEntries = 1077 + goldenGalleryNames = 1072 ) func loadRealBackendLibrary(t *testing.T) []catalog.Release { diff --git a/core/gallery/importers/importers.go b/core/gallery/importers/importers.go index ce1a14055..b66eadd5e 100644 --- a/core/gallery/importers/importers.go +++ b/core/gallery/importers/importers.go @@ -139,8 +139,6 @@ var defaultImporters = []Importer{ // and would otherwise swallow the C++ port's GGUF bundles. &VibeVoiceCppImporter{}, &VibeVoiceImporter{}, - // LiquidAudio (Python) — keep before LlamaCPP so non-GGUF LFM2-Audio repos route here. - &LiquidAudioImporter{}, &CoquiImporter{}, // Image/Video (Batch 3) &StableDiffusionGGMLImporter{}, diff --git a/core/gallery/importers/liquid-audio.go b/core/gallery/importers/liquid-audio.go deleted file mode 100644 index dbea16020..000000000 --- a/core/gallery/importers/liquid-audio.go +++ /dev/null @@ -1,145 +0,0 @@ -package importers - -import ( - "encoding/json" - "path/filepath" - "strings" - - "github.com/mudler/LocalAI/core/config" - "github.com/mudler/LocalAI/core/gallery" - "github.com/mudler/LocalAI/core/schema" - "go.yaml.in/yaml/v2" -) - -var _ Importer = &LiquidAudioImporter{} - -// LiquidAudioImporter recognises LiquidAI's LFM2-Audio family (LFM2-Audio-1.5B, -// LFM2.5-Audio-1.5B, community finetunes) and routes them to the Python -// `liquid-audio` backend. Detection is by repo-name substring so third-party -// mirrors still match. preferences.backend="liquid-audio" overrides detection. -// -// Once upstream llama.cpp PR #18641 lands and the GGUF gallery entries are -// added, GGUF mirrors of these models should route to llama-cpp; that's -// handled by ordering LlamaCPPImporter after this one and by the explicit -// "-gguf" exclusion below. -type LiquidAudioImporter struct{} - -func (i *LiquidAudioImporter) Name() string { return "liquid-audio" } -func (i *LiquidAudioImporter) Modality() string { return "tts" } -func (i *LiquidAudioImporter) AutoDetects() bool { return true } - -func (i *LiquidAudioImporter) Match(details Details) bool { - preferences, err := details.Preferences.MarshalJSON() - if err != nil { - return false - } - preferencesMap := make(map[string]any) - if len(preferences) > 0 { - if err := json.Unmarshal(preferences, &preferencesMap); err != nil { - return false - } - } - - if b, ok := preferencesMap["backend"].(string); ok && b == "liquid-audio" { - return true - } - - matchRepo := func(repo string) bool { - r := strings.ToLower(repo) - // Cede GGUF mirrors to the (later-ordered) llama-cpp importer. - if strings.HasSuffix(r, "-gguf") { - return false - } - return strings.Contains(r, "lfm2-audio") || strings.Contains(r, "lfm2.5-audio") - } - - if details.HuggingFace != nil { - repoName := details.HuggingFace.ModelID - if idx := strings.Index(repoName, "/"); idx >= 0 { - repoName = repoName[idx+1:] - } - if matchRepo(repoName) { - return true - } - } - - if _, repo, ok := HFOwnerRepoFromURI(details.URI); ok { - return matchRepo(repo) - } - return false -} - -func (i *LiquidAudioImporter) Import(details Details) (gallery.ModelConfig, error) { - preferences, err := details.Preferences.MarshalJSON() - if err != nil { - return gallery.ModelConfig{}, err - } - preferencesMap := make(map[string]any) - if len(preferences) > 0 { - if err := json.Unmarshal(preferences, &preferencesMap); err != nil { - return gallery.ModelConfig{}, err - } - } - - name, ok := preferencesMap["name"].(string) - if !ok { - name = filepath.Base(details.URI) - } - - description, ok := preferencesMap["description"].(string) - if !ok { - description = "Imported from " + details.URI - } - - model := details.URI - if details.HuggingFace != nil && details.HuggingFace.ModelID != "" { - model = details.HuggingFace.ModelID - } - - // Preferences may pin the mode (chat / asr / tts / s2s / finetune). - // Default to s2s — the headline any-to-any use case. - mode, _ := preferencesMap["mode"].(string) - if mode == "" { - mode = "s2s" - } - - options := []string{"mode:" + mode} - if voice, ok := preferencesMap["voice"].(string); ok && voice != "" { - options = append(options, "voice:"+voice) - } - - usecases := []string{"chat"} - switch mode { - case "asr": - usecases = []string{"transcript"} - case "tts": - usecases = []string{"tts"} - case "s2s": - // realtime_audio surfaces the model on the Talk page; chat/tts/ - // transcript/vad keep the standalone OpenAI-compatible endpoints - // working since liquid-audio implements all of them. - usecases = []string{"realtime_audio", "chat", "tts", "transcript", "vad"} - } - - modelConfig := config.ModelConfig{ - Name: name, - Description: description, - Backend: "liquid-audio", - KnownUsecaseStrings: usecases, - Options: options, - PredictionOptions: schema.PredictionOptions{ - BasicModelRequest: schema.BasicModelRequest{Model: model}, - }, - } - - data, err := yaml.Marshal(modelConfig) - if err != nil { - return gallery.ModelConfig{}, err - } - - return gallery.ModelConfig{ - Name: name, - Description: description, - ConfigFile: string(data), - }, nil -} diff --git a/core/gallery/importers/liquid-audio_test.go b/core/gallery/importers/liquid-audio_test.go deleted file mode 100644 index f7dc5e046..000000000 --- a/core/gallery/importers/liquid-audio_test.go +++ /dev/null @@ -1,91 +0,0 @@ -package importers_test - -import ( - "encoding/json" - "fmt" - - "github.com/mudler/LocalAI/core/gallery/importers" - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" -) - -var _ = Describe("LiquidAudioImporter", func() { - Context("detection from HuggingFace", func() { - It("matches LiquidAI/LFM2.5-Audio-1.5B", func() { - uri := "https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B" - preferences := json.RawMessage(`{}`) - - modelConfig, err := importers.DiscoverModelConfig(uri, preferences) - - Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err)) - Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: liquid-audio")) - Expect(modelConfig.ConfigFile).To(ContainSubstring("LiquidAI/LFM2.5-Audio-1.5B")) - }) - - It("matches LiquidAI/LFM2-Audio-1.5B (older variant)", func() { - uri := "https://huggingface.co/LiquidAI/LFM2-Audio-1.5B" - preferences := json.RawMessage(`{}`) - - modelConfig, err := importers.DiscoverModelConfig(uri, preferences) - - Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err)) - Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: liquid-audio")) - }) - - It("cedes -GGUF mirrors to the llama-cpp importer", func() { - // LiquidAI/LFM2.5-Audio-1.5B-GGUF should NOT route to liquid-audio. - // Once upstream PR #18641 lands and the GGUF gallery entry exists, - // this is the path that lets users opt into the C++ runtime. - uri := "https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B-GGUF" - preferences := json.RawMessage(`{}`) - - modelConfig, err := importers.DiscoverModelConfig(uri, preferences) - - Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err)) - Expect(modelConfig.ConfigFile).ToNot(ContainSubstring("backend: liquid-audio"), - fmt.Sprintf("GGUF repo should not match Python importer; got: %s", modelConfig.ConfigFile)) - }) - }) - - Context("preference override", func() { - It("honours preferences.backend=liquid-audio for arbitrary URIs", func() { - uri := "https://example.com/some-unrelated-model" - preferences := json.RawMessage(`{"backend": "liquid-audio"}`) - - modelConfig, err := importers.DiscoverModelConfig(uri, preferences) - - Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err)) - Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: liquid-audio")) - }) - - It("picks up the mode preference", func() { - uri := "https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B" - preferences := json.RawMessage(`{"mode": "asr"}`) - - modelConfig, err := importers.DiscoverModelConfig(uri, preferences) - - Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err)) - Expect(modelConfig.ConfigFile).To(ContainSubstring("mode:asr")) - Expect(modelConfig.ConfigFile).To(ContainSubstring("transcript")) - }) - - It("picks up the voice preference", func() { - uri := "https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B" - preferences := json.RawMessage(`{"mode": "tts", "voice": "uk_male"}`) - - modelConfig, err := importers.DiscoverModelConfig(uri, preferences) - - Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err)) - Expect(modelConfig.ConfigFile).To(ContainSubstring("voice:uk_male")) - }) - }) - - Context("Importer interface metadata", func() { - It("exposes name/modality/autodetect", func() { - imp := &importers.LiquidAudioImporter{} - Expect(imp.Name()).To(Equal("liquid-audio")) - Expect(imp.Modality()).To(Equal("tts")) - Expect(imp.AutoDetects()).To(BeTrue()) - }) - }) -}) diff --git a/core/gallery/importers/llama-cpp.go b/core/gallery/importers/llama-cpp.go index c803f8ae2..78d0d55fb 100644 --- a/core/gallery/importers/llama-cpp.go +++ b/core/gallery/importers/llama-cpp.go @@ -467,6 +467,32 @@ func maybeApplyTTSUsecase(modelConfig *config.ModelConfig, cfg *gallery.ModelCon return } + // An LFM3 audio encoder next to the gen-audio decoder is an any-to-any + // model: it listens, answers as speech and transcribes, so it imports as a + // speech-to-speech model served by AudioToAudioStream. Qwen3-TTS carries + // both encoder flags too, which is why the encoder family decides here. + // The parallel slot lets the session and the transcription or TTS calls run + // at once. + if config.HasSpeechToSpeechProjector(f) { + anyToAny := true + modelConfig.KnownUsecaseStrings = []string{ + config.UsecaseRealtimeAudio, config.UsecaseChat, config.UsecaseTranscript, config.UsecaseTTS, + } + modelConfig.Pipeline.AnyToAny = &anyToAny + if !config.HasParallelOption(modelConfig.Options) { + modelConfig.Options = append(modelConfig.Options, "parallel:2") + } + modelConfig.TemplateConfig.UseTokenizerTemplate = true + // The model writes its tool calls in the LFM3 template format. The + // backend parses them; this is how the server reads the raw text if + // the backend found none. + if modelConfig.FunctionsConfig.XMLFormatPreset == "" { + modelConfig.FunctionsConfig.XMLFormatPreset = "lfm3" + } + xlog.Info("[tts-importer] audio encoder and gen-audio projector detected; importing as a speech-to-speech model", "name", modelConfig.Name) + return + } + modelConfig.KnownUsecaseStrings = []string{config.UsecaseTTS} xlog.Info("[tts-importer] gen-audio projector detected; importing as a TTS model", "name", modelConfig.Name) } diff --git a/core/gallery/importers/llama-cpp_audio_test.go b/core/gallery/importers/llama-cpp_audio_test.go new file mode 100644 index 000000000..c27de9de5 --- /dev/null +++ b/core/gallery/importers/llama-cpp_audio_test.go @@ -0,0 +1,157 @@ +package importers + +import ( + "bytes" + "encoding/binary" + "net/http" + "net/http/httptest" + "time" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/gallery" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// ggufMetadata builds a GGUF v3 header with no tensors and the given +// key/value metadata, so the importer's probe can read the mmproj flags over +// a local HTTP server without a real model download. +func ggufMetadata(kvs map[string]any) []byte { + var buf bytes.Buffer + writeString := func(s string) { + _ = binary.Write(&buf, binary.LittleEndian, uint64(len(s))) + buf.WriteString(s) + } + buf.WriteString("GGUF") + _ = binary.Write(&buf, binary.LittleEndian, uint32(3)) + _ = binary.Write(&buf, binary.LittleEndian, uint64(0)) + _ = binary.Write(&buf, binary.LittleEndian, uint64(len(kvs))) + for key, value := range kvs { + writeString(key) + switch v := value.(type) { + case string: + _ = binary.Write(&buf, binary.LittleEndian, uint32(8)) + writeString(v) + case bool: + _ = binary.Write(&buf, binary.LittleEndian, uint32(7)) + var b byte + if v { + b = 1 + } + buf.WriteByte(b) + } + } + return buf.Bytes() +} + +var _ = Describe("maybeApplyTTSUsecase on an mmproj probe", func() { + // serveMMProj serves the synthetic header and returns the URL the import + // would fetch its mmproj from. + serveMMProj := func(kvs map[string]any) string { + // The parser refuses a projector that declares no type, as llama.cpp + // does, so every fixture carries one. + kvs["clip.projector_type"] = "mlp" + payload := ggufMetadata(kvs) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.ServeContent(w, r, "mmproj.gguf", time.Time{}, bytes.NewReader(payload)) + })) + DeferCleanup(server.Close) + return server.URL + "/mmproj.gguf" + } + + // importedConfig is the chat config the llama-cpp importer starts from, + // with the mmproj the import selected. + importedConfig := func(mmprojURL string) (*config.ModelConfig, *gallery.ModelConfig) { + modelConfig := &config.ModelConfig{ + Name: "lfm3-audio-2b", + KnownUsecaseStrings: []string{config.UsecaseChat}, + Options: []string{"use_jinja:true"}, + LLMConfig: config.LLMConfig{MMProj: "mmproj-LFM3-Audio-2B-Q8_0.gguf"}, + } + cfg := &gallery.ModelConfig{ + Files: []gallery.File{{Filename: "mmproj-LFM3-Audio-2B-Q8_0.gguf", URI: mmprojURL}}, + } + return modelConfig, cfg + } + + It("imports an LFM3 speech-to-speech model with audio encoder and gen-audio decoder", func() { + url := serveMMProj(map[string]any{ + "general.architecture": "clip", + "clip.has_audio_encoder": true, + "clip.has_gen_audio_encoder": true, + "clip.audio.projector_type": "lfm3a", + }) + modelConfig, cfg := importedConfig(url) + + maybeApplyTTSUsecase(modelConfig, cfg) + + Expect(modelConfig.KnownUsecaseStrings).To(Equal([]string{ + config.UsecaseRealtimeAudio, config.UsecaseChat, config.UsecaseTranscript, config.UsecaseTTS, + })) + Expect(modelConfig.Pipeline.AnyToAny).NotTo(BeNil()) + Expect(*modelConfig.Pipeline.AnyToAny).To(BeTrue()) + Expect(modelConfig.Options).To(ContainElements("use_jinja:true", "parallel:2")) + Expect(modelConfig.TemplateConfig.UseTokenizerTemplate).To(BeTrue()) + Expect(modelConfig.FunctionsConfig.XMLFormatPreset).To(Equal("lfm3")) + }) + + It("keeps a parallel option the model already carries", func() { + url := serveMMProj(map[string]any{ + "general.architecture": "clip", + "clip.has_audio_encoder": true, + "clip.has_gen_audio_encoder": true, + "clip.audio.projector_type": "lfm3a", + }) + modelConfig, cfg := importedConfig(url) + modelConfig.Options = append(modelConfig.Options, "parallel:4") + + maybeApplyTTSUsecase(modelConfig, cfg) + + Expect(modelConfig.Options).To(ContainElement("parallel:4")) + Expect(modelConfig.Options).NotTo(ContainElement("parallel:2")) + }) + + It("keeps a Qwen3-TTS projector, which also sets both encoder flags, as TTS", func() { + url := serveMMProj(map[string]any{ + "general.architecture": "clip", + "clip.has_audio_encoder": true, + "clip.has_gen_audio_encoder": true, + "clip.audio.projector_type": "qwen3tts_spkenc", + }) + modelConfig, cfg := importedConfig(url) + + maybeApplyTTSUsecase(modelConfig, cfg) + + Expect(modelConfig.KnownUsecaseStrings).To(Equal([]string{config.UsecaseTTS})) + Expect(modelConfig.Pipeline.AnyToAny).To(BeNil()) + Expect(modelConfig.FunctionsConfig.XMLFormatPreset).To(BeEmpty()) + }) + + It("keeps the TTS-only result for a gen-audio projector without an audio encoder", func() { + url := serveMMProj(map[string]any{ + "general.architecture": "clip", + "clip.has_gen_audio_encoder": true, + }) + modelConfig, cfg := importedConfig(url) + + maybeApplyTTSUsecase(modelConfig, cfg) + + Expect(modelConfig.KnownUsecaseStrings).To(Equal([]string{config.UsecaseTTS})) + Expect(modelConfig.Pipeline.AnyToAny).To(BeNil()) + Expect(modelConfig.Options).To(Equal([]string{"use_jinja:true"})) + }) + + It("leaves a vision model with an audio encoder as a chat model", func() { + url := serveMMProj(map[string]any{ + "general.architecture": "clip", + "clip.has_audio_encoder": true, + }) + modelConfig, cfg := importedConfig(url) + + maybeApplyTTSUsecase(modelConfig, cfg) + + Expect(modelConfig.KnownUsecaseStrings).To(Equal([]string{config.UsecaseChat})) + Expect(modelConfig.Pipeline.AnyToAny).To(BeNil()) + }) +}) diff --git a/core/gallery/lfm3_audio_entries_test.go b/core/gallery/lfm3_audio_entries_test.go new file mode 100644 index 000000000..cd72f11d7 --- /dev/null +++ b/core/gallery/lfm3_audio_entries_test.go @@ -0,0 +1,143 @@ +// SPDX-License-Identifier: MIT +package gallery_test + +import ( + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/gallery" +) + +// The LFM3-Audio entries are any-to-any models: the llama-cpp backend serves +// the realtime speech-to-speech session, transcription and TTS from one loaded +// model, so the pipeline flag, the parallel slot and the mmproj must all agree +// with what the importer writes for the same projector. A typo here installs +// cleanly and fails at the first session. +var _ = Describe("gallery/index.yaml LFM3-Audio entries", func() { + const ( + repoResolve = "https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/" + mmprojQ8 = "mmproj-LFM3-Audio-2B-Q8_0.gguf" + mmprojSHA = "26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e" + ) + + byName := func() map[string]gallery.GalleryModel { + entries, err := loadGalleryIndex() + Expect(err).ToNot(HaveOccurred()) + return indexEntriesByName(entries) + } + fileByName := func(e gallery.GalleryModel, name string) (gallery.File, bool) { + for _, f := range e.AdditionalFiles { + if f.Filename == name { + return f, true + } + } + return gallery.File{}, false + } + + type shape struct { + model string // weight file the entry points the backend at + pipeline bool // chained pipeline instead of one speech-to-speech session + vad bool + } + shapes := map[string]shape{ + "lfm3-audio-2b": {model: "LFM3-Audio-2B-Q4_K_M.gguf"}, + "lfm3-audio-2b-q8": {model: "LFM3-Audio-2B-Q8_0.gguf"}, + "lfm3-audio-2b-vad": {model: "LFM3-Audio-2B-Q4_K_M.gguf", vad: true}, + "lfm3-audio-2b-q8-vad": {model: "LFM3-Audio-2B-Q8_0.gguf", vad: true}, + "lfm3-audio-2b-pipeline": {model: "LFM3-Audio-2B-Q4_K_M.gguf", pipeline: true}, + "lfm3-audio-2b-q8-pipeline": {model: "LFM3-Audio-2B-Q8_0.gguf", pipeline: true}, + "lfm3-audio-2b-pipeline-vad": {model: "LFM3-Audio-2B-Q4_K_M.gguf", pipeline: true, vad: true}, + "lfm3-audio-2b-q8-pipeline-vad": {model: "LFM3-Audio-2B-Q8_0.gguf", pipeline: true, vad: true}, + } + + It("declares every any-to-any and chained-pipeline combination of quantization and VAD", func() { + entries := byName() + for name, want := range shapes { + e, ok := entries[name] + Expect(ok).To(BeTrue(), name) + Expect(e.Overrides["backend"]).To(Equal("llama-cpp"), name) + Expect(e.License).To(Equal("lfm1.0"), name) + Expect(e.Overrides["options"]).To(ConsistOf("parallel:2"), name) + Expect(e.Overrides["mmproj"]).To(Equal(mmprojQ8), name) + Expect(e.Overrides["template"]).To(HaveKeyWithValue("use_tokenizer_template", true), name) + Expect(e.Overrides["parameters"]).To(HaveKeyWithValue("model", want.model), name) + Expect(e.Overrides["tts"]).To(HaveKeyWithValue("voice", "female"), name) + // The template writes calls as . + Expect(e.Overrides["function"]).To(HaveKeyWithValue("xml_format_preset", "lfm3"), name) + + pipe, _ := e.Overrides["pipeline"].(map[string]any) + if want.vad { + Expect(pipe).To(HaveKeyWithValue("vad", "silero-vad-ggml"), name) + Expect(strings.Join(strings.Fields(e.Description), " ")).To(ContainSubstring("install the silero-vad-ggml model first"), name) + } else { + Expect(pipe).ToNot(HaveKey("vad"), name) + } + if want.pipeline { + // A chained pipeline is not a realtime_audio model: llama-cpp + // models of that kind must be any-to-any, so the entry names + // itself in each slot instead. + Expect(e.Overrides["known_usecases"]).To(ConsistOf("chat", "transcript", "tts"), name) + Expect(pipe).ToNot(HaveKey("any_to_any"), name) + for _, slot := range []string{"llm", "transcription", "tts"} { + Expect(pipe).To(HaveKeyWithValue(slot, name), name+" "+slot) + } + } else { + Expect(e.Overrides["known_usecases"]).To(ConsistOf("realtime_audio", "chat", "transcript", "tts"), name) + Expect(pipe).To(HaveKeyWithValue("any_to_any", true), name) + Expect(pipe).ToNot(HaveKey("llm"), name) + } + } + }) + + It("points every VAD entry at a VAD model that exists in the gallery", func() { + vad, ok := byName()["silero-vad-ggml"] + Expect(ok).To(BeTrue()) + Expect(vad.Overrides["known_usecases"]).To(ConsistOf("vad")) + }) + + // The gallery has no model-to-model dependencies, so the entries stay + // independent. A variants list on the Q4 entry would let the installer + // pick the larger Q8 build on any host with enough memory, which is not + // what a plain install of lfm3-audio-2b should do. + It("keeps every entry independent so a plain install gets exactly the named build", func() { + entries := byName() + for name := range shapes { + Expect(entries[name].Variants).To(BeEmpty(), name) + } + for _, e := range entries { + for _, v := range e.Variants { + Expect(shapes).ToNot(HaveKey(v.Model), e.Name+" offers "+v.Model+" as a variant") + } + } + // Nothing to select between, whatever memory the host has. + base := entries["lfm3-audio-2b"] + desc, err := gallery.DescribeVariants(nil, &base, gallery.ResolveEnv{AvailableMemory: 1 << 40}) + Expect(err).ToNot(HaveOccurred()) + Expect(desc).To(BeNil()) + }) + + It("installs each weight and projector with the published sha256 from the GGUF repo", func() { + entries := byName() + shaOf := map[string]string{ + "LFM3-Audio-2B-Q4_K_M.gguf": "9a773289bd321ed879cd3a76545f6aca506e403851a45fb315420aee45625be2", + "LFM3-Audio-2B-Q8_0.gguf": "d4e36902ba8f0ee7b333596030a59d650f88ac76660b4d9ba5ff77c3c2258fc0", + } + want := map[string]map[string]string{} + for name, sh := range shapes { + want[name] = map[string]string{sh.model: shaOf[sh.model], mmprojQ8: mmprojSHA} + } + for name, files := range want { + e := entries[name] + Expect(e.AdditionalFiles).To(HaveLen(len(files)), name) + for filename, sha := range files { + f, ok := fileByName(e, filename) + Expect(ok).To(BeTrue(), name+": "+filename) + Expect(f.SHA256).To(Equal(sha), name+": "+filename) + Expect(f.URI).To(Equal(repoResolve+filename), name+": "+filename) + Expect(strings.HasPrefix(f.URI, "https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/")).To(BeTrue(), name) + } + } + }) +}) diff --git a/core/gallery/testdata/backend_library/digests.tsv b/core/gallery/testdata/backend_library/digests.tsv index 1e5c26910..fe90c2754 100644 --- a/core/gallery/testdata/backend_library/digests.tsv +++ b/core/gallery/testdata/backend_library/digests.tsv @@ -63,8 +63,6 @@ cpu-kokoro 869db7ec4fd3f79d5dc63aff55138db9b044876ada65084873b2e077c8682c8f cpu-kokoro-development dae85b0a4d475937f461ebb8a46432539206f761a089bf9635b78ddb8f21ab40 cpu-kokoros 68b9d5982243625b8c6e8230357785bfaeb87ad2b0133b1d974efef7741febdb cpu-kokoros-development 2564819a1a0575408c8fa9c99603a0d442a3c74c35cf4b079fea3a15d97ffc86 -cpu-liquid-audio 7931b099ed7e059ad68948dac71bd82dd58009f50c19ce84bba265cfaf40089b -cpu-liquid-audio-development a3ad83385a16be50062de843780db73a0dadde71a6d8fccc7f41975c5c2e056f cpu-llama-cpp 089fce4f9c5b4caa9d1b53924f00362ea339ab7f477d16fe3c14dce6d97a7f7f cpu-llama-cpp-development 32f1f041c97e729409c11527fa085efd7884eb737ee0360187020807ed1239c7 cpu-llama-cpp-quantization 1984b8cc00b96f14bd2c85b45e9542ace0db0e4a75f5937da684eaddfe383d09 @@ -202,8 +200,6 @@ cuda12-insightface df52e82414d0702e1d76983ea4413b67b769754169fad6eb910b3514da189 cuda12-insightface-development e44107b648d8ca5908a49c7be0cdc5bd418c1297904afab99876fcc6d485ed3b cuda12-kokoro 027f82c0fc30faacb73100795801d021adc0b933b8f1725a798730061acd4ec0 cuda12-kokoro-development 7436124820215d576105b0dd54af86b2afc93f6b8f675881aacf97c4cf8c4a47 -cuda12-liquid-audio 4f5d533c9264994c0a3c3954bca7e0f962d9426bf3a2a1761e2f93dbfadeef22 -cuda12-liquid-audio-development a767514072a8209a70bd051ef96838c185e9f9daf2c974c31aaf7a3d0dd4b6c5 cuda12-llama-cpp 723f951db4648282d5389203cc1112b80f970a164084a50f02969f74e6e7a178 cuda12-llama-cpp-development 4f1875e28f019dd5c700823ced32875e591f04f2d35a996d99946dd56c088d88 cuda12-locate-anything-cpp 734d0cd9f4b52df96f76b2763b48bbacdd85cf8e8f6b2e033cc984f6af8c74f2 @@ -320,8 +316,6 @@ cuda13-funasr 3b3482bd737c962be108e5f8a216227695b5289f7056234261080a12e694bb28 cuda13-funasr-development 3911500e0189f7101c7f0e1fe40727a752967df91bbf879d49344561f7b7558d cuda13-kokoro 5f49b78d6a8f583e7f727451d226318a92d5e092354caf1c1702c7f667a439ec cuda13-kokoro-development 37a0bc04c32560ecbd0774721b5e6e0b69faae6889cf85829600fe2ed370aa42 -cuda13-liquid-audio ed946797b019ba791b9ca9899b3d7c7af62b8064176435ac4144f0f10799585b -cuda13-liquid-audio-development 1ee6e65806fe1f6229ba24b09d265fef6939194702031adceca4a2a53ed02361 cuda13-llama-cpp 0d8350986e2105d6fb837f1e2a812281f8c70f393730bdec91ce341ea7b15984 cuda13-llama-cpp-development e71146b09527c80cd817591ad57e762abbb734591f6b22d7a20d4638324593b4 cuda13-locate-anything-cpp 69ebd49c7f6d60757987c70f01229c6e3851e34b3c56d995fb50abe9ab3e0102 @@ -372,8 +366,6 @@ cuda13-nvidia-l4t-arm64-fish-speech aa82c87424f6750e2b67e9574769ed2e81f878037b11 cuda13-nvidia-l4t-arm64-fish-speech-development 5bb6c84055222b2913455bfe941620c7bdacd238bfbfd40e2538d37eac4068b2 cuda13-nvidia-l4t-arm64-funasr c34d5120bd774eb2d9b86d363a81da20482c94c35f6e74ff64bacba054374f29 cuda13-nvidia-l4t-arm64-funasr-development 2d34783712cc9697762bb94f233787db4ecded6e1b5fcc151e81b341e08dbec9 -cuda13-nvidia-l4t-arm64-liquid-audio 151e6795486e76c00ef117a85ab8ed83fc03bd56cf14531cd74999bbf119d8cb -cuda13-nvidia-l4t-arm64-liquid-audio-development cc252ec4ea3a0a89d0dbd8de0b3dbd7a25f6086488a9741697ce304c74b67b9b cuda13-nvidia-l4t-arm64-llama-cpp 98bbbb68cdc416e6656898ae16096eb52ff3f948f12f3d191f04345a570909e8 cuda13-nvidia-l4t-arm64-llama-cpp-development 0b05ba7fc075464cd4788135917495805c42929e64afc3be55358a3145c8fe7b cuda13-nvidia-l4t-arm64-locate-anything-cpp dffb7c19175541348914c3af43da986950925b1aabcea0a5c529383e4367c149 @@ -520,8 +512,6 @@ intel-funasr d687a1cce470fe1b2184d362ed7ed15f4c5b49080c5a40113b673ffa41733cd1 intel-funasr-development c44626a8a8c4cc1ea575376338d31304ca9b96f0e54cbc7b6eb755998d93a1f3 intel-kokoro 29a5b437b8587d95c41e4dd6e1d960ef227540c8124fdfc9eb1de586ae448c65 intel-kokoro-development f30a3c5589959b72abfd9524453536c094dac2f05d190f20d8c59c92c0043f80 -intel-liquid-audio f4a98da010372594cf5531bbcaf09d39858c311a56ff62847756b514e0a08608 -intel-liquid-audio-development 7b5adc7a64c2330af5dd374f7d12a3a167b3352709d3ec3ea1f2178d6e866eed intel-nemo ad7faba8a39703035ef45e86bd3af5f31e4f24a92a11ec7d5b9ccaeb6b142fc7 intel-nemo-development 923f0f2cdafc2619bf5f41f506bca3165f608ac820c9b57c8e202208c3fe7b49 intel-pocket-tts d9f77d793ff7b8e5ed7f912431650bd0faaf7fa4e293ade7d181b6838dbdac35 @@ -634,8 +624,6 @@ kokoro e82d6204152d0a193ce1237f018d9db36aee161f9c7c0264941c6b8f51b7c6bc kokoro-development 30a26928839ef921a069d055f5c0f750434dada3ab5c0023e2511adaafd03ee1 kokoros 5048779785215bef21bc41ad8d2104522ea9e8a7fe9899e91a5a8f07a6820a88 kokoros-development b36a4c3876f03944cb629a521cc81eb70f836275ae47b2bdbe6a2d819a43e379 -liquid-audio 5aa50adb758b99f4ab4490c7fabaa31a6501927cfc981f92b17f3830e0a9efb4 -liquid-audio-development 09aeb9af2cec89c51596c7053d4dde680bdc0a9e3d0224632e8e65015a52b119 llama-cpp c9a915aa113f55ded561a6de42248cce09f8feb4187e87bcdbdb98c4dfaa62ca llama-cpp-development 3d0b607024d0b3923e272fab513dec0ed567441f136b19fd8918c67fb245ec0d llama-cpp-quantization 0e4ae5fb2a71b700e6f9166b2ef0e5d110f8c4602c940fa1c57d3acdf84205f1 @@ -683,8 +671,6 @@ metal-kitten-tts e5d17f7639a43564a410ed74ad1d440eb3762d2e4727d2943f2b01a26d13f61 metal-kitten-tts-development 74d796c23d7edeb7b37f3ea3128d56d19c28d60349a804cdba7fcff39dfa061e metal-kokoro 98a94376583c1273a0290a943fd43f7b14b6e3682b2b329c044d7b004724fb5e metal-kokoro-development d24022b136abf0d9c690cd0de0965f84617d11e1802442522ec044135f38fb41 -metal-liquid-audio 98c135cfdc7a2b23426a61e49a0c87d265d542bccd99dc5e0c128d41fecade18 -metal-liquid-audio-development bfb9ab63d562a4d2b05c0b9a6b7f904bc551d0280f4b00e39059cc6add537b80 metal-llama-cpp 57bcce78e31ab622f8ed26094372fdd638e5c1593f009016b0565865367eea49 metal-llama-cpp-development cc157d7c83f7c8a9aeae5fd1c23370b16e762793de861a98d9f45d144dad94b7 metal-local-store 64ea22cbdd27026a2ae16ee5f79e4dec42b47dde7360be5e4f9c2d4cb8c79c2c @@ -937,8 +923,6 @@ rocm-funasr 33a464dc6f8d8ff9c7bd8d2b9614d66e9f8f4b5eb88ec026400ed218be0a920c rocm-funasr-development 56f59a140ce06446ee6c2e2c1ee320df3fa1318ee515a847b628f92d94e7d3aa rocm-kokoro e6938fcdf2b7bb7f5656d3fa0ad0241d4775ae0f994eb89d07c1799e09a408e7 rocm-kokoro-development 01d809272335fdc06f687feae8725ab2a2b080453b6263b2e11f854fd76ff660 -rocm-liquid-audio 6a43532475e2d8bb27a069a15e5207db550e00e8624573b1627116ce3b40593b -rocm-liquid-audio-development b3e02e6331d54c8d28da0d9d88eda46655712d65be1785e801f324a05be1c824 rocm-llama-cpp 3769a76cf777fbec39497a70ebb71b21427d9b0e2a461b102378688eca142210 rocm-llama-cpp-development 5779f8d22bfff14cf0ba914b6e94bd16d731e8d0c3be878e3dedb7d0a3b8bb8a rocm-magpie-tts-cpp 867b637ba9e90f0d5f5af2cd57a6473462f064c74559b9217d039c28775ae2a9 diff --git a/core/gallery/testdata/backend_library/gallery-before.yaml b/core/gallery/testdata/backend_library/gallery-before.yaml index d9e75c73c..b044e1ca5 100644 --- a/core/gallery/testdata/backend_library/gallery-before.yaml +++ b/core/gallery/testdata/backend_library/gallery-before.yaml @@ -1762,36 +1762,6 @@ nvidia-l4t-cuda-12: "nvidia-l4t-vibevoice" nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-vibevoice" icon: https://avatars.githubusercontent.com/u/6154722?s=200&v=4 -- &liquid-audio - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - license: LFM-Open-License-v1.0 - name: "liquid-audio" - alias: "liquid-audio" - capabilities: - nvidia: "cuda12-liquid-audio" - intel: "intel-liquid-audio" - amd: "rocm-liquid-audio" - default: "cpu-liquid-audio" - nvidia-cuda-13: "cuda13-liquid-audio" - nvidia-cuda-12: "cuda12-liquid-audio" - nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-liquid-audio" - metal: "metal-liquid-audio" - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - &qwen-tts urls: - https://github.com/QwenLM/Qwen3-TTS @@ -6348,89 +6318,6 @@ uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-vibevoice" mirrors: - localai/localai-backends:master-metal-darwin-arm64-vibevoice -## liquid-audio -- !!merge <<: *liquid-audio - name: "liquid-audio-development" - capabilities: - nvidia: "cuda12-liquid-audio-development" - intel: "intel-liquid-audio-development" - amd: "rocm-liquid-audio-development" - default: "cpu-liquid-audio-development" - nvidia-cuda-13: "cuda13-liquid-audio-development" - nvidia-cuda-12: "cuda12-liquid-audio-development" - nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-liquid-audio-development" - metal: "metal-liquid-audio-development" -- !!merge <<: *liquid-audio - name: "cpu-liquid-audio" - uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-liquid-audio" - mirrors: - - localai/localai-backends:latest-cpu-liquid-audio -- !!merge <<: *liquid-audio - name: "cpu-liquid-audio-development" - uri: "quay.io/go-skynet/local-ai-backends:master-cpu-liquid-audio" - mirrors: - - localai/localai-backends:master-cpu-liquid-audio -- !!merge <<: *liquid-audio - name: "metal-liquid-audio" - uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-liquid-audio" - mirrors: - - localai/localai-backends:latest-metal-darwin-arm64-liquid-audio -- !!merge <<: *liquid-audio - name: "metal-liquid-audio-development" - uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-liquid-audio" - mirrors: - - localai/localai-backends:master-metal-darwin-arm64-liquid-audio -- !!merge <<: *liquid-audio - name: "cuda12-liquid-audio" - uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-liquid-audio" - mirrors: - - localai/localai-backends:latest-gpu-nvidia-cuda-12-liquid-audio -- !!merge <<: *liquid-audio - name: "cuda12-liquid-audio-development" - uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-liquid-audio" - mirrors: - - localai/localai-backends:master-gpu-nvidia-cuda-12-liquid-audio -- !!merge <<: *liquid-audio - name: "cuda13-liquid-audio" - uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-liquid-audio" - mirrors: - - localai/localai-backends:latest-gpu-nvidia-cuda-13-liquid-audio -- !!merge <<: *liquid-audio - name: "cuda13-liquid-audio-development" - uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-liquid-audio" - mirrors: - - localai/localai-backends:master-gpu-nvidia-cuda-13-liquid-audio -- !!merge <<: *liquid-audio - name: "intel-liquid-audio" - uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-liquid-audio" - mirrors: - - localai/localai-backends:latest-gpu-intel-liquid-audio -- !!merge <<: *liquid-audio - name: "intel-liquid-audio-development" - uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-liquid-audio" - mirrors: - - localai/localai-backends:master-gpu-intel-liquid-audio -- !!merge <<: *liquid-audio - name: "rocm-liquid-audio" - uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-liquid-audio" - mirrors: - - localai/localai-backends:latest-gpu-rocm-hipblas-liquid-audio -- !!merge <<: *liquid-audio - name: "rocm-liquid-audio-development" - uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-liquid-audio" - mirrors: - - localai/localai-backends:master-gpu-rocm-hipblas-liquid-audio -- !!merge <<: *liquid-audio - name: "cuda13-nvidia-l4t-arm64-liquid-audio" - uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-liquid-audio" - mirrors: - - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-liquid-audio -- !!merge <<: *liquid-audio - name: "cuda13-nvidia-l4t-arm64-liquid-audio-development" - uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-liquid-audio" - mirrors: - - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-liquid-audio -## qwen-tts - !!merge <<: *qwen-tts name: "qwen-tts-development" capabilities: diff --git a/core/gallery/testdata/backend_library/pins.tsv b/core/gallery/testdata/backend_library/pins.tsv index 50c31b27b..e96f8d7a6 100644 --- a/core/gallery/testdata/backend_library/pins.tsv +++ b/core/gallery/testdata/backend_library/pins.tsv @@ -79,8 +79,6 @@ cpu-kokoro-development linux amd64 cpu-kokoro-development linux arm64 cpu-kokoros linux amd64 sha256:9b6ad2d1b6ca7da7ba437af0470f8459d6321ed15f6adfeaa69d873c4d1d8785 cpu-kokoros-development linux amd64 -cpu-liquid-audio linux amd64 sha256:ad9798b82dae6f2e8d12b077bf1e029c586f44480cea2ff2428671fbbbd4a870 -cpu-liquid-audio-development linux amd64 cpu-llama-cpp linux amd64 sha256:cc0eaf6d6db4c5c22b9e431e7d90b917906a2262add102da65f241b4cfe06bf7 cpu-llama-cpp linux arm64 sha256:de6aa44e96e52a715b55ec6ac48b7bde5711401267817fef3af052872b63935c cpu-llama-cpp-development linux amd64 @@ -286,8 +284,6 @@ cuda12-insightface linux amd64 sha256:771e0ef97bec577a9e73176de2215cd8a832a5b705 cuda12-insightface-development linux amd64 cuda12-kokoro linux amd64 sha256:04b040393fad829bd6816c2a52a8e1620eccd9960e80812f6c7e026014745c34 cuda12-kokoro-development linux amd64 -cuda12-liquid-audio linux amd64 sha256:ca554511dd390a5ffcef6d02164e27f1c22226d51598365bb6126219c0c870b8 -cuda12-liquid-audio-development linux amd64 cuda12-llama-cpp linux amd64 sha256:21800474dce13aa477b392a61a84015d488cb14bc6cbaf7e6c367521f1190ae3 cuda12-llama-cpp-development linux amd64 cuda12-locate-anything-cpp linux amd64 sha256:ce8611790ea22f4807963c2cf6a48556e6e8055888b90a098248169c8d71c181 @@ -404,8 +400,6 @@ cuda13-funasr linux amd64 sha256:01db0d81834756fc1bee6b29c0f7e53cd2e0c943db6e350 cuda13-funasr-development linux amd64 cuda13-kokoro linux amd64 sha256:1577a61547955fbcf47d9065642a0b9170e9079236e7456702ffeedfec39a52b cuda13-kokoro-development linux amd64 -cuda13-liquid-audio linux amd64 sha256:d1ec28a025abf0b7397d5dae2bfd08b5a864dc7c80a87359bd9071bdc8b6b063 -cuda13-liquid-audio-development linux amd64 cuda13-llama-cpp linux amd64 sha256:db9a34f0532badb72551f68c028969bf8bd2b37035d973954d0f47a3e41af034 cuda13-llama-cpp-development linux amd64 cuda13-locate-anything-cpp linux amd64 sha256:9ce5321d47c6f0623d110db6e83327ba0dbbfb0d34658fc0d56689ec9b0bfa66 @@ -456,8 +450,6 @@ cuda13-nvidia-l4t-arm64-fish-speech linux arm64 sha256:d8ed573b1a0622fe7a6eda8b4 cuda13-nvidia-l4t-arm64-fish-speech-development linux arm64 cuda13-nvidia-l4t-arm64-funasr linux arm64 sha256:b9e4659ed178da1ce73af4b65f7cf0d364aef107d5df45019092d895e2b4cad1 cuda13-nvidia-l4t-arm64-funasr-development linux arm64 -cuda13-nvidia-l4t-arm64-liquid-audio linux arm64 sha256:fb521024116a1c2c69fcdc241922a25631c46555e42b3fcf66aa2d23da9506f2 -cuda13-nvidia-l4t-arm64-liquid-audio-development linux arm64 cuda13-nvidia-l4t-arm64-llama-cpp linux arm64 sha256:18824d9c35c313f92c430c816726ce9743809d2b48b03285754b8b490b6ca57d cuda13-nvidia-l4t-arm64-llama-cpp-development linux arm64 cuda13-nvidia-l4t-arm64-locate-anything-cpp linux arm64 sha256:d3017733e7f376f1492361cb795bc8b0facf7546ab9d2c0cd9c9c8b40c5e5807 @@ -582,8 +574,6 @@ intel-funasr linux amd64 sha256:6283ffaf34bf7c3fec112cbf66cea6573cdf3c16f27743a0 intel-funasr-development linux amd64 intel-kokoro linux amd64 sha256:367b1857cbf7d0a5b67fcff482a8e79eb12e139f064bd9be91d2bb587f5e3c8f intel-kokoro-development linux amd64 -intel-liquid-audio linux amd64 sha256:e331eab911829e84d23634483beeef813e0fb334bc58743f95e3a9eba9db360b -intel-liquid-audio-development linux amd64 intel-nemo linux amd64 sha256:7347434ab262b06e0a7159b624f1a3eaf47d0f172c46da6468f481b522d65f32 intel-nemo-development linux amd64 intel-pocket-tts linux amd64 sha256:8140d0f885716d0c0f045b03ef3f01c3366ef131e5466707afc79d4f9def564a @@ -721,8 +711,6 @@ metal-kitten-tts darwin arm64 sha256:2626da6d0f0ba6abd41bf62431459aa5554ceb55d2f metal-kitten-tts-development darwin arm64 metal-kokoro darwin arm64 sha256:c53faaa7724694b9957fbafb79ba9eb830688ead472504944692ec328c44fdac metal-kokoro-development darwin arm64 -metal-liquid-audio darwin arm64 sha256:f744bafbc2026d852d109161c9aad3efd99d5b339fcdea0856d058e1a50e6d78 -metal-liquid-audio-development darwin arm64 metal-llama-cpp darwin arm64 sha256:090580162ce329b460ebb51a40f9c19cc576ef8f446e976bfeef6e5ba63a2b91 metal-llama-cpp-development darwin arm64 metal-local-store darwin arm64 sha256:58e8e057ab81e50a4a2d6d699692c9dd788908a3b4bd12a25119e26b6eca1623 @@ -927,8 +915,6 @@ rocm-funasr linux amd64 sha256:f02fe7f955725d005ae707f56be269980f5ed46cbb067d055 rocm-funasr-development linux amd64 rocm-kokoro linux amd64 sha256:992505e2c68ab49261761708cad4dd0c0ba08d2dec9cb1ee2dd26a307375368a rocm-kokoro-development linux amd64 -rocm-liquid-audio linux amd64 sha256:6fafe2ba12f0810292899fc37f35553ebdada0e073d7b5d7fcde2c8fcc3b50d4 -rocm-liquid-audio-development linux amd64 rocm-llama-cpp linux amd64 sha256:3668d32f62aa63431e1214129b06eb964813c73105be58ff51c2afcff0f12803 rocm-llama-cpp-development linux amd64 rocm-magpie-tts-cpp linux amd64 sha256:b820d07693b65ac905bbae1289bd8ed431d9cb1735ac300d750316660ca86b99 diff --git a/core/gallery/testdata/model_library/digests.tsv b/core/gallery/testdata/model_library/digests.tsv index e403a1bdd..591c35cd2 100644 --- a/core/gallery/testdata/model_library/digests.tsv +++ b/core/gallery/testdata/model_library/digests.tsv @@ -741,10 +741,6 @@ lfm2.5-8b-a1b-turbo-brilliance-q8 2b5e7d14f21d34aa 1de63f4ae75cc4c9 lfm2.5-8b-a1b-ud-q4-k-m 0ac4fb822288834f 263f8313abda67e4 lfm2.5-8b-a1b-ud-q5-k-m b83925a5e3b29a5c 0ff938a9bd62b57b lfm2.5-8b-a1b-ud-q6-k ce01c23fcca939f5 0b77cad30dbf9f37 -lfm2.5-audio-1.5b-asr 74afda871e1aa19a cb76eb7d6450549a -lfm2.5-audio-1.5b-chat 885d4ce952880708 bbebd03a511bdeb5 -lfm2.5-audio-1.5b-realtime 741b42e8456b210f 5cec0853616f6dbf -lfm2.5-audio-1.5b-tts 69fa289e787fc130 737be4078fe19136 lfm2.5-vl-1.6b 479cc1a80370ec8e 4fe05d3586ad5ec3 lfm2.5-vl-1.6b-q8 bd0fdd818daeca7e a4255a36c8befa24 lfm2.5-vl-3b 46353b2ffd227315 f7e2e762a6f52e30 diff --git a/core/http/endpoints/localai/audio_transform.go b/core/http/endpoints/localai/audio_transform.go index db74c43f0..7edc88dfb 100644 --- a/core/http/endpoints/localai/audio_transform.go +++ b/core/http/endpoints/localai/audio_transform.go @@ -253,7 +253,7 @@ func AudioTransformEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, // and on schema.AudioTransformStreamControl. // // @Summary Bidirectional realtime audio transform over WebSocket. -// @Description Streams binary PCM frames in (interleaved stereo: ch0=audio, ch1=reference) and out (mono). The model must support the audio_transform use case. Any-to-any models such as liquid-audio use the OpenAI Realtime API instead. The first message must be a JSON `session.update` envelope describing model + sample format + frame size + backend params. Server emits binary PCM on the same cadence. +// @Description Streams binary PCM frames in (interleaved stereo: ch0=audio, ch1=reference) and out (mono). The model must support the audio_transform use case. Any-to-any realtime_audio models use the OpenAI Realtime API instead. The first message must be a JSON `session.update` envelope describing model + sample format + frame size + backend params. Server emits binary PCM on the same cadence. // @Tags audio // @Router /audio/transformations/stream [get] func AudioTransformStreamEndpoint(app *application.Application) echo.HandlerFunc { diff --git a/core/http/endpoints/localai/audio_transform_internal_test.go b/core/http/endpoints/localai/audio_transform_internal_test.go index 0f831883b..0a21a214c 100644 --- a/core/http/endpoints/localai/audio_transform_internal_test.go +++ b/core/http/endpoints/localai/audio_transform_internal_test.go @@ -129,7 +129,7 @@ var _ = Describe("audio transform stream model validation", func() { It("rejects realtime any-to-any models", func() { usecases := config.FLAG_REALTIME_AUDIO - cfg := &config.ModelConfig{KnownUsecases: &usecases, Backend: "liquid-audio"} + cfg := &config.ModelConfig{KnownUsecases: &usecases, Backend: "llama-cpp"} err := validateAudioTransformStreamModel(cfg) Expect(err).To(MatchError(ContainSubstring("OpenAI Realtime API"))) }) diff --git a/core/http/endpoints/openai/realtime.go b/core/http/endpoints/openai/realtime.go index 6c6099991..532d58aaa 100644 --- a/core/http/endpoints/openai/realtime.go +++ b/core/http/endpoints/openai/realtime.go @@ -152,8 +152,7 @@ type Session struct { // TTS so the client receives only response.output_text.* events. OutputModalities []types.Modality // MaxHistoryItems caps the number of MessageItems passed to the LLM each - // turn (0 = unlimited). Small models — especially the LFM2.5-Audio 1.5B - // served via the liquid-audio backend — degrade quickly past a handful + // turn (0 = unlimited). Small speech models degrade quickly past a handful // of turns. Counted from the tail; FunctionCall + FunctionCallOutput // pairs are kept together so we never feed an orphaned tool result. MaxHistoryItems int @@ -216,6 +215,9 @@ type Session struct { // commitOrderMu guards commitTail AND is the shared ordering boundary // between the two commit producers (see issueCommit). commitOrderMu sync.Mutex + // s2s is the backend stream of an any-to-any session (pipeline.any_to_any); + // nil for chained pipelines. Set before the read loop starts. + s2s *s2sSession // commitTail is the newest commit slot. Slots form a chain that orders // user-item appends in speech order, not transcription-completion order // (see commitSlot). @@ -247,6 +249,9 @@ type commitSlot struct { // conversation. It is written by the commit goroutine before done closes // and must only be read after done is closed. appended bool + // source is who issued the commit. An any-to-any session answers a VAD + // commit itself and waits for response.create after a client commit. + source respcoord.Source } // nextCommitSlot claims the next position in the commit order. It is called @@ -290,6 +295,7 @@ func (session *Session) issueCommit(parent context.Context, source respcoord.Sou session.commitOrderMu.Lock() defer session.commitOrderMu.Unlock() slot := session.nextCommitSlotLocked() + slot.source = source // A non-interrupting VAD commit must not supersede a response at speech end. ifIdle := source == respcoord.SourceVAD && !session.interruptResponseEnabled() session.respSink.issueWithPolicy(parent, source, ifIdle, func(ctx context.Context) { @@ -644,12 +650,20 @@ func prepareRealtimeConfig(cfg *config.ModelConfig) (errCode, errMsg string, ok return "invalid_model", "Model is not a pipeline model", false } - // Self-contained any-to-any models (e.g. liquid-audio) own the whole + // Self-contained any-to-any models own the whole // loop in one engine — surface them by populating empty pipeline slots // with the model's own name so newModel can resolve a config for each // role. The user can still pin individual slots (e.g. Pipeline.VAD = // silero-vad) and those wins. + if cfg.Pipeline.IsAnyToAny() { + return prepareAnyToAnyConfig(cfg) + } if cfg.HasUsecases(config.FLAG_REALTIME_AUDIO) { + // llama.cpp has no VAD, transcription or TTS RPCs, so a model that + // names no pipeline would fail later with UNIMPLEMENTED on every slot. + if config.IsLlamaCppBackend(cfg.Backend) { + return "invalid_pipeline", "a llama-cpp realtime_audio model must set pipeline.any_to_any: true", false + } if cfg.Pipeline.VAD == "" { cfg.Pipeline.VAD = cfg.Name } @@ -744,8 +758,13 @@ func runRealtimeSession(application *application.Application, t Transport, model // defaultTurnDetection seeds server_vad by default, or semantic_vad when the // pipeline opts in (turn_detection.type: semantic_vad); clients can still // override per session via session.update. - turnDetection := defaultTurnDetection(cfg) + // Without a VAD model nothing detects turns, so the client commits them. + turnDetection := defaultSessionTurnDetection(cfg) inputAudioTranscription := &types.AudioTranscription{Model: sttModel} + anyToAny := cfg.Pipeline.IsAnyToAny() + if anyToAny && s2sInputTranscriptionModel(cfg) == "" { + inputAudioTranscription = nil // the stream answers from the audio; see s2sInputTranscriptionModel + } if soundOnly { turnDetection = nil // turn_detection none: no VAD inputAudioTranscription = nil // no transcription stage @@ -756,8 +775,11 @@ func runRealtimeSession(application *application.Application, t Transport, model // session's default voice instructions. Order matches chat.go's // hasSystemMessage check — assistant prompt comes first. instructions := defaultInstructions + if anyToAny { + instructions = s2sDefaultInstructions + } if assistantSystemPrompt != "" { - instructions = assistantSystemPrompt + "\n\n" + defaultInstructions + instructions = assistantSystemPrompt + "\n\n" + instructions } sessionID := generateSessionID() @@ -784,6 +806,10 @@ func runRealtimeSession(application *application.Application, t Transport, model SoundDetectionHopMs: cfg.Pipeline.SoundDetectionHopMs, } session.CompactionEnabled, session.CompactionTrigger, session.MaxSummaryTokens, session.SummaryModel = resolveCompaction(cfg, session.MaxHistoryItems) + if anyToAny { + // The context lives in the backend; there is nothing to summarize here. + session.CompactionEnabled = false + } classifier, err := classifierConfigFromPipeline(cfg.Pipeline.Classifier) if err != nil { sendError(t, "invalid_pipeline", "pipeline classifier: "+err.Error(), "", "") @@ -909,6 +935,15 @@ func runRealtimeSession(application *application.Application, t Transport, model } } + if opener, ok := m.(s2sOpener); ok { + if err := openS2SForSession(session, opener, t); err != nil { + xlog.Error("failed to open the speech-to-speech stream", "error", err) + sendError(t, "model_load_error", "Failed to open the speech-to-speech stream: "+err.Error(), "", "") + return + } + defer session.closeS2S() + } + if session.SummaryModel != "" { summaryModelName := session.SummaryModel sid := sessionID @@ -1070,6 +1105,7 @@ func runRealtimeSession(application *application.Application, t Transport, model } toggleVAD() + session.s2sSyncConfig() sendEvent(t, types.SessionUpdatedEvent{ ServerEventBase: types.ServerEventBase{ @@ -1107,6 +1143,7 @@ func runRealtimeSession(application *application.Application, t Transport, model // Append to InputAudioBuffer session.InputAudioBuffer = append(session.InputAudioBuffer, decodedAudio...) + session.s2sOnAppendLocked(decodedAudio) session.AudioBufferLock.Unlock() case types.InputAudioBufferCommitEvent: @@ -1122,21 +1159,7 @@ func runRealtimeSession(application *application.Application, t Transport, model continue } - session.AudioBufferLock.Lock() - allAudio := make([]byte, len(session.InputAudioBuffer)) - copy(allAudio, session.InputAudioBuffer) - session.InputAudioBuffer = nil - session.AudioBufferLock.Unlock() - - sendEvent(t, types.InputAudioBufferCommittedEvent{ - ServerEventBase: types.ServerEventBase{}, - ItemID: generateItemID(), - }) - - // Issue through the shared commit-ordering boundary and record the - // in-flight commit so the back-to-back response.create waits for the - // STT before the LLM starts (client-commit LLM/STT race). - session.issueClientCommit(allAudio, conversation, t) + commitClientAudio(session, conversation, t) case types.InputAudioBufferClearEvent: xlog.Debug("recv", "message", string(msg)) @@ -1151,6 +1174,14 @@ func runRealtimeSession(application *application.Application, t Transport, model xlog.Debug("recv", "message", string(msg)) // Add the item to the conversation item := e.Item + if session.s2s != nil && !s2sItemSupported(item) { + sendError(t, "invalid_request_error", "this conversation item is "+s2sUnsupportedMessage, "", e.EventID) + continue + } + if item.FunctionCallOutput != nil && session.s2sUserTurnOpen() { + sendError(t, "invalid_request_error", "a tool result cannot be added while the user is speaking; send it after the turn ends", "", e.EventID) + continue + } // Ensure IDs are present if item.User != nil && item.User.ID == "" { item.User.ID = generateItemID() @@ -1182,6 +1213,10 @@ func runRealtimeSession(application *application.Application, t Transport, model case types.ConversationItemDeleteEvent: xlog.Debug("recv", "message", string(msg)) + if session.s2s != nil { + sendError(t, "invalid_request_error", "conversation.item.delete is "+s2sUnsupportedMessage, "", e.EventID) + continue + } if e.ItemID == "" { sendError(t, "invalid_item_id", "Need item_id, but none specified", "", "event_TODO") continue @@ -1201,6 +1236,10 @@ func runRealtimeSession(application *application.Application, t Transport, model case types.ConversationItemTruncateEvent: xlog.Debug("recv", "message", string(msg)) + if session.s2s != nil { + sendError(t, "invalid_request_error", "conversation.item.truncate is "+s2sUnsupportedMessage, "", e.EventID) + continue + } conversation.Lock.Lock() ok := truncateAssistantText(conversation.Items, e.ItemID, e.ContentIndex) conversation.Lock.Unlock() @@ -1273,6 +1312,13 @@ func runRealtimeSession(application *application.Application, t Transport, model conversation.Lock.Unlock() } + // A speech-to-speech session with client-driven turns holds the audio + // back until a commit; a response.create over uncommitted audio + // commits it first, so the model hears what the client sent. + if session.s2s != nil && session.s2sManualTurns() && session.hasBufferedInput() { + commitClientAudio(session, conversation, t) + } + // With client-driven turns the commit and this response.create are // sent back-to-back; the response waits for the in-flight commit's // STT (and user-item append) to finish so the LLM answers the real @@ -1415,19 +1461,33 @@ func updateSession(session *Session, update *types.SessionUnion, cl *config.Mode } rt := update.Realtime + if session.s2s != nil { + if err := checkS2SUpdate(session, rt); err != nil { + return err + } + } explicitVoice := rt.Audio != nil && rt.Audio.Output != nil && rt.Audio.Output.Voice != "" rebuild := rt.Model != "" || explicitVoice || (rt.Audio != nil && rt.Audio.Input != nil && rt.Audio.Input.Transcription != nil) + if session.s2s != nil { + // The model is fixed for the session: a voice or transcription + // language change needs no rebuild. + rebuild = false + if explicitVoice { + session.Voice = string(rt.Audio.Output.Voice) + } + } + candidateModelName := session.Model candidateConfig := session.ModelConfig candidateTranscription := session.InputAudioTranscription - if rt.Model != "" { + if rt.Model != "" && session.s2s == nil { cfg, err := cl.LoadModelConfigFileByNameDefaultOptions(rt.Model, appConfig) if err != nil { return err } - if cfg == nil || (cfg.Pipeline.VAD == "" || cfg.Pipeline.Transcription == "" || cfg.Pipeline.TTS == "" || cfg.Pipeline.LLM == "") { + if cfg == nil || (cfg.Pipeline.Transcription == "" || cfg.Pipeline.TTS == "" || cfg.Pipeline.LLM == "") { return fmt.Errorf("model is not a valid pipeline model: %s", rt.Model) } @@ -1453,11 +1513,18 @@ func updateSession(session *Session, update *types.SessionUnion, cl *config.Mode } } + // A pipeline without a VAD model cannot detect turns: refuse to turn + // detection on, and drop it when the update moves to such a pipeline. + noVAD := session.s2s == nil && candidateConfig.Pipeline.VAD == "" + if noVAD && rt.Audio != nil && rt.Audio.Input != nil && rt.Audio.Input.TurnDetectionSet && turnDetectionActive(rt.Audio.Input.TurnDetection) { + return errors.New("turn detection needs a VAD model; set pipeline.vad on the model, or commit turns manually") + } + candidateModel := session.ModelInterface candidateVoice := session.Voice candidateParams := maps.Clone(session.ttsParams) var candidateRelease func() - selectVoice := rt.Model != "" || explicitVoice + selectVoice := (rt.Model != "" || explicitVoice) && session.s2s == nil if rebuild { m, err := newModel(&candidateConfig.Pipeline, cl, ml, appConfig, evaluator, routing) if err != nil { @@ -1520,6 +1587,9 @@ func updateSession(session *Session, update *types.SessionUnion, cl *config.Mode } } + if noVAD { + session.TurnDetection = nil + } if rt.Audio != nil && rt.Audio.Input != nil && rt.Audio.Input.TurnDetectionSet { session.TurnDetection = rt.Audio.Input.TurnDetection } @@ -1634,6 +1704,7 @@ func decodeOpusLoop(session *Session, opusBackend grpc.Backend, done chan struct newSize := len(session.InputAudioBuffer) + len(pcmBytes) if newSize <= maxAudioBufferSize { session.InputAudioBuffer = append(session.InputAudioBuffer, pcmBytes...) + session.s2sOnAppendLocked(pcmBytes) } session.AudioBufferLock.Unlock() case <-done: @@ -1642,6 +1713,28 @@ func decodeOpusLoop(session *Session, opusBackend grpc.Backend, done chan struct } } +// commitClientAudio commits the buffered input audio on a client's request. On +// a speech-to-speech session the audio goes to the backend only now, ahead of +// the commit, so a cleared buffer never reaches it. +func commitClientAudio(session *Session, conversation *Conversation, t Transport) { + session.AudioBufferLock.Lock() + allAudio := make([]byte, len(session.InputAudioBuffer)) + copy(allAudio, session.InputAudioBuffer) + session.InputAudioBuffer = nil + session.AudioBufferLock.Unlock() + session.s2sCommitAudio(allAudio, conversation) + + sendEvent(t, types.InputAudioBufferCommittedEvent{ + ServerEventBase: types.ServerEventBase{}, + ItemID: generateItemID(), + }) + + // Issue through the shared commit-ordering boundary and record the + // in-flight commit so the back-to-back response.create waits for the + // STT before the LLM starts (client-commit LLM/STT race). + session.issueClientCommit(allAudio, conversation, t) +} + // noSpeechHoldbackSec is how much of the tail of an inspected, segment-free // buffer survives the periodic no-speech clear. It must cover the VAD's // onset-detection latency: a word can already be underway in the newest part @@ -1873,6 +1966,7 @@ func vadTick(sink *turnSink, silenceThreshold float64) { if err := sink.coord.Apply(turncoord.Abort{Reason: turncoord.AbortNoSpeech}); err != nil { xlog.Error("turncoord: abort(no_speech) failed", "error", err) } + session.s2sAbortTurn() } return } @@ -1885,6 +1979,11 @@ func vadTick(sink *turnSink, silenceThreshold float64) { // open" and "speech started" can never disagree. The turn id is minted // here and carried by the coordinator through to the committed event. sink.onsetAudio = aints + if session.s2s != nil { + if _, speaking := sink.coord.State().(turncoord.Speaking); !speaking { + session.s2sBeginTurn(float64(segments[0].Start)) + } + } if err := sink.coord.Apply(turncoord.Onset{Turn: turncoord.TurnID(generateItemID())}); err != nil { xlog.Error("turncoord: onset failed", "error", err) } @@ -1977,6 +2076,7 @@ func vadCommit(sink *turnSink, retranscribe bool, aints []int16, inspectedBytes // of an utterance started right after a commit). session.InputAudioBuffer = dropInspectedPrefix(session.InputAudioBuffer, inspectedBytes, 0) session.AudioBufferLock.Unlock() + session.s2sEndTurn() // Commit the turn through the coordinator: it emits speech_stopped // (EmitSpeechStopped) then the committed event, finalizes the live @@ -2057,6 +2157,14 @@ func commitUtteranceWithTranscript(ctx context.Context, utt []byte, live *liveUt return } + // An any-to-any session needs no transcript to answer: the audio is + // already with the backend. The speaker gate is refused for these models + // at session setup (prepareAnyToAnyConfig). + if session.s2s != nil { + s2sCommitTurn(ctx, utt, itemID, session, conv, t, slot) + return + } + f, err := os.CreateTemp("", "realtime-audio-chunk-*.wav") if err != nil { xlog.Error("failed to create temp file", "error", err) @@ -2660,6 +2768,10 @@ func triggerResponse(ctx context.Context, session *Session, conv *Conversation, } func triggerResponseAtTurn(ctx context.Context, session *Session, conv *Conversation, t Transport, overrides *types.ResponseCreateParams, toolTurn int, r *liveResponse) { + if session.s2s != nil { + s2sResponseTurn(ctx, session, conv, t, overrides, toolTurn, r) + return + } config := session.ModelInterface.PredictConfig() // Default values @@ -2857,6 +2969,12 @@ func triggerResponseAtTurn(ctx context.Context, session *Session, conv *Conversa predFunc, err := session.ModelInterface.Predict(ctx, conversationHistory, images, nil, nil, nil, tools, toolChoice, nil, nil, nil) if err != nil { + if ctx.Err() != nil { + // A barge-in cancelled the response: a cancel has its own terminal. + xlog.Debug("Response cancelled before LLM inference (barge-in)") + r.outcome = outcomeCancelled + return + } sendError(t, "inference_failed", fmt.Sprintf("backend error: %v", err), "", "") // item.Assistant.ID is unknown here r.outcome = outcomeFailed return @@ -2864,6 +2982,14 @@ func triggerResponseAtTurn(ctx context.Context, session *Session, conv *Conversa pred, err := predFunc() if err != nil { + if ctx.Err() != nil { + // The barge-in landed while the model was generating: the call fails + // with the context error. Without this the response ended as failed, + // with an error and no response.done, and the client waited for it. + xlog.Debug("Response cancelled during LLM inference (barge-in)") + r.outcome = outcomeCancelled + return + } sendError(t, "prediction_failed", fmt.Sprintf("backend error: %v", err), "", "") r.outcome = outcomeFailed return diff --git a/core/http/endpoints/openai/realtime_chained_novad_test.go b/core/http/endpoints/openai/realtime_chained_novad_test.go new file mode 100644 index 000000000..978914540 --- /dev/null +++ b/core/http/endpoints/openai/realtime_chained_novad_test.go @@ -0,0 +1,102 @@ +package openai + +import ( + "context" + "os" + "path/filepath" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/http/endpoints/openai/types" + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/pkg/model" + "github.com/mudler/LocalAI/pkg/system" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// A chained pipeline whose four slots all name one llama-cpp model (the shape +// of the lfm3-audio-2b-pipeline gallery entries) must work with and without a +// VAD slot. Without one, nothing detects turns, so the client commits them. +var _ = Describe("chained pipeline on one llama-cpp model", func() { + const selfModel = "name: lfm3-chain\n" + + "backend: llama-cpp\n" + + "parameters:\n model: lfm3.gguf\n" + + "known_usecases: [chat, transcript, tts]\n" + + "pipeline:\n" + + " transcription: lfm3-chain\n llm: lfm3-chain\n tts: lfm3-chain\n disable_warmup: true\n" + + type env struct { + loader *config.ModelConfigLoader + models *model.ModelLoader + appConfig *config.ApplicationConfig + } + setup := func(body string) (*env, *config.ModelConfig) { + dir := GinkgoT().TempDir() + Expect(os.WriteFile(filepath.Join(dir, "lfm3-chain.yaml"), []byte(body), 0o644)).To(Succeed()) + loader := config.NewModelConfigLoader(dir) + Expect(loader.LoadModelConfigsFromPath(dir)).To(Succeed()) + state, err := system.GetSystemState(system.WithModelPath(dir)) + Expect(err).NotTo(HaveOccurred()) + e := &env{loader: loader, models: model.NewModelLoader(state), appConfig: config.NewApplicationConfig(config.WithSystemState(state))} + cfg, err := loader.LoadModelConfigFileByNameDefaultOptions("lfm3-chain", e.appConfig) + Expect(err).NotTo(HaveOccurred()) + return e, cfg + } + withVAD := func(body string) string { + return body + " vad: lfm3-vad\n" + } + + It("is accepted as a pipeline model although it is neither realtime_audio nor any-to-any", func() { + _, cfg := setup(selfModel) + code, msg, ok := prepareRealtimeConfig(cfg) + Expect(ok).To(BeTrue(), code+": "+msg) + Expect(cfg.Pipeline.IsAnyToAny()).To(BeFalse()) + Expect(cfg.Pipeline.LLM).To(Equal("lfm3-chain")) + }) + + It("builds the chained model without a VAD and answers VAD calls with Unimplemented", func() { + e, cfg := setup(selfModel) + m, err := newModel(&cfg.Pipeline, e.loader, e.models, e.appConfig, nil, nil) + Expect(err).NotTo(HaveOccurred()) + w, ok := m.(*wrappedModel) + Expect(ok).To(BeTrue(), "a chained pipeline must not take the any-to-any path") + Expect(w.VADConfig).To(BeNil()) + Expect(w.TranscriptionConfig.Name).To(Equal("lfm3-chain")) + Expect(w.LLMConfig.Name).To(Equal("lfm3-chain")) + Expect(w.TTSConfig.Name).To(Equal("lfm3-chain")) + _, err = m.VAD(context.Background(), &schema.VADRequest{}) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("no VAD model")) + }) + + It("starts without turn detection when no VAD is named and with server_vad when one is", func() { + _, noVAD := setup(selfModel) + Expect(defaultSessionTurnDetection(noVAD)).To(BeNil()) + + _, vad := setup(withVAD(selfModel)) + td := defaultSessionTurnDetection(vad) + Expect(td).NotTo(BeNil()) + Expect(td.ServerVad).NotTo(BeNil()) + }) + + It("rejects server_vad on a session update when the pipeline has no VAD", func() { + e, cfg := setup(selfModel) + m, err := newModel(&cfg.Pipeline, e.loader, e.models, e.appConfig, nil, nil) + Expect(err).NotTo(HaveOccurred()) + session := &Session{Model: "lfm3-chain", ModelConfig: cfg, ModelInterface: m} + err = updateSession(session, &types.SessionUnion{Realtime: &types.RealtimeSession{ + Audio: &types.RealtimeSessionAudio{Input: &types.SessionAudioInput{ + TurnDetectionSet: true, + TurnDetection: &types.TurnDetectionUnion{ServerVad: &types.ServerVad{}}, + }}, + }}, e.loader, e.models, e.appConfig, nil, nil, nil) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("VAD")) + Expect(session.TurnDetection).To(BeNil()) + + // Turning detection off stays allowed. + Expect(updateSession(session, &types.SessionUnion{Realtime: &types.RealtimeSession{ + Audio: &types.RealtimeSessionAudio{Input: &types.SessionAudioInput{TurnDetectionSet: true}}, + }}, e.loader, e.models, e.appConfig, nil, nil, nil)).To(Succeed()) + }) +}) diff --git a/core/http/endpoints/openai/realtime_conncoord.go b/core/http/endpoints/openai/realtime_conncoord.go index d94c6132e..1dddeb364 100644 --- a/core/http/endpoints/openai/realtime_conncoord.go +++ b/core/http/endpoints/openai/realtime_conncoord.go @@ -87,6 +87,8 @@ func (s *connSink) Perform(e conncoord.Effect) { xlog.Debug("Stopping VAD goroutine...") close(s.vadDone) s.vadDone = nil + // A turn the VAD left open must not keep forwarding into manual mode. + s.session.s2sStopVAD() case conncoord.Teardown: // Tear down in dependency order, driving every child machine to its // terminal state so none outlives the session (the hierarchy invariant in diff --git a/core/http/endpoints/openai/realtime_doubles_test.go b/core/http/endpoints/openai/realtime_doubles_test.go index 2a1c87cb6..93ec6b596 100644 --- a/core/http/endpoints/openai/realtime_doubles_test.go +++ b/core/http/endpoints/openai/realtime_doubles_test.go @@ -3,6 +3,7 @@ package openai import ( "context" "strings" + "os" "sync" "github.com/mudler/LocalAI/core/backend" @@ -88,6 +89,9 @@ type fakeModel struct { transcribeDeltas []string transcribeFinal *schema.TranscriptionResult transcribeErr error + // transcribeBlock, when set, holds every transcription until it closes or + // the call's context ends. + transcribeBlock chan struct{} lastDiarize bool // diarize flag of the last Transcribe/TranscribeStream call // TranscribeLive scripting: liveErr makes the open fail (degrade path); @@ -112,6 +116,13 @@ type fakeModel struct { predictChunkDeltas [][]*proto.ChatDelta predictResp backend.LLMResponse predictErr error + // transcribedWAV is the content of each file handed to Transcribe. + transcribedMu sync.Mutex + transcribedWAV [][]byte + // predictUntilCancel makes the deferred predict function wait for the + // context of Predict and fail with its error, like a backend call that a + // barge-in cancels while the model is still generating. + predictUntilCancel bool // ClassifyTurn scripting: classifyScores is returned as the option // distribution (in option order); classifyErr fails the call. @@ -190,7 +201,28 @@ func (m *fakeModel) VAD(_ context.Context, req *schema.VADRequest) (*schema.VADR return &schema.VADResponse{Segments: m.vadSegments}, nil } -func (m *fakeModel) Transcribe(_ context.Context, _, _ string, _, diarize bool, _ string) (*schema.TranscriptionResult, error) { +func (m *fakeModel) waitTranscribeBlock(ctx context.Context) error { + if m.transcribeBlock == nil { + return nil + } + select { + case <-m.transcribeBlock: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} + +func (m *fakeModel) Transcribe(ctx context.Context, audioPath, _ string, _, diarize bool, _ string) (*schema.TranscriptionResult, error) { + if err := m.waitTranscribeBlock(ctx); err != nil { + return nil, err + } + // keep what the transcription was handed: the file is removed after the call + if raw, err := os.ReadFile(audioPath); err == nil { + m.transcribedMu.Lock() + m.transcribedWAV = append(m.transcribedWAV, raw) + m.transcribedMu.Unlock() + } m.lastDiarize = diarize return m.transcribeFinal, m.transcribeErr } @@ -205,12 +237,16 @@ func (m *fakeModel) SoundDetection(ctx context.Context, _ string, _ int, _ float return m.soundDetectionResult, nil } -func (m *fakeModel) Predict(_ context.Context, msgs schema.Messages, _, _, _ []string, cb func(string, backend.TokenUsage) bool, _ []types.ToolUnion, _ *types.ToolChoiceUnion, _, _ *int, _ map[string]float64) (func() (backend.LLMResponse, error), error) { +func (m *fakeModel) Predict(ctx context.Context, msgs schema.Messages, _, _, _ []string, cb func(string, backend.TokenUsage) bool, _ []types.ToolUnion, _ *types.ToolChoiceUnion, _, _ *int, _ map[string]float64) (func() (backend.LLMResponse, error), error) { m.lastMessages = msgs if m.predictErr != nil { return nil, m.predictErr } return func() (backend.LLMResponse, error) { + if m.predictUntilCancel { + <-ctx.Done() + return backend.LLMResponse{}, ctx.Err() + } for i, tok := range m.predictTokens { if cb == nil { continue @@ -241,7 +277,10 @@ func (m *fakeModel) TTSStream(_ context.Context, _, _, _ string, onAudio func(pc return nil } -func (m *fakeModel) TranscribeStream(_ context.Context, _, _ string, _, diarize bool, _ string, onDelta func(text string)) (*schema.TranscriptionResult, error) { +func (m *fakeModel) TranscribeStream(ctx context.Context, _, _ string, _, diarize bool, _ string, onDelta func(text string)) (*schema.TranscriptionResult, error) { + if err := m.waitTranscribeBlock(ctx); err != nil { + return nil, err + } m.lastDiarize = diarize for _, d := range m.transcribeDeltas { onDelta(d) diff --git a/core/http/endpoints/openai/realtime_gate_test.go b/core/http/endpoints/openai/realtime_gate_test.go index 0b86e7f1f..f7f564f27 100644 --- a/core/http/endpoints/openai/realtime_gate_test.go +++ b/core/http/endpoints/openai/realtime_gate_test.go @@ -50,15 +50,30 @@ var _ = Describe("prepareRealtimeConfig", func() { It("accepts a self-contained realtime_audio model and self-pipelines empty slots", func() { cfg := &config.ModelConfig{ - Name: "lfm2.5-audio-realtime", + Name: "s2s-realtime", + Backend: "custom-engine", KnownUsecases: withUsecases(config.FLAG_REALTIME_AUDIO), } _, _, ok := prepareRealtimeConfig(cfg) Expect(ok).To(BeTrue()) - Expect(cfg.Pipeline.VAD).To(Equal("lfm2.5-audio-realtime")) - Expect(cfg.Pipeline.Transcription).To(Equal("lfm2.5-audio-realtime")) - Expect(cfg.Pipeline.LLM).To(Equal("lfm2.5-audio-realtime")) - Expect(cfg.Pipeline.TTS).To(Equal("lfm2.5-audio-realtime")) + Expect(cfg.Pipeline.VAD).To(Equal("s2s-realtime")) + Expect(cfg.Pipeline.Transcription).To(Equal("s2s-realtime")) + Expect(cfg.Pipeline.LLM).To(Equal("s2s-realtime")) + Expect(cfg.Pipeline.TTS).To(Equal("s2s-realtime")) + }) + + It("rejects a llama-cpp realtime_audio model that lacks pipeline.any_to_any", func() { + // llama-cpp has no VAD, transcription or TTS RPCs: self-pipelining + // the slots would fail later with UNIMPLEMENTED. + cfg := &config.ModelConfig{ + Name: "lfm3", + Backend: "llama-cpp", + KnownUsecases: withUsecases(config.FLAG_REALTIME_AUDIO), + } + code, msg, ok := prepareRealtimeConfig(cfg) + Expect(ok).To(BeFalse()) + Expect(code).To(Equal("invalid_pipeline")) + Expect(msg).To(ContainSubstring("pipeline.any_to_any")) }) It("preserves user-pinned pipeline slots on a realtime_audio model", func() { @@ -66,6 +81,7 @@ var _ = Describe("prepareRealtimeConfig", func() { // model own only STT/LLM/TTS. cfg := &config.ModelConfig{ Name: "lfm-with-external-vad", + Backend: "custom-engine", KnownUsecases: withUsecases(config.FLAG_REALTIME_AUDIO), Pipeline: config.Pipeline{ VAD: "silero-vad", diff --git a/core/http/endpoints/openai/realtime_interrupt_test.go b/core/http/endpoints/openai/realtime_interrupt_test.go index 35b875729..5410d0875 100644 --- a/core/http/endpoints/openai/realtime_interrupt_test.go +++ b/core/http/endpoints/openai/realtime_interrupt_test.go @@ -95,7 +95,7 @@ var _ = Describe("realtime interrupt_response", func() { }) It("applies session updates to interruption and echoes false", func() { - s := &Session{TurnDetection: defaultTurnDetection(nil), ModelConfig: &config.ModelConfig{}} + s := &Session{TurnDetection: defaultTurnDetection(nil), ModelConfig: &config.ModelConfig{Pipeline: config.Pipeline{VAD: "vad"}}} var update types.SessionUpdateEvent Expect(json.Unmarshal([]byte(`{"type":"session.update","session":{"type":"realtime","audio":{"input":{"turn_detection":{"type":"semantic_vad","interrupt_response":false}}}}}`), &update)).To(Succeed()) Expect(updateSession(s, &update.Session, nil, nil, nil, nil, nil, nil)).To(Succeed()) diff --git a/core/http/endpoints/openai/realtime_lfm3_gallery_test.go b/core/http/endpoints/openai/realtime_lfm3_gallery_test.go new file mode 100644 index 000000000..90bc60a62 --- /dev/null +++ b/core/http/endpoints/openai/realtime_lfm3_gallery_test.go @@ -0,0 +1,80 @@ +package openai + +import ( + "os" + "path/filepath" + + "github.com/mudler/LocalAI/core/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" +) + +// Every LFM3-Audio gallery entry must be accepted by the realtime endpoint with +// the pipeline shape it advertises: any-to-any entries open one speech-to-speech +// session, the pipeline entries chain the model through its own slots. The +// overrides are read from the generated gallery and loaded as a model config the +// way an install writes them. +var _ = Describe("LFM3-Audio gallery entries in the realtime endpoint", func() { + type entry struct { + Name string `yaml:"name"` + Overrides map[string]any `yaml:"overrides"` + } + load := func(name string) *config.ModelConfig { + raw, err := os.ReadFile(filepath.Join("..", "..", "..", "..", "gallery", "index.yaml")) + Expect(err).NotTo(HaveOccurred()) + var entries []entry + Expect(yaml.Unmarshal(raw, &entries)).To(Succeed()) + for _, e := range entries { + if e.Name != name { + continue + } + body := map[string]any{"name": name} + for k, v := range e.Overrides { + body[k] = v + } + out, err := yaml.Marshal(body) + Expect(err).NotTo(HaveOccurred()) + cfg := &config.ModelConfig{} + Expect(yaml.Unmarshal(out, cfg)).To(Succeed()) + return cfg + } + Fail("gallery entry not found: " + name) + return nil + } + + DescribeTable("accepts the entry", + func(name string, anyToAny, vad bool) { + cfg := load(name) + code, msg, ok := prepareRealtimeConfig(cfg) + Expect(ok).To(BeTrue(), code+": "+msg) + Expect(cfg.Pipeline.IsAnyToAny()).To(Equal(anyToAny)) + if vad { + Expect(cfg.Pipeline.VAD).To(Equal("silero-vad-ggml")) + Expect(defaultSessionTurnDetection(cfg)).NotTo(BeNil()) + } else { + Expect(cfg.Pipeline.VAD).To(BeEmpty(), "no VAD slot is filled in") + Expect(defaultSessionTurnDetection(cfg)).To(BeNil()) + } + Expect(cfg.Pipeline.LLM).To(Equal(name)) + if anyToAny { + Expect(cfg.Pipeline.Transcription).To(BeEmpty(), "the input is not transcribed by default") + Expect(s2sInputTranscriptionModel(cfg)).To(BeEmpty()) + Expect(cfg.Pipeline.TTS).To(BeEmpty(), "speech comes from the same stream") + } else { + Expect(cfg.Pipeline.Transcription).To(Equal(name)) + Expect(cfg.Pipeline.TTS).To(Equal(name)) + Expect(cfg.KnownUsecases).NotTo(BeNil()) + Expect(*cfg.KnownUsecases & config.FLAG_REALTIME_AUDIO).To(BeZero()) + } + }, + Entry("any-to-any Q4", "lfm3-audio-2b", true, false), + Entry("any-to-any Q8", "lfm3-audio-2b-q8", true, false), + Entry("any-to-any Q4 with VAD", "lfm3-audio-2b-vad", true, true), + Entry("any-to-any Q8 with VAD", "lfm3-audio-2b-q8-vad", true, true), + Entry("pipeline Q4", "lfm3-audio-2b-pipeline", false, false), + Entry("pipeline Q8", "lfm3-audio-2b-q8-pipeline", false, false), + Entry("pipeline Q4 with VAD", "lfm3-audio-2b-pipeline-vad", false, true), + Entry("pipeline Q8 with VAD", "lfm3-audio-2b-q8-pipeline-vad", false, true), + ) +}) diff --git a/core/http/endpoints/openai/realtime_model.go b/core/http/endpoints/openai/realtime_model.go index f8de16dce..82d74a65c 100644 --- a/core/http/endpoints/openai/realtime_model.go +++ b/core/http/endpoints/openai/realtime_model.go @@ -34,8 +34,10 @@ import ( ) var ( - _ Model = new(wrappedModel) - _ Model = new(transcriptOnlyModel) + _ Model = new(wrappedModel) + _ Model = new(transcriptOnlyModel) + _ Model = new(anyToAnyModel) + _ s2sOpener = new(anyToAnyModel) ) // wrappedModel represent a model which does not support Any-to-Any operations @@ -99,18 +101,34 @@ type wrappedModel struct { tuneLLM func(cfg *config.ModelConfig) } -// anyToAnyModel represent a model which supports Any-to-Any operations -// We have to wrap this out as well because we want to load two models one for VAD and one for the actual model. -// In the future there could be models that accept continous audio input only so this design will be useful for that +// anyToAnyModel is a realtime model that serves the whole speech loop itself +// through the backend's AudioToAudioStream RPC (pipeline.any_to_any). Audio in, +// audio, transcripts and tool calls out, on one duplex stream the session opens +// once (see realtime_s2s.go). +// +// The chained-pipeline operations (Predict, TTS, classifier mode) do not exist +// here and return errS2SOnly. VAD and Transcribe delegate to the optional +// pipeline.vad model and to pipeline.transcription (the S2S model itself by +// default): the model's own stream returns the output transcript, but the +// transcript of the user's speech is a separate ASR call on a second slot. type anyToAnyModel struct { + // LLMConfig is the speech-to-speech model. LLMConfig *config.ModelConfig - VADConfig *config.ModelConfig + // VADConfig is nil when no pipeline.vad is configured: the session then + // has no server VAD and the client commits turns itself. + VADConfig *config.ModelConfig + TranscriptionConfig *config.ModelConfig appConfig *config.ApplicationConfig modelLoader *model.ModelLoader confLoader *config.ModelConfigLoader + + stageRouter } +// errS2SOnly marks an operation that a speech-to-speech model does not have. +var errS2SOnly = errors.New("not available for speech-to-speech models") + type transcriptOnlyModel struct { TranscriptionConfig *config.ModelConfig VADConfig *config.ModelConfig @@ -215,7 +233,104 @@ func (m *transcriptOnlyModel) Warmup(ctx context.Context) error { }) } +func (m *anyToAnyModel) VAD(ctx context.Context, request *schema.VADRequest) (*schema.VADResponse, error) { + if m.VADConfig == nil { + return nil, status.Error(codes.Unimplemented, "no VAD model is configured (pipeline.vad)") + } + var res *schema.VADResponse + err := m.stageCall(ctx, config.PipelineStageVAD, m.VADConfig, func(cfg *config.ModelConfig, _ func()) error { + var err error + res, err = backend.VAD(request, ctx, m.modelLoader, m.appConfig, *cfg) + return err + }) + return res, err +} + +func (m *anyToAnyModel) Transcribe(ctx context.Context, audio, language string, translate bool, diarize bool, prompt string) (*schema.TranscriptionResult, error) { + var res *schema.TranscriptionResult + err := m.stageCall(ctx, config.PipelineStageTranscription, m.TranscriptionConfig, func(cfg *config.ModelConfig, _ func()) error { + var err error + res, err = backend.ModelTranscription(ctx, audio, language, translate, diarize, prompt, m.modelLoader, *cfg, m.appConfig) + return err + }) + return res, err +} + +func (m *anyToAnyModel) TranscribeStream(ctx context.Context, audio, language string, translate, diarize bool, prompt string, onDelta func(text string)) (*schema.TranscriptionResult, error) { + var res *schema.TranscriptionResult + err := m.stageCall(ctx, config.PipelineStageTranscription, m.TranscriptionConfig, func(cfg *config.ModelConfig, commit func()) error { + var err error + res, err = transcribeStream(ctx, m.modelLoader, *cfg, m.appConfig, audio, language, translate, diarize, prompt, func(s string) { + commit() + onDelta(s) + }) + return err + }) + return res, err +} + +// TranscribeLive is what semantic_vad needs; a speech-to-speech session does +// not offer that turn-detection mode. +func (m *anyToAnyModel) TranscribeLive(context.Context, string, func(backend.LiveTranscriptionEvent)) (backend.LiveTranscriptionSession, error) { + return nil, fmt.Errorf("live transcription: %w", errS2SOnly) +} + +func (m *anyToAnyModel) SoundDetection(context.Context, string, int, float32) (*schema.SoundClassificationResult, error) { + return nil, fmt.Errorf("sound detection: %w", errS2SOnly) +} + +func (m *anyToAnyModel) Predict(context.Context, schema.Messages, []string, []string, []string, func(string, backend.TokenUsage) bool, []types.ToolUnion, *types.ToolChoiceUnion, *int, *int, map[string]float64) (func() (backend.LLMResponse, error), error) { + return nil, fmt.Errorf("predict: %w", errS2SOnly) +} + +func (m *anyToAnyModel) TTS(context.Context, string, string, string) (string, *proto.Result, error) { + return "", nil, fmt.Errorf("tts: %w", errS2SOnly) +} + +func (m *anyToAnyModel) TTSStream(context.Context, string, string, string, func(pcm []byte, sampleRate int) error) error { + return fmt.Errorf("tts: %w", errS2SOnly) +} + +func (m *anyToAnyModel) ClassifyTurn(context.Context, schema.Messages, []types.ClassifierOption, string) ([]router.LabelScore, error) { + return nil, fmt.Errorf("classifier mode: %w", errS2SOnly) +} + +func (m *anyToAnyModel) FillToolArguments(context.Context, schema.Messages, []types.ClassifierOption, string, *types.ClassifierOption) (string, map[string]string, error) { + return "", nil, fmt.Errorf("classifier mode: %w", errS2SOnly) +} + +func (m *anyToAnyModel) PrewarmClassifier(context.Context, []types.ClassifierOption, string) {} + +func (m *anyToAnyModel) PredictConfig() *config.ModelConfig { return m.LLMConfig } + +// Warmup loads the S2S model and, when they are other models, the VAD and +// transcription models. The transcription stage defaults to the S2S model +// itself, so it is skipped then to avoid loading one model twice at once. +func (m *anyToAnyModel) Warmup(ctx context.Context) error { + stages := []backend.PreloadStage{ + {Role: config.PipelineStageLLM, Cfg: m.LLMConfig}, + {Role: config.PipelineStageVAD, Cfg: m.VADConfig}, + } + if m.TranscriptionConfig != nil && m.TranscriptionConfig.Name != m.LLMConfig.Name { + stages = append(stages, backend.PreloadStage{Role: config.PipelineStageTranscription, Cfg: m.TranscriptionConfig}) + } + return m.warmStages(ctx, m.modelLoader, m.appConfig, stages) +} + +// OpenS2S opens the duplex stream to the backend. ctx bounds the stream's +// lifetime: cancelling it ends the stream. +func (m *anyToAnyModel) OpenS2S(ctx context.Context) (S2SStream, error) { + stream, err := backend.ModelAudioToAudioStream(ctx, m.modelLoader, m.appConfig, *m.LLMConfig) + if err != nil { + return nil, err + } + return stream, nil +} + func (m *wrappedModel) VAD(ctx context.Context, request *schema.VADRequest) (*schema.VADResponse, error) { + if m.VADConfig == nil { + return nil, status.Error(codes.Unimplemented, "no VAD model is configured (pipeline.vad)") + } var res *schema.VADResponse err := m.stageCall(ctx, config.PipelineStageVAD, m.VADConfig, func(cfg *config.ModelConfig, _ func()) error { var err error @@ -1074,17 +1189,26 @@ func newModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model sr := newStageRouter(fm, cl, ml, appConfig) resolveStage := sr.resolveStage - cfgVAD, err := cl.LoadResolvedModelConfig(pipeline.VAD, ml.ModelPath, appConfig.ToConfigLoaderOptions()...) - if err == nil { - cfgVAD, err = resolveStage(config.PipelineStageVAD, cfgVAD) - } - if err != nil { - - return nil, fmt.Errorf("failed to load backend config: %w", err) + if pipeline.IsAnyToAny() { + return newAnyToAnyModel(pipeline, cl, ml, appConfig, sr) } - if valid, _ := cfgVAD.Validate(); !valid { - return nil, fmt.Errorf("failed to validate config: %w", err) + // The VAD slot is optional: without it the client commits each turn. + var cfgVAD *config.ModelConfig + var err error + if pipeline.VAD != "" { + cfgVAD, err = cl.LoadResolvedModelConfig(pipeline.VAD, ml.ModelPath, appConfig.ToConfigLoaderOptions()...) + if err == nil { + cfgVAD, err = resolveStage(config.PipelineStageVAD, cfgVAD) + } + if err != nil { + + return nil, fmt.Errorf("failed to load backend config: %w", err) + } + + if valid, _ := cfgVAD.Validate(); !valid { + return nil, fmt.Errorf("failed to validate config: %w", err) + } } // TODO: Do we always need a transcription model? It can be disabled. Note that any-to-any instruction following models don't transcribe as such, so if transcription is required it is a separate process @@ -1101,25 +1225,6 @@ func newModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model return nil, fmt.Errorf("failed to validate config: %w", err) } - // TODO: Decide when we have a real any-to-any model - // if false { - // - // cfgAnyToAny, err := cl.LoadModelConfigFileByName(pipeline.LLM, ml.ModelPath) - // if err != nil { - // - // return nil, fmt.Errorf("failed to load backend config: %w", err) - // } - // - // if valid, _ := cfgAnyToAny.Validate(); !valid { - // return nil, fmt.Errorf("failed to validate config: %w", err) - // } - // - // return &anyToAnyModel{ - // LLMConfig: cfgAnyToAny, - // VADConfig: cfgVAD, - // }, nil - // } - xlog.Debug("Loading a wrapped model") // Otherwise we want to return a wrapped model, which is a "virtual" model that re-uses other models to perform operations @@ -1225,6 +1330,51 @@ func newModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model return wm, nil } +// newAnyToAnyModel resolves the configs of a speech-to-speech session. The +// model itself is pipeline.llm; transcription defaults to it as well, and the +// VAD stays optional. +func newAnyToAnyModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfig *config.ApplicationConfig, sr stageRouter) (Model, error) { + load := func(stage, name string) (*config.ModelConfig, error) { + cfg, err := cl.LoadResolvedModelConfig(name, ml.ModelPath, appConfig.ToConfigLoaderOptions()...) + if err == nil { + cfg, err = sr.resolveStage(stage, cfg) + } + if err != nil { + return nil, fmt.Errorf("failed to load %s config %q: %w", stage, name, err) + } + if valid, verr := cfg.Validate(); !valid { + return nil, fmt.Errorf("failed to validate %s config %q: %w", stage, name, verr) + } + return cfg, nil + } + + cfgLLM, err := load(config.PipelineStageLLM, pipeline.LLM) + if err != nil { + return nil, err + } + var cfgVAD *config.ModelConfig + if pipeline.VAD != "" { + if cfgVAD, err = load(config.PipelineStageVAD, pipeline.VAD); err != nil { + return nil, err + } + } + cfgSST := cfgLLM + if pipeline.Transcription != "" && pipeline.Transcription != pipeline.LLM { + if cfgSST, err = load(config.PipelineStageTranscription, pipeline.Transcription); err != nil { + return nil, err + } + } + return &anyToAnyModel{ + LLMConfig: cfgLLM, + VADConfig: cfgVAD, + TranscriptionConfig: cfgSST, + confLoader: cl, + modelLoader: ml, + appConfig: appConfig, + stageRouter: sr, + }, nil +} + // liveVoiceOptions selects the registered voices a live session may name speakers // with. It stays empty without a speaker_model or speaker_component, or a voice registry. func liveVoiceOptions(ctx context.Context, registry voicerecognition.Registry, cfg *config.ModelConfig) []backend.LiveOption { diff --git a/core/http/endpoints/openai/realtime_s2s.go b/core/http/endpoints/openai/realtime_s2s.go new file mode 100644 index 000000000..0a50bbfff --- /dev/null +++ b/core/http/endpoints/openai/realtime_s2s.go @@ -0,0 +1,1570 @@ +package openai + +import ( + "context" + "encoding/base64" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "strings" + "sync" + "sync/atomic" + "time" + + "github.com/mudler/LocalAI/core/backend" + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/http/endpoints/openai/respcoord" + "github.com/mudler/LocalAI/core/http/endpoints/openai/types" + "github.com/mudler/LocalAI/pkg/functions" + "github.com/mudler/LocalAI/pkg/grpc/proto" + "github.com/mudler/LocalAI/pkg/sound" + "github.com/mudler/xlog" +) + +// Speech-to-speech (S2S) sessions. +// +// A model with pipeline.any_to_any serves the whole speech loop itself. The +// session opens one AudioToAudioStream to the backend and keeps it for its +// whole life. The backend owns the conversation context (the audio rows are +// prefilled as they arrive), so the endpoint sends only what is new: input +// audio frames, turn boundaries, text and tool-result items, and response +// controls. Everything OpenAI-facing (event names, item ids, VAD, barge-in, +// WebRTC audio) stays in this package. +// +// Event flow of one turn, manual commit: +// +// client input_audio_buffer.append* -> held in the input buffer +// (input_audio_buffer.clear drops it) +// client input_audio_buffer.commit -> backend frames (resampled), then +// input_audio_buffer.commit, +// committed + conversation.item.added +// (+ input transcription, see below) +// client response.create -> backend response.create, then +// response.created, output_item.added, content_part.added, +// response.output_audio.delta* / response.output_audio_transcript.delta*, +// response.output_audio.done, response.output_audio_transcript.done, +// response.content_part.done, response.output_item.done, response.done +// +// With server VAD (only when a VAD model is configured) the VAD goroutine +// drives the same steps: input is forwarded from speech start minus the +// pre-roll, the commit and the response follow speech end. +// +// A tool call arrives as response.function_call_arguments.done and is emitted +// through emitToolCallItems, so its events match the chained path. The client +// answers with conversation.item.create (function_call_output) and +// response.create; the output is forwarded to the backend just before that +// response. +// +// The user's own words are transcribed on the side: the committed audio goes to +// pipeline.transcription and is reported as +// conversation.item.input_audio_transcription.* on the user item. A transcription +// model other than the stream's runs in parallel with the response. The default, +// the same model on a second slot, runs before the response of its turn: both +// share one llama.cpp context, and together they make it rebuild its compute +// buffers on every step. A failure there emits ...failed and never affects the response. + +const ( + // s2sDefaultInstructions is what the model card prescribes for interleaved + // text and audio output; a client's own instructions replace it. + s2sDefaultInstructions = "Respond with interleaved text and audio." + + // s2sDefaultPrerollMs is the audio kept before the VAD's speech start + // when the session does not set prefix_padding_ms. + s2sDefaultPrerollMs = 300 + + s2sOpenTimeout = 60 * time.Second + // s2sFrameChunkMs splits a long forward (the pre-roll plus the buffered + // speech at onset) into messages of this size. + s2sFrameChunkMs = 500 + + // s2sUnsupportedMessage is the answer to editing operations that cannot be + // applied to a context that lives in the backend. + s2sUnsupportedMessage = "not supported for any-to-any models" +) + +// S2SStream is the duplex stream to a speech-to-speech backend. +type S2SStream interface { + Send(*proto.AudioToAudioRequest) error + Recv() (*proto.AudioToAudioResponse, error) + // Close ends the send side and releases the stream. + Close() error +} + +// s2sOpener is implemented by models that can open an S2SStream. +type s2sOpener interface { + OpenS2S(ctx context.Context) (S2SStream, error) +} + +// s2sConfig is the first message of the stream. +type s2sConfig struct { + Instructions string + Voice string + ToolsJSON string + ToolChoice string // "auto", "none" or "required"; empty: the backend default + OutputModalities []string // nil: the model chooses +} + +// s2sWireModalities maps the OpenAI output modalities to what the backend +// takes. "audio" is sent as text plus audio, so a spoken reply has a +// transcript (the OpenAI audio modality includes one). With no explicit choice +// it returns nil and the backend picks the mode of each response from the tools +// it has when the response starts: text plus audio without tools, and with +// tools the model switches between text and audio itself, so it can write a +// tool call instead of speaking it. A mode sent at open would be fixed for the +// whole session and stay in force when tools are added later. +func s2sWireModalities(explicit []types.Modality) []string { + if len(explicit) == 0 { + return nil + } + if modalitiesContainAudio(explicit) { + return []string{"text", "audio"} + } + return []string{"text"} +} + +// s2sSession is the per-session state of the stream. Fields marked +// "AudioBufferLock" are guarded by Session.AudioBufferLock, which also orders +// every frame and the commit on the wire. +type s2sSession struct { + stream S2SStream + cancel context.CancelFunc + + // outbound is the ordered write queue. Frames and controls are written + // by one goroutine, so a caller holding AudioBufferLock never blocks on + // gRPC flow control and the wire order is the enqueue order. + outMu sync.Mutex + outQ []s2sOutbound + outWake chan struct{} + quit chan struct{} + closed atomic.Bool + + // events carries response-scoped events from the reader goroutine to the + // response in flight. dead closes when the stream fails or ends. + events chan *proto.AudioToAudioResponse + created chan struct{} + dead chan struct{} + deadErr error + + // sem allows one response on the stream at a time. A superseded response + // keeps it until the backend acknowledges its cancel. + sem chan struct{} + // active is true from response.create until the backend's response.done; + // the reader drops response-scoped events outside that window. + active atomic.Bool + // cancelSent makes the cancel control idempotent per response. + cancelSent atomic.Bool + + inputRate int + outputRate int + + // forwarding is true while speech of a VAD turn is being forwarded + // (AudioBufferLock). turnHasAudio is true once the turn queued any audio: + // the backend refuses a commit of an empty buffer (AudioBufferLock). + forwarding bool + turnHasAudio bool + + mu sync.Mutex + sentItems map[string]struct{} + last s2sSentConfig + + onError func(errType, message string) + // onDead is called when the stream ends outside a response and the + // session was not closed on purpose (guarded by mu). + onDead func(err error) + closeOnce sync.Once + // onFatal closes the client transport (guarded by mu); fatalOnce runs it + // once. + onFatal func() + fatalOnce sync.Once +} + +func (s *s2sSession) setOnDead(f func(err error)) { + s.mu.Lock() + s.onDead = f + s.mu.Unlock() +} + +func (s *s2sSession) setOnFatal(f func()) { + s.mu.Lock() + s.onFatal = f + s.mu.Unlock() +} + +// s2sOutbound is one queued request. done is nil for a frame, whose failure +// is only logged; a control waits on it. +type s2sOutbound struct { + req *proto.AudioToAudioRequest + done chan error +} + +// s2sCancelGrace bounds the wait for the backend to acknowledge a cancel with +// response.done. Past it the stream is considered stuck. A variable so a spec +// can shorten it. +var s2sCancelGrace = 15 * time.Second + +// s2sEOFGrace bounds the wait for the real status when a Send fails with a +// bare EOF. +const s2sEOFGrace = 2 * time.Second + +// s2sSentConfig remembers what the backend already has, so a session.update +// sends only what changed. +type s2sSentConfig struct { + instructions string + voice string + tools string + toolChoice string + modalities string +} + +func newS2SSession(stream S2SStream, cancel context.CancelFunc, onError func(errType, message string)) *s2sSession { + s := &s2sSession{ + stream: stream, + cancel: cancel, + events: make(chan *proto.AudioToAudioResponse, 64), + created: make(chan struct{}), + dead: make(chan struct{}), + sem: make(chan struct{}, 1), + sentItems: map[string]struct{}{}, + onError: onError, + outWake: make(chan struct{}, 1), + quit: make(chan struct{}), + } + go s.pump() + go s.writer() + return s +} + +// writer is the only caller of stream.Send and of stream.Close (CloseSend): +// gRPC does not allow them to run at the same time. close() cancels the +// stream context, which unblocks a Send that waits on flow control, and the +// writer closes the stream when it exits. +func (s *s2sSession) writer() { + defer func() { _ = s.stream.Close() }() + for { + s.outMu.Lock() + batch := s.outQ + s.outQ = nil + s.outMu.Unlock() + if len(batch) == 0 { + select { + case <-s.outWake: + continue + case <-s.quit: + return + } + } + for i, o := range batch { + if s.closed.Load() { + for _, rest := range batch[i:] { + if rest.done != nil { + rest.done <- errS2SClosed + } + } + break + } + var err error + if o.req != nil { + err = s.sendErr(s.stream.Send(o.req)) + } + if o.done != nil { + o.done <- err + } else if err != nil { + xlog.Warn("s2s: sending to the backend failed", "error", err) + } + } + } +} + +var errS2SClosed = errors.New("the speech-to-speech session is closed") + +// sendErr replaces a bare EOF from Send with the stream's real status. gRPC +// reports a failed call (UNIMPLEMENTED, a full context) as io.EOF on Send and +// puts the status on Recv, which only the pump reads. +func (s *s2sSession) sendErr(err error) error { + if err == nil || !errors.Is(err, io.EOF) { + return err + } + select { + case <-s.dead: + if s.deadErr != nil && !errors.Is(s.deadErr, io.EOF) { + return s.deadErr + } + case <-time.After(s2sEOFGrace): + } + return err +} + +func (s *s2sSession) enqueue(o s2sOutbound) { + s.outMu.Lock() + if s.closed.Load() { + // Nobody writes any more: do not let the queue grow. + s.outMu.Unlock() + if o.done != nil { + o.done <- errS2SClosed + } + return + } + s.outQ = append(s.outQ, o) + s.outMu.Unlock() + select { + case s.outWake <- struct{}{}: + default: + } +} + +// flush waits until everything queued so far has been written. +func (s *s2sSession) flush() { + _ = s.send(nil) +} + +// pump is the only reader of the stream. Setup events are consumed here, +// response events go to the response in flight, and a backend error outside a +// response is reported to the client. +func (s *s2sSession) pump() { + defer close(s.dead) + createdOnce := sync.Once{} + for { + resp, err := s.stream.Recv() + if err != nil { + s.deadErr = err + s.mu.Lock() + onDead := s.onDead + s.mu.Unlock() + if onDead != nil && !s.closed.Load() && !s.active.Load() { + go onDead(err) + } + // A response in flight sees s.dead itself and ends the session. + return + } + switch resp.GetEvent() { + case "session.created": + var meta struct { + InputRate int `json:"input_sample_rate"` + OutputRate int `json:"output_sample_rate"` + } + _ = json.Unmarshal(resp.GetMeta(), &meta) + createdOnce.Do(func() { + s.inputRate, s.outputRate = meta.InputRate, meta.OutputRate + close(s.created) + }) + case "input_audio_buffer.committed": + xlog.Debug("s2s: backend committed the input audio") + default: + if s.active.Load() { + select { + case s.events <- resp: + case <-s.quit: + // Nobody reads the events any more (the turn ended on an + // error or gave up): do not outlive the session. + return + } + continue + } + if resp.GetEvent() == "error" { + typ, msg, _ := s2sErrorMeta(resp.GetMeta()) + if typ == "response_cancel_not_active" { + continue + } + if s.onError != nil { + s.onError(typ, msg) + } + continue + } + xlog.Debug("s2s: dropping an event outside a response", "event", resp.GetEvent()) + } + } +} + +// s2sErrorMeta reads an error event. responseScoped is true when the error +// answers the response.create (or ends the stream) and no response.done follows +// it. Any other error is about something else, such as a refused session.update, +// and a running response goes on. +func s2sErrorMeta(meta []byte) (errType, message string, responseScoped bool) { + var m struct { + Type string `json:"type"` + Message string `json:"message"` + ResponseScoped bool `json:"response_scoped"` + } + _ = json.Unmarshal(meta, &m) + if m.Type == "" { + m.Type = "server_error" + } + if m.Message == "" { + m.Message = "the speech-to-speech backend reported an error" + } + return m.Type, m.Message, m.ResponseScoped +} + +// open sends the config and waits for session.created, which carries the +// sample rates the backend expects and produces. +func (s *s2sSession) open(ctx context.Context, cfg s2sConfig) error { + params := map[string]string{} + if len(cfg.OutputModalities) > 0 { + b, _ := json.Marshal(cfg.OutputModalities) + params["output_modalities"] = string(b) + } + if cfg.ToolChoice != "" { + params["tool_choice"] = cfg.ToolChoice + } + if err := s.send(&proto.AudioToAudioRequest{Payload: &proto.AudioToAudioRequest_Config{Config: &proto.AudioToAudioConfig{ + SystemPrompt: cfg.Instructions, + Voice: cfg.Voice, + Tools: cfg.ToolsJSON, + Params: params, + }}}); err != nil { + return fmt.Errorf("send the stream config: %w", err) + } + select { + case <-s.created: + case <-s.dead: + return fmt.Errorf("the stream ended before session.created: %w", s.deadErr) + case <-ctx.Done(): + return fmt.Errorf("waiting for session.created: %w", ctx.Err()) + } + s.mu.Lock() + s.last = s2sSentConfig{ + instructions: cfg.Instructions, + voice: cfg.Voice, + tools: cfg.ToolsJSON, + toolChoice: cfg.ToolChoice, + modalities: strings.Join(cfg.OutputModalities, ","), + } + s.mu.Unlock() + return nil +} + +// close ends the stream. Safe to call more than once. +func (s *s2sSession) close() { + s.closeOnce.Do(func() { + s.closed.Store(true) + close(s.quit) + if s.cancel != nil { + s.cancel() + } + // Drop what is still queued. The writer closes the stream once it has + // stopped sending. + s.outMu.Lock() + rest := s.outQ + s.outQ = nil + s.outMu.Unlock() + for _, o := range rest { + if o.done != nil { + o.done <- errS2SClosed + } + } + }) +} + +// fatal ends the session after the stream failed during a response: it closes +// the stream and, once, the client transport, so later requests are not +// accepted and no more audio is forwarded to a dead stream. +func (s *s2sSession) fatal() { + s.close() + s.mu.Lock() + f := s.onFatal + s.mu.Unlock() + if f != nil { + s.fatalOnce.Do(f) + } +} + +// send queues req behind everything already queued and waits for the write. A +// nil req only waits for the queue to drain. +func (s *s2sSession) send(req *proto.AudioToAudioRequest) error { + done := make(chan error, 1) + s.enqueue(s2sOutbound{req: req, done: done}) + select { + case err := <-done: + return err + case <-s.quit: + return errS2SClosed + } +} + +// sendAsync queues req without waiting. The caller may hold AudioBufferLock. +func (s *s2sSession) sendAsync(req *proto.AudioToAudioRequest) { + s.enqueue(s2sOutbound{req: req}) +} + +// control sends one control message; payload is JSON-encoded when not nil. +func (s *s2sSession) control(event string, payload any) error { + var raw []byte + if payload != nil { + b, err := json.Marshal(payload) + if err != nil { + return err + } + raw = b + } + return s.send(&proto.AudioToAudioRequest{Payload: &proto.AudioToAudioRequest_Control{ + Control: &proto.AudioToAudioControl{Event: event, Payload: raw}, + }}) +} + +// sendCancel asks the backend to stop the response in flight. It acts at once +// on the backend and is sent at most once per response. It only queues the +// message: with backpressure on the stream the write can take long, and the +// caller has to go on reading events and watching its grace timer. +func (s *s2sSession) sendCancel() { + if !s.cancelSent.CompareAndSwap(false, true) { + return + } + s.sendAsync(&proto.AudioToAudioRequest{Payload: &proto.AudioToAudioRequest_Control{ + Control: &proto.AudioToAudioControl{Event: "response.cancel"}, + }}) +} + +// forwardPCM resamples session-rate PCM to the backend input rate and queues it +// as frames of at most s2sFrameChunkMs. It never blocks on the stream. It +// returns the number of bytes queued. +func (s *s2sSession) forwardPCM(pcm []byte, rate int) int { + if len(pcm) < 2 || s.closed.Load() { + return 0 + } + pcm = pcm[:len(pcm)&^1] + if s.inputRate > 0 && rate > 0 && rate != s.inputRate { + pcm = sound.Int16toBytesLE(sound.ResampleInt16(sound.BytesToInt16sLE(pcm), rate, s.inputRate)) + } + outRate := s.inputRate + if outRate == 0 { + outRate = rate + } + chunk := max(2, outRate*s2sFrameChunkMs/1000*2) + for off := 0; off < len(pcm); off += chunk { + end := min(off+chunk, len(pcm)) + // The slice is not retained past Send, but copy anyway: gRPC may keep + // the message until it is written. + frame := append([]byte(nil), pcm[off:end]...) + s.sendAsync(&proto.AudioToAudioRequest{Payload: &proto.AudioToAudioRequest_Frame{ + Frame: &proto.AudioToAudioFrame{Pcm: frame}, + }}) + } + return len(pcm) +} + +func (s *s2sSession) markSent(id string) { + s.mu.Lock() + s.sentItems[id] = struct{}{} + s.mu.Unlock() +} + +// claim marks id as sent and reports whether the caller is the first to do so. +// The commit handler and a response can both forward pending items. +func (s *s2sSession) claim(id string) bool { + s.mu.Lock() + defer s.mu.Unlock() + if _, ok := s.sentItems[id]; ok { + return false + } + s.sentItems[id] = struct{}{} + return true +} + +// forwardToolResult queues one tool result unless a user turn is open or the +// result was already sent. It reports whether the result was queued. +func (s *s2sSession) forwardToolResult(out *types.MessageItemFunctionCallOutput, id string, audioLock sync.Locker) bool { + audioLock.Lock() + defer audioLock.Unlock() + if s.forwarding || s.turnHasAudio { + return false + } + if !s.claim(id) { + return false + } + payload, err := json.Marshal(map[string]any{ + "type": "function_call_output", + "call_id": out.CallID, + "output": out.Output, + }) + if err != nil { + return false + } + s.sendAsync(&proto.AudioToAudioRequest{Payload: &proto.AudioToAudioRequest_Control{ + Control: &proto.AudioToAudioControl{Event: "conversation.item.create", Payload: payload}, + }}) + return true +} + +// forwardPendingItems sends the client's text messages and tool results that +// the backend has not seen yet. Audio turns are not here: their audio is +// already forwarded and they are marked sent when committed. +// +// The backend refuses a tool result while a user turn is open, so a result +// stays pending until the turn is closed: it is claimed and queued under +// audioLock, the lock that orders the turn's frames, so no turn can open +// between the check and the write. Whatever is still pending goes with the +// next commit or response. +func (s *s2sSession) forwardPendingItems(conv *Conversation, audioLock sync.Locker) { + conv.Lock.Lock() + items := append([]*types.MessageItemUnion(nil), conv.Items...) + conv.Lock.Unlock() + + for _, item := range items { + id := itemID(item) + if id == "" { + continue + } + switch { + case item.FunctionCallOutput != nil: + if !s.forwardToolResult(item.FunctionCallOutput, id, audioLock) { + continue + } + case item.User != nil: + if !s.claim(id) { + continue + } + var text strings.Builder + for _, c := range item.User.Content { + if c.Type == types.MessageContentTypeInputText { + text.WriteString(c.Text) + } + } + if text.Len() == 0 { + continue + } + if err := s.control("conversation.item.create", map[string]any{ + "type": "message", + "role": "user", + "content": []map[string]any{ + {"type": "input_text", "text": text.String()}, + }, + }); err != nil { + xlog.Warn("s2s: forwarding a text message failed", "error", err) + } + } + } +} + +// s2sItemSupported reports whether a client-created conversation item can be +// handed to a speech-to-speech backend: user text and tool results. +func s2sItemSupported(item types.MessageItemUnion) bool { + switch { + case item.FunctionCallOutput != nil: + return true + case item.User != nil: + for _, c := range item.User.Content { + if c.Type == types.MessageContentTypeInputText { + return true + } + } + } + return false +} + +// openS2SForSession opens the backend stream for a new session and attaches it. +// The stream lives until the session ends (closeS2S). +func openS2SForSession(session *Session, opener s2sOpener, t Transport) error { + ctx, cancel := context.WithCancel(context.Background()) + stream, err := opener.OpenS2S(ctx) + if err != nil { + cancel() + return err + } + st := newS2SSession(stream, cancel, func(errType, message string) { + sendError(t, errType, message, "", "") + }) + toolsJSON, _ := realtimeToolsJSON(session.Tools, nil) + octx, ocancel := context.WithTimeout(ctx, s2sOpenTimeout) + defer ocancel() + if err := st.open(octx, s2sConfig{ + Instructions: session.Instructions, + Voice: session.Voice, + ToolsJSON: toolsJSON, + ToolChoice: s2sToolChoiceMode(session.ToolChoice), + OutputModalities: s2sWireModalities(session.OutputModalities), + }); err != nil { + st.close() + return err + } + if toolsJSON != "" { + xlog.Debug("s2s: opened the stream with session tools", "count", s2sToolCount(toolsJSON)) + } + st.setOnDead(func(err error) { + // The stream is gone and every later response would fail the same + // way: tell the client and end the session. + xlog.Error("s2s: the stream ended outside a response", "error", err) + sendError(t, "server_error", fmt.Sprintf("the speech-to-speech stream ended: %v", err), "", "") + _ = t.Close() + }) + st.setOnFatal(func() { _ = t.Close() }) + session.s2s = st + return nil +} + +// closeS2S ends the backend stream. +func (s *Session) closeS2S() { + if s.s2s != nil { + s.s2s.close() + } +} + +// s2sSyncConfig sends the parts of the session config that changed since the +// last send. Before the first turn the backend re-derives its prompt; after it, +// the backend rejects changes it cannot apply and the error reaches the client. +func (s *Session) s2sSyncConfig() { + st := s.s2s + if st == nil { + return + } + sessionLock.Lock() + wireMods := s2sWireModalities(s.OutputModalities) + cur := s2sSentConfig{ + instructions: s.Instructions, + voice: s.Voice, + modalities: strings.Join(wireMods, ","), + } + toolsJSON, _ := realtimeToolsJSON(s.Tools, nil) + cur.tools = toolsJSON + cur.toolChoice = s2sToolChoiceMode(s.ToolChoice) + sessionLock.Unlock() + + st.mu.Lock() + prev := st.last + st.last = cur + st.mu.Unlock() + + payload := map[string]any{} + if cur.instructions != prev.instructions { + payload["instructions"] = cur.instructions + } + if cur.voice != prev.voice { + payload["voice"] = cur.voice + } + if cur.tools != prev.tools { + if cur.tools == "" { + payload["tools"] = json.RawMessage("[]") + } else { + payload["tools"] = json.RawMessage(cur.tools) + } + } + if cur.toolChoice != prev.toolChoice && cur.toolChoice != "" { + payload["tool_choice"] = cur.toolChoice + } + if cur.modalities != prev.modalities && wireMods != nil { + payload["output_modalities"] = wireMods + } + if len(payload) == 0 { + return + } + if _, ok := payload["tools"]; ok { + xlog.Debug("s2s: forwarding the session tools to the backend", "count", s2sToolCount(cur.tools)) + } + if err := st.control("session.update", payload); err != nil { + xlog.Warn("s2s: forwarding session.update failed", "error", err) + } +} + +// s2sToolCount counts the tools of a tools JSON array, for the log. +func s2sToolCount(toolsJSON string) int { + var tools []json.RawMessage + _ = json.Unmarshal([]byte(toolsJSON), &tools) + return len(tools) +} + +// s2sManualTurns reports whether the client, not a VAD, decides when a turn +// ends. Call it before taking AudioBufferLock. +func (s *Session) s2sManualTurns() bool { + sessionLock.Lock() + defer sessionLock.Unlock() + return !turnDetectionActive(s.TurnDetection) +} + +// s2sOnAppendLocked forwards a chunk that was just appended to the input +// buffer while a VAD turn is open. Without a VAD (manual commit) nothing is +// forwarded: the audio stays in the input buffer until the client commits it, +// so input_audio_buffer.clear really discards it and idle audio never reaches +// the model's context. The caller holds AudioBufferLock, which keeps the chunk +// ordered after the pre-roll and before the commit; the write itself is queued +// and does not block. +func (s *Session) s2sOnAppendLocked(chunk []byte) { + st := s.s2s + if st == nil || len(chunk) == 0 { + return + } + if st.forwarding && st.forwardPCM(chunk, s.InputSampleRate) > 0 { + st.turnHasAudio = true + } +} + +// s2sCommitAudio sends a client-committed turn to the backend: first the text +// and tool-result items created so far, so a text item keeps its place before +// the audio committed after it, then the audio, then the commit. Call it +// without AudioBufferLock. +func (s *Session) s2sCommitAudio(audio []byte, conv *Conversation) { + st := s.s2s + if st == nil { + return + } + st.forwardPendingItems(conv, &s.AudioBufferLock) + if st.forwardPCM(audio, s.InputSampleRate) == 0 { + return + } + if err := st.control("input_audio_buffer.commit", nil); err != nil { + xlog.Warn("s2s: committing the input audio failed", "error", err) + } +} + +// hasBufferedInput reports whether uncommitted input audio is waiting. +func (s *Session) hasBufferedInput() bool { + s.AudioBufferLock.Lock() + defer s.AudioBufferLock.Unlock() + return len(s.InputAudioBuffer) > 0 +} + +// s2sUserTurnOpen reports whether a VAD turn is being streamed to the backend. +func (s *Session) s2sUserTurnOpen() bool { + if s.s2s == nil { + return false + } + s.AudioBufferLock.Lock() + defer s.AudioBufferLock.Unlock() + return s.s2s.forwarding +} + +// s2sBeginTurn starts forwarding a VAD turn. speechStartSec is where speech +// begins in the current input buffer. Forwarding starts one pre-roll earlier, +// so the model gets the onset of the first word and not the silence before it. +// When the session interrupts responses and one is running, the cancel control +// goes first: the backend then applies it before the new speech. +func (s *Session) s2sBeginTurn(speechStartSec float64) { + st := s.s2s + if st == nil { + return + } + prerollMs := int64(s2sDefaultPrerollMs) + sessionLock.Lock() + if td := s.TurnDetection; td != nil && td.ServerVad != nil && td.ServerVad.PrefixPaddingMs > 0 { + prerollMs = td.ServerVad.PrefixPaddingMs + } + sessionLock.Unlock() + if s.interruptResponseEnabled() && st.active.Load() { + st.sendCancel() + } + + s.AudioBufferLock.Lock() + defer s.AudioBufferLock.Unlock() + if st.forwarding { + return + } + startSample := int((speechStartSec - float64(prerollMs)/1000) * float64(s.InputSampleRate)) + startByte := max(0, startSample) * 2 + if startByte > len(s.InputAudioBuffer) { + startByte = len(s.InputAudioBuffer) + } + st.turnHasAudio = st.forwardPCM(s.InputAudioBuffer[startByte:], s.InputSampleRate) > 0 + st.forwarding = true +} + +// s2sEndTurn stops forwarding and commits the input audio on the backend. +func (s *Session) s2sEndTurn() { + st := s.s2s + if st == nil { + return + } + s.AudioBufferLock.Lock() + defer s.AudioBufferLock.Unlock() + st.forwarding = false + if !st.turnHasAudio { + // A turn that forwarded nothing has nothing to commit. + return + } + st.turnHasAudio = false + // Queued, not awaited: the lock is held and the queue keeps the order. + st.sendAsync(&proto.AudioToAudioRequest{Payload: &proto.AudioToAudioRequest_Control{ + Control: &proto.AudioToAudioControl{Event: "input_audio_buffer.commit"}, + }}) +} + +// s2sStopVAD ends the turn that was open when the VAD stopped (turn detection +// switched to manual). Without it forwarding would go on and a later client +// commit would send the same audio again. The forwarded audio is committed on +// the backend and dropped from the input buffer, so the manual commit that +// follows carries only audio that arrives after this point. +func (s *Session) s2sStopVAD() { + st := s.s2s + if st == nil { + return + } + s.AudioBufferLock.Lock() + defer s.AudioBufferLock.Unlock() + if !st.forwarding { + return + } + st.forwarding = false + if st.turnHasAudio { + st.turnHasAudio = false + st.sendAsync(&proto.AudioToAudioRequest{Payload: &proto.AudioToAudioRequest_Control{ + Control: &proto.AudioToAudioControl{Event: "input_audio_buffer.commit"}, + }}) + } + s.InputAudioBuffer = nil +} + +// s2sAbortTurn stops forwarding a turn the VAD dropped as noise. Audio already +// forwarded stays in the backend's context. +func (s *Session) s2sAbortTurn() { + st := s.s2s + if st == nil { + return + } + s.AudioBufferLock.Lock() + st.forwarding = false + s.AudioBufferLock.Unlock() +} + +// s2sCommitTurn records a committed user turn: the user item, the input +// transcription on the side, and, for a VAD turn, the response. A client commit +// does not respond; the client sends response.create. The backend commit +// control was already sent in order with the audio (s2sEndTurn). +func s2sCommitTurn(ctx context.Context, utt []byte, itemID string, session *Session, conv *Conversation, t Transport, slot *commitSlot) { + sctx := session.sessionCtx + if sctx == nil { + sctx = context.Background() + } + // Append in speech order, like the chained path. + if slot.prevDone != nil { + select { + case <-slot.prevDone: + case <-sctx.Done(): + return + } + } + if itemID == "" { + itemID = generateItemID() + } + rate := session.InputSampleRate + if slot.source == respcoord.SourceVAD { + rate = localSampleRate + } + + session.s2s.markSent(itemID) + item := types.MessageItemUnion{ + User: &types.MessageItemUser{ + ID: itemID, + Status: types.ItemStatusCompleted, + Content: []types.MessageContentInput{{ + Type: types.MessageContentTypeInputAudio, + Audio: base64.StdEncoding.EncodeToString(utt), + }}, + }, + } + conv.Lock.Lock() + conv.Items = append(conv.Items, &item) + conv.Lock.Unlock() + slot.appended = true + sendEvent(t, types.ConversationItemAddedEvent{Item: item}) + + // The transcript is for the client only: the model already has the audio. + // It runs detached under the session context, so the commit slot is + // released at once. Waiting for it would put the ASR latency in front of + // every reply, and on a backend with few slots a busy ASR could block it. + // + // On the model that holds the stream the transcript must not run while a + // response does: llama.cpp keeps one context for both, and a normal slot and + // the realtime slot ask for different output modes of it, so each loop + // iteration re-reserves the compute buffers (about 175 ms twice per step on + // a GB10) and the reply slows to a few tokens per second. So it runs first + // and in line with the commit: the response of this turn (a VAD turn here, a + // client response.create waits for the commit slot) starts once the + // transcript is out. The transcript event therefore comes before any output + // of that response. A transcription model of its own runs in parallel. + transcribe := session.InputAudioTranscription != nil && !session.TranscriptionOnly + if transcribe { + if s2sTranscribesOnStreamModel(session) { + s2sTranscribeInput(sctx, session, conv, t, itemID, utt, rate) + } else { + go s2sTranscribeInput(sctx, session, conv, t, itemID, utt, rate) + } + } + // A turn whose response context is already cancelled (a newer turn, or a + // response that is not interrupted) commits its item only. + if slot.source == respcoord.SourceVAD && ctx.Err() == nil { + triggerResponse(ctx, session, conv, t, nil) + } +} + +// s2sInputTranscriptionModel is the model that transcribes the input audio of +// an any-to-any session, or "" for none. The stream already holds the audio and +// answers from it, so the transcript is only for the client, and on the model of +// the stream it is costly: llama.cpp serves the transcript request and the +// realtime stream from one context and rebuilds its compute buffers on every +// step while both are active (a GB10 run: 26 to 29 reserves of 175 ms per +// turn, against 0 to 1 on the llama.cpp server). So an any-to-any model +// transcribes the input only when pipeline.transcription asks for it: another +// model runs in parallel with the reply, the model itself (named explicitly) +// runs before the reply. The output transcript always comes from the stream. +func s2sInputTranscriptionModel(cfg *config.ModelConfig) string { + if cfg == nil { + return "" + } + return cfg.Pipeline.Transcription +} + +// s2sTranscribesOnStreamModel tells if the input transcription of the session +// runs on the same model as the speech-to-speech stream, so on the same +// llama.cpp context. +func s2sTranscribesOnStreamModel(session *Session) bool { + cfg := session.ModelConfig + return session.s2s != nil && cfg != nil && cfg.Pipeline.Transcription != "" && cfg.Pipeline.Transcription == cfg.Name +} + +// exclusive runs fn while no response runs on the stream and none starts, and +// reports false when ctx ended first. +func (s *s2sSession) exclusive(ctx context.Context, fn func()) bool { + select { + case s.sem <- struct{}{}: + case <-ctx.Done(): + return false + case <-s.dead: + return false + } + defer func() { <-s.sem }() + fn() + return true +} + +// s2sTranscribeInput transcribes the committed audio of a user turn on the +// side and reports it on the user item. It runs under the session context, so a +// barge-in does not lose it. A failure is reported on the item and never +// reaches the response. +func s2sTranscribeInput(ctx context.Context, session *Session, conv *Conversation, t Transport, itemID string, utt []byte, rate int) { + if s2sTranscribesOnStreamModel(session) { + // One response at a time on the stream, and none while this runs. + session.s2s.exclusive(ctx, func() { s2sTranscribeInputNow(ctx, session, conv, t, itemID, utt, rate) }) + return + } + s2sTranscribeInputNow(ctx, session, conv, t, itemID, utt, rate) +} + +func s2sTranscribeInputNow(ctx context.Context, session *Session, conv *Conversation, t Transport, itemID string, utt []byte, rate int) { + path, err := writeWindowWAV(utt, rate) + if err == nil { + defer func() { _ = os.Remove(path) }() + var transcript string + transcript, err = emitTranscription(ctx, t, session, itemID, path, nil) + if err == nil { + conv.Lock.Lock() + for _, it := range conv.Items { + if it.User != nil && it.User.ID == itemID && len(it.User.Content) > 0 { + it.User.Content[0].Transcript = transcript + } + } + conv.Lock.Unlock() + return + } + } + if ctx.Err() != nil { + return + } + xlog.Warn("s2s: input transcription failed", "error", err) + sendEvent(t, types.ConversationItemInputAudioTranscriptionFailedEvent{ + ServerEventBase: types.ServerEventBase{EventID: "event_TODO"}, + ItemID: itemID, + ContentIndex: 0, + Error: types.Error{ + Type: "server_error", + Code: "transcription_failed", + Message: err.Error(), + }, + }) +} + +const s2sNamedToolChoiceMessage = "an any-to-any model cannot be forced to call one function; use tool_choice auto, none or required" + +// s2sNamedToolChoice tells if tc names a function. The backend knows only the +// modes, so a named function is refused instead of silently keeping the +// previous mode. +func s2sNamedToolChoice(tc *types.ToolChoiceUnion) bool { + return tc != nil && tc.Mode == "" && tc.Function != nil +} + +// s2sToolChoiceMode is the tool_choice the backend understands: the mode. A +// named function cannot be forced on a response, so it maps to nothing. +func s2sToolChoiceMode(tc *types.ToolChoiceUnion) string { + if tc == nil { + return "" + } + switch tc.Mode { + case types.ToolChoiceModeAuto, types.ToolChoiceModeNone, types.ToolChoiceModeRequired: + return string(tc.Mode) + } + return "" +} + +// s2sResponseToolCalls picks the tool calls of a finished response. The calls +// the backend parsed win. Without any, and only if the request had tools, the +// raw text of the response is parsed the way the chained pipeline parses the +// text of its LLM, so a model whose call format the backend does not know still +// works. content is the text around the calls and is meaningful only when +// contentKnown is set: the config captures it, or the call was XML markup that +// could be cut out. content may then be empty (the reply was only a call). +func s2sResponseToolCalls(parsed []functions.FuncCallResults, rawText string, hasTools bool, fc functions.FunctionsConfig) (calls []functions.FuncCallResults, content string, contentKnown, fromText bool) { + if len(parsed) > 0 { + return parsed, "", false, false + } + if !hasTools || rawText == "" { + return nil, "", false, false + } + calls = functions.ParseFunctionCall(functions.CleanupLLMResult(rawText, fc), fc) + if len(calls) == 0 { + return nil, "", false, false + } + if captured := functions.ParseTextContent(rawText, fc); captured != "" { + return calls, captured, true, true + } + // No capture regex (none of the LFM3-Audio gallery entries set one): the + // text left after the call markup is cut out is the content. + if stripped := functions.StripToolCallMarkup(rawText); stripped != strings.TrimSpace(rawText) { + return calls, stripped, true, true + } + return calls, "", false, true +} + +// s2sFallbackToolsAllowed tells if the raw text of a response may be parsed +// for tool calls. The backend knows only the tools of the session, so those are +// the tools of every response. With tool_choice "none" (the response value wins +// over the session value) the backend builds no tool parser and a call the +// model writes anyway is plain text. +func s2sFallbackToolsAllowed(session *Session, overrides *types.ResponseCreateParams) bool { + if len(session.Tools) == 0 { + return false + } + choice := session.ToolChoice + if overrides != nil && overrides.ToolChoice != nil { + choice = overrides.ToolChoice + } + return s2sToolChoiceMode(choice) != string(types.ToolChoiceModeNone) +} + +// s2sTurn is the state of one response while its events arrive. +type s2sTurn struct { + ctx context.Context + session *Session + conv *Conversation + t Transport + r *liveResponse + st *s2sSession + textOnly bool + isWebRTC bool + // hasTools is true if the request had tools; funcCfg is how the model + // writes a call, for parsing the raw text of a response. + hasTools bool + funcCfg functions.FunctionsConfig + + item *types.MessageItemUnion + text strings.Builder + audio []byte + toolCalls []functions.FuncCallResults + // stopped is true once the response is cancelled: nothing more reaches the + // client except the closing events. + stopped bool + // after runs once the response has released the stream (tool calls). + after func() +} + +// ensureItem opens the assistant item on the first audio or text event, so a +// response that is only a tool call has no empty message. +func (tn *s2sTurn) ensureItem() { + if tn.item != nil { + return + } + part := types.MessageContentOutput{Type: types.MessageContentTypeOutputAudio} + if tn.textOnly { + part.Type = types.MessageContentTypeOutputText + } + tn.item = &types.MessageItemUnion{Assistant: &types.MessageItemAssistant{ + ID: generateItemID(), + Status: types.ItemStatusInProgress, + Content: []types.MessageContentOutput{part}, + }} + tn.conv.Lock.Lock() + tn.conv.Items = append(tn.conv.Items, tn.item) + tn.conv.Lock.Unlock() + sendEvent(tn.t, types.ResponseOutputItemAddedEvent{ + ResponseID: tn.r.id, OutputIndex: 0, Item: *tn.item, + }) + sendEvent(tn.t, types.ResponseContentPartAddedEvent{ + ResponseID: tn.r.id, ItemID: tn.item.Assistant.ID, OutputIndex: 0, ContentIndex: 0, Part: part, + }) +} + +func (tn *s2sTurn) onAudio(pcm []byte, rate int) { + // A cancel sent by the VAD at speech onset precedes the context + // cancellation; audio behind it must not reach the client either. + if tn.stopped || tn.st.cancelSent.Load() || len(pcm) == 0 || tn.textOnly { + return + } + tn.ensureItem() + if rate == 0 { + rate = tn.st.outputRate + } + if err := tn.t.SendAudio(tn.ctx, pcm, rate); err != nil { + return + } + if tn.isWebRTC { + return + } + out := pcm + if rate != 0 && rate != tn.session.OutputSampleRate { + out = sound.Int16toBytesLE(sound.ResampleInt16(sound.BytesToInt16sLE(pcm), rate, tn.session.OutputSampleRate)) + } + tn.audio = append(tn.audio, out...) + sendEvent(tn.t, types.ResponseOutputAudioDeltaEvent{ + ResponseID: tn.r.id, ItemID: tn.item.Assistant.ID, OutputIndex: 0, ContentIndex: 0, + Delta: base64.StdEncoding.EncodeToString(out), + }) +} + +func (tn *s2sTurn) onText(delta string) { + if tn.stopped || tn.st.cancelSent.Load() || delta == "" { + return + } + tn.ensureItem() + tn.text.WriteString(delta) + if tn.textOnly { + sendEvent(tn.t, types.ResponseOutputTextDeltaEvent{ + ResponseID: tn.r.id, ItemID: tn.item.Assistant.ID, OutputIndex: 0, ContentIndex: 0, Delta: delta, + }) + return + } + sendEvent(tn.t, types.ResponseOutputAudioTranscriptDeltaEvent{ + ResponseID: tn.r.id, ItemID: tn.item.Assistant.ID, OutputIndex: 0, ContentIndex: 0, Delta: delta, + }) +} + +// closeItem emits the closing events of the assistant item. fallbackText is +// the full text from response.done, used when no delta carried any. A non-nil +// override replaces the streamed text: it is the content once the calls were +// cut out of the raw text. +func (tn *s2sTurn) closeItem(fallbackText string, override *string, incomplete bool) { + if tn.item == nil { + return + } + text := tn.text.String() + if text == "" { + text = fallbackText + } + if override != nil { + text = *override + } + id := tn.item.Assistant.ID + if tn.textOnly { + sendEvent(tn.t, types.ResponseOutputTextDoneEvent{ + ResponseID: tn.r.id, ItemID: id, OutputIndex: 0, ContentIndex: 0, Text: text, + }) + } else { + if !tn.isWebRTC { + sendEvent(tn.t, types.ResponseOutputAudioDoneEvent{ + ResponseID: tn.r.id, ItemID: id, OutputIndex: 0, ContentIndex: 0, + }) + } + sendEvent(tn.t, types.ResponseOutputAudioTranscriptDoneEvent{ + ResponseID: tn.r.id, ItemID: id, OutputIndex: 0, ContentIndex: 0, Transcript: text, + }) + } + + tn.conv.Lock.Lock() + part := &tn.item.Assistant.Content[0] + if tn.textOnly { + part.Text = text + } else { + part.Transcript = text + if !tn.isWebRTC { + part.Audio = base64.StdEncoding.EncodeToString(tn.audio) + } + } + tn.item.Assistant.Status = types.ItemStatusCompleted + if incomplete { + tn.item.Assistant.Status = types.ItemStatusIncomplete + } + partCopy := *part + tn.conv.Lock.Unlock() + + sendEvent(tn.t, types.ResponseContentPartDoneEvent{ + ResponseID: tn.r.id, ItemID: id, OutputIndex: 0, ContentIndex: 0, Part: partCopy, + }) + sendEvent(tn.t, types.ResponseOutputItemDoneEvent{ + ResponseID: tn.r.id, OutputIndex: 0, Item: *tn.item, + }) + tn.r.addItem(*tn.item) +} + +// s2sResponseTurn runs one response on the stream. It is the speech-to-speech +// counterpart of the generation step of triggerResponseAtTurn: it sends +// response.create, turns the backend's events into the realtime events, and +// records the outcome on r. triggerResponse emits response.created and the +// single response.done. +func s2sResponseTurn(ctx context.Context, session *Session, conv *Conversation, t Transport, overrides *types.ResponseCreateParams, toolTurn int, r *liveResponse) { + // The tool calls run after the stream is released: a server-side tool + // triggers a follow-up response on this same stream, which would wait on + // the slot this response still held. + if after := s2sRunTurn(ctx, session, conv, t, overrides, toolTurn, r); after != nil { + after() + } +} + +// s2sRunTurn runs the response while it holds the stream. It returns what is +// left to do once the stream is free again, or nil. +func s2sRunTurn(ctx context.Context, session *Session, conv *Conversation, t Transport, overrides *types.ResponseCreateParams, toolTurn int, r *liveResponse) (after func()) { + st := session.s2s + + if overrides != nil && s2sNamedToolChoice(overrides.ToolChoice) { + sendError(t, "invalid_request_error", s2sNamedToolChoiceMessage, "tool_choice", "") + r.outcome = outcomeFailed + return nil + } + + // One response at a time on the stream. A superseded response holds it + // until the backend acknowledges the cancel. + select { + case st.sem <- struct{}{}: + case <-ctx.Done(): + r.outcome = outcomeCancelled + return nil + } + defer func() { <-st.sem }() + if ctx.Err() != nil { + r.outcome = outcomeCancelled + return nil + } + + st.forwardPendingItems(conv, &session.AudioBufferLock) + + var respMods []types.Modality + maxOutputTokens := session.MaxOutputTokens + if overrides != nil { + respMods = modalitiesWithAlias(overrides.OutputModalities, overrides.Modalities) + if overrides.MaxOutputTokens != 0 { + maxOutputTokens = overrides.MaxOutputTokens + } + } + modalities := resolveOutputModalities(session.OutputModalities, respMods) + explicit := respMods + if len(explicit) == 0 { + explicit = session.OutputModalities + } + payload := map[string]any{} + if wire := s2sWireModalities(explicit); wire != nil { + payload["output_modalities"] = wire + } + if maxOutputTokens != 0 && !maxOutputTokens.IsInf() { + payload["max_output_tokens"] = int(maxOutputTokens) + } + // The mode of the session travels with the config; only an override of the + // response is sent here. + if overrides != nil { + if mode := s2sToolChoiceMode(overrides.ToolChoice); mode != "" { + payload["tool_choice"] = mode + } + } + + _, isWebRTC := t.(*WebRTCTransport) + tn := &s2sTurn{ + ctx: ctx, session: session, conv: conv, t: t, r: r, st: st, + textOnly: !modalitiesContainAudio(modalities), + isWebRTC: isWebRTC, + } + // A response cannot carry tools of its own on this path: the backend has + // the tools of the session and nothing else, so Go and the backend agree. + tn.hasTools = s2sFallbackToolsAllowed(session, overrides) + if session.ModelInterface == nil { + // no model interface, no function config + } else if mc := session.ModelInterface.PredictConfig(); mc != nil { + tn.funcCfg = mc.FunctionsConfig + } + + // Events left over from an earlier response must not reach this one. + for drained := false; !drained; { + select { + case <-st.events: + default: + drained = true + } + } + st.cancelSent.Store(false) + st.active.Store(true) + defer st.active.Store(false) + if err := st.control("response.create", payload); err != nil { + sendError(t, "inference_failed", fmt.Sprintf("backend error: %v", err), "", "") + r.outcome = outcomeFailed + return nil + } + + done := ctx.Done() + var grace <-chan time.Time + stop := func() { + tn.stopped = true + done = nil + // The timer starts first: queuing the cancel must not delay it. + grace = time.After(s2sCancelGrace) + st.sendCancel() + } + for { + select { + case <-done: + stop() + case <-grace: + sendError(t, "server_error", "the speech-to-speech backend did not acknowledge the cancel", "", "") + r.outcome = outcomeFailed + st.fatal() + return nil + case <-st.dead: + sendError(t, "server_error", fmt.Sprintf("the speech-to-speech stream ended: %v", st.deadErr), "", "") + r.outcome = outcomeFailed + st.fatal() + return nil + case ev := <-st.events: + // A cancel may land between the select above and this event: + // check it before anything is forwarded. + if !tn.stopped && ctx.Err() != nil { + stop() + } + if finished := tn.handle(ev, toolTurn); finished { + return tn.after + } + } + } +} + +// handle processes one backend event and reports whether the response is over. +func (tn *s2sTurn) handle(ev *proto.AudioToAudioResponse, toolTurn int) bool { + switch ev.GetEvent() { + case "response.audio.delta": + tn.onAudio(ev.GetPcm(), int(ev.GetSampleRate())) + case "response.audio_transcript.delta", "response.text.delta": + var m struct { + Delta string `json:"delta"` + } + _ = json.Unmarshal(ev.GetMeta(), &m) + tn.onText(m.Delta) + case "response.function_call_arguments.done": + var m struct { + Name string `json:"name"` + Arguments json.RawMessage `json:"arguments"` + } + _ = json.Unmarshal(ev.GetMeta(), &m) + args := string(m.Arguments) + var asString string + if json.Unmarshal(m.Arguments, &asString) == nil { + args = asString + } + if m.Name != "" && !tn.stopped { + tn.toolCalls = append(tn.toolCalls, functions.FuncCallResults{Name: m.Name, Arguments: args}) + } + case "error": + typ, msg, scoped := s2sErrorMeta(ev.GetMeta()) + if typ == "response_cancel_not_active" { + // The response ended just as the cancel went out; its own + // response.done is on the way. + return false + } + sendError(tn.t, typ, msg, "", "") + if !scoped { + // The backend reports problems that are not about this response + // (a refused session.update, an empty commit) and goes on + // generating. Its response.done still comes. + return false + } + tn.r.outcome = outcomeFailed + return true + case "response.done": + var m struct { + Status string `json:"status"` + Text string `json:"text"` + // RawText is what the model wrote, tool-call markup included. + RawText string `json:"raw_text"` + Usage struct { + OutputTokens int `json:"output_tokens"` + } `json:"usage"` + } + _ = json.Unmarshal(ev.GetMeta(), &m) + tn.r.addUsage(backend.TokenUsage{Completion: m.Usage.OutputTokens}) + cancelled := tn.stopped || m.Status == "cancelled" + var override *string + if !cancelled { + calls, content, contentKnown, fromText := s2sResponseToolCalls(tn.toolCalls, m.RawText, tn.hasTools, tn.funcCfg) + tn.toolCalls = calls + if fromText && contentKnown { + override = &content + } + } + tn.closeItem(m.Text, override, cancelled) + if cancelled { + tn.r.outcome = outcomeCancelled + return true + } + if len(tn.toolCalls) > 0 { + hasContent := tn.item != nil + tn.after = func() { + emitToolCallItems(tn.ctx, tn.session, tn.conv, tn.t, tn.r, tn.toolCalls, hasContent, toolTurn) + } + } + return true + } + return false +} + +// checkS2SUpdate rejects session.update fields that an any-to-any session +// cannot honour: switching the model, a classifier, and turn detection modes +// that need other models. +func checkS2SUpdate(session *Session, rt *types.RealtimeSession) error { + if rt.Model != "" && rt.Model != session.Model { + return fmt.Errorf("the model of an any-to-any session cannot change: %w", errS2SOnly) + } + if rt.LocalAIClassifier != nil { + return fmt.Errorf("classifier mode: %w", errS2SOnly) + } + if s2sNamedToolChoice(rt.ToolChoice) { + return errors.New(s2sNamedToolChoiceMessage) + } + if rt.Audio != nil && rt.Audio.Input != nil { + if tr := rt.Audio.Input.Transcription; tr != nil && tr.Model != "" && session.InputAudioTranscription != nil && tr.Model != session.InputAudioTranscription.Model { + return errors.New("the transcription model of an any-to-any session cannot change") + } + if rt.Audio.Input.TurnDetectionSet && rt.Audio.Input.TurnDetection != nil { + td := rt.Audio.Input.TurnDetection + if td.SemanticVad != nil { + return errors.New("semantic_vad is not supported for any-to-any models") + } + if td.ServerVad != nil { + if m, ok := session.ModelInterface.(*anyToAnyModel); ok && m.VADConfig == nil { + return errors.New("server_vad needs a VAD model; set pipeline.vad on the model, or commit turns manually") + } + } + } + } + return nil +} + +// prepareAnyToAnyConfig validates a model that runs on one speech-to-speech +// stream and fills the slots it uses. The VAD slot stays empty: server VAD is +// on only when the model names one. Features that need a separate LLM call +// cannot work on this path and are refused up front. +func prepareAnyToAnyConfig(cfg *config.ModelConfig) (errCode, errMsg string, ok bool) { + p := &cfg.Pipeline + reject := func(what string) (string, string, bool) { + return "invalid_pipeline", what + " is not supported for any-to-any models", false + } + switch { + case p.Classifier != nil && p.Classifier.Enabled: + return reject("pipeline.classifier") + case p.Compaction != nil && p.Compaction.Enabled: + return reject("pipeline.compaction") + case p.StreamLLM(): + return reject("pipeline.streaming.llm") + case p.VoiceGateEnabled(): + return reject("pipeline.voice_recognition") + case p.SoundDetection != "": + return reject("pipeline.sound_detection") + case p.TurnDetectionSemantic(): + return reject("pipeline.turn_detection semantic_vad") + } + if p.LLM == "" { + p.LLM = cfg.Name + } + // pipeline.transcription is left as the config has it: empty means the + // session does not transcribe the input (see s2sInputTranscriptionModel). + return "", "", true +} diff --git a/core/http/endpoints/openai/realtime_s2s_test.go b/core/http/endpoints/openai/realtime_s2s_test.go new file mode 100644 index 000000000..e0e3904b3 --- /dev/null +++ b/core/http/endpoints/openai/realtime_s2s_test.go @@ -0,0 +1,1359 @@ +package openai + +import ( + "context" + "encoding/binary" + "encoding/json" + "errors" + "io" + "slices" + "sync" + "sync/atomic" + "time" + + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/http/endpoints/openai/respcoord" + "github.com/mudler/LocalAI/core/http/endpoints/openai/types" + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/pkg/functions" + "github.com/mudler/LocalAI/pkg/grpc/proto" +) + +// fakeS2SStream is a scripted backend stream. script runs inline in Send, so a +// control message can trigger the backend's answer. +type fakeS2SStream struct { + mu sync.Mutex + sent []*proto.AudioToAudioRequest + recv chan *proto.AudioToAudioResponse + script func(f *fakeS2SStream, req *proto.AudioToAudioRequest) + once sync.Once + owner *s2sSession // flushed before the sent requests are read + + // A script may push from its own goroutine, so a push can run while the + // session closes the stream. closing releases a push that waits for room; + // recvMu keeps recv open while a push sends on it. + closing chan struct{} + recvMu sync.RWMutex +} + +func newFakeS2SStream() *fakeS2SStream { + return &fakeS2SStream{recv: make(chan *proto.AudioToAudioResponse, 256), closing: make(chan struct{})} +} + +func (f *fakeS2SStream) Send(req *proto.AudioToAudioRequest) error { + f.mu.Lock() + f.sent = append(f.sent, req) + f.mu.Unlock() + if f.script != nil { + f.script(f, req) + } + return nil +} + +func (f *fakeS2SStream) Recv() (*proto.AudioToAudioResponse, error) { + r, ok := <-f.recv + if !ok { + return nil, io.EOF + } + return r, nil +} + +func (f *fakeS2SStream) Close() error { + f.once.Do(func() { + close(f.closing) + f.recvMu.Lock() + close(f.recv) + f.recvMu.Unlock() + }) + return nil +} + +func (f *fakeS2SStream) push(event string, pcm []byte, meta any) { + b, _ := json.Marshal(meta) + // Nothing is sent once the stream is closed: the backend has hung up. + f.recvMu.RLock() + defer f.recvMu.RUnlock() + select { + case <-f.closing: + return + default: + } + select { + case f.recv <- &proto.AudioToAudioResponse{Event: event, Pcm: pcm, SampleRate: 24000, Meta: b}: + case <-f.closing: + } +} + +func (f *fakeS2SStream) requests() []*proto.AudioToAudioRequest { + if f.owner != nil { + f.owner.flush() + } + f.mu.Lock() + defer f.mu.Unlock() + return append([]*proto.AudioToAudioRequest(nil), f.sent...) +} + +// controls returns the control event names in send order. +func (f *fakeS2SStream) controls() []string { + var out []string + for _, r := range f.requests() { + if c := r.GetControl(); c != nil { + out = append(out, c.GetEvent()) + } + } + return out +} + +// eofStream fails like a gRPC client stream whose call was rejected: Send +// returns a bare io.EOF and Recv has the real status. +type eofStream struct { + status error + failed chan struct{} + once sync.Once +} + +func (e *eofStream) Send(*proto.AudioToAudioRequest) error { + e.once.Do(func() { close(e.failed) }) + return io.EOF +} + +func (e *eofStream) Recv() (*proto.AudioToAudioResponse, error) { + <-e.failed + return nil, e.status +} + +func (e *eofStream) Close() error { return nil } + +// fakeToolExecutor treats every name in names as a server-side tool. +type fakeToolExecutor struct { + names map[string]bool + output string +} + +func (f *fakeToolExecutor) DiscoverTools(context.Context) ([]functions.Function, error) { + return nil, nil +} +func (f *fakeToolExecutor) IsTool(name string) bool { return f.names[name] } +func (f *fakeToolExecutor) ExecuteTool(context.Context, string, string) (string, error) { + return f.output, nil +} +func (f *fakeToolExecutor) HasTools() bool { return len(f.names) > 0 } + +// overlapStream reports a Close that runs while a Send is in progress, which +// gRPC forbids for CloseSend and SendMsg. +type overlapStream struct { + fakeS2SStream + inSend atomic.Bool + overlap atomic.Bool +} + +func (o *overlapStream) Send(req *proto.AudioToAudioRequest) error { + o.inSend.Store(true) + time.Sleep(20 * time.Millisecond) + o.inSend.Store(false) + return o.fakeS2SStream.Send(req) +} + +func (o *overlapStream) Close() error { + if o.inSend.Load() { + o.overlap.Store(true) + } + return o.fakeS2SStream.Close() +} + +func isControl(r *proto.AudioToAudioRequest, event string) bool { + return r.GetControl() != nil && r.GetControl().GetEvent() == event +} + +func eventTypeNames(evs []types.ServerEvent) []string { + out := make([]string, 0, len(evs)) + for _, e := range evs { + out = append(out, string(e.ServerEventType())) + } + return out +} + +// fromResponseCreated returns the event names starting at response.created. +func fromResponseCreated(evs []types.ServerEvent) []string { + names := eventTypeNames(evs) + for i, n := range names { + if n == "response.created" { + return names[i:] + } + } + return nil +} + +var _ = Describe("realtime speech-to-speech", func() { + const rate = 16000 + + newHarness := func(stream *fakeS2SStream, m *fakeModel) (*Session, *fakeTransport, *Conversation) { + st := newS2SSession(stream, nil, nil) + st.inputRate, st.outputRate = rate, 24000 + stream.owner = st + session := &Session{ + s2s: st, + TurnDetection: &types.TurnDetectionUnion{ServerVad: &types.ServerVad{}}, + InputAudioTranscription: &types.AudioTranscription{}, + ModelInterface: m, + ModelConfig: &config.ModelConfig{}, + InputSampleRate: rate, + OutputSampleRate: 24000, + respSink: newResponseSink(), + } + session.sessionCtx, session.sessionCancel = context.WithCancel(context.Background()) + DeferCleanup(func() { + session.sessionCancel() + session.respSink.shutdown() + st.close() + }) + return session, &fakeTransport{}, &Conversation{} + } + baseModel := func() *fakeModel { + return &fakeModel{ + cfg: &config.ModelConfig{}, + transcribeFinal: &schema.TranscriptionResult{Text: "hello there"}, + } + } + replyScript := func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + if isControl(req, "response.create") { + f.push("response.audio_transcript.delta", nil, map[string]any{"delta": "hi "}) + f.push("response.audio.delta", []byte{1, 2, 3, 4}, nil) + f.push("response.audio_transcript.delta", nil, map[string]any{"delta": "there"}) + f.push("response.audio.delta", []byte{5, 6, 7, 8}, nil) + f.push("response.done", nil, map[string]any{"status": "completed", "text": "hi there", "usage": map[string]any{"output_tokens": 2}}) + } + } + + It("answers a manual commit and response.create in the documented order", func() { + stream := newFakeS2SStream() + stream.script = replyScript + session, tr, conv := newHarness(stream, baseModel()) + + session.InputAudioBuffer = make([]byte, 640) + session.s2sBeginTurn(0) + session.s2sEndTurn() + session.issueClientCommit([]byte{1, 2, 3, 4}, conv, tr) + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + session.respSink.wait() + + // The transcript of the input comes on the side, at any point. + var inOrder []string + for _, n := range fromResponseCreated(tr.events()) { + if n != "conversation.item.input_audio_transcription.completed" { + inOrder = append(inOrder, n) + } + } + Expect(inOrder).To(Equal([]string{ + "response.created", + "response.output_item.added", + "response.content_part.added", + "response.output_audio_transcript.delta", + "response.output_audio.delta", + "response.output_audio_transcript.delta", + "response.output_audio.delta", + "response.output_audio.done", + "response.output_audio_transcript.done", + "response.content_part.done", + "response.output_item.done", + "response.done", + })) + Expect(tr.transcriptDeltaText()).To(Equal("hi there")) + Expect(stream.controls()).To(Equal([]string{"input_audio_buffer.commit", "response.create"})) + + // A client commit does not start a response on its own. + Expect(tr.countEvents(types.ServerEventTypeResponseCreated)).To(Equal(1)) + // The user item is added and its transcript reported. + Expect(eventTypeNames(tr.events())).To(ContainElement("conversation.item.added")) + Eventually(func() []string { return eventTypeNames(tr.events()) }, 2*time.Second). + Should(ContainElement("conversation.item.input_audio_transcription.completed")) + }) + + It("does not hold the response back while the input is transcribed", func() { + stream := newFakeS2SStream() + stream.script = replyScript + m := baseModel() + m.transcribeBlock = make(chan struct{}) + session, tr, conv := newHarness(stream, m) + + session.s2sEndTurn() + session.issueClientCommit([]byte{1, 2, 3, 4}, conv, tr) + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + + Eventually(func() int { return tr.countEvents(types.ServerEventTypeResponseDone) }, 2*time.Second).Should(Equal(1)) + Expect(tr.countEvents(types.ServerEventTypeConversationItemInputAudioTranscriptionCompleted)).To(BeZero()) + + // The transcript still arrives once the ASR is done. + close(m.transcribeBlock) + Eventually(func() int { + return tr.countEvents(types.ServerEventTypeConversationItemInputAudioTranscriptionCompleted) + }, 2*time.Second).Should(Equal(1)) + }) + + It("reports the transcript of an input transcribed on the stream model before the reply", func() { + stream := newFakeS2SStream() + stream.script = replyScript + m := baseModel() + m.transcribeBlock = make(chan struct{}) + session, tr, conv := newHarness(stream, m) + // pipeline.transcription names the model of the stream: one llama.cpp context + session.ModelConfig = &config.ModelConfig{Name: "s2s", Pipeline: config.Pipeline{Transcription: "s2s"}} + + session.s2sEndTurn() + session.issueClientCommit([]byte{1, 2, 3, 4}, conv, tr) + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + + // The reply does not run next to the transcript: they would make llama.cpp + // rebuild its buffers on every step. + Consistently(func() int { return tr.countEvents(types.ServerEventTypeResponseOutputAudioDelta) }, 300*time.Millisecond).Should(BeZero()) + Expect(stream.controls()).ToNot(ContainElement("response.create")) + + close(m.transcribeBlock) + session.respSink.wait() + + // A client reads up to response.done of its reply. The transcript must be in + // that window, ahead of the output of the reply. + names := eventTypeNames(tr.events()) + completed := slices.Index(names, "conversation.item.input_audio_transcription.completed") + Expect(completed).To(BeNumerically(">=", 0), "no transcript: %v", names) + Expect(completed).To(BeNumerically("<", slices.Index(names, "response.output_audio.delta"))) + Expect(completed).To(BeNumerically("<", slices.Index(names, "response.done"))) + }) + + // The WAV the transcription gets must hold the whole turn and say the rate it + // was recorded at: a client commit is at the session rate, a VAD turn at 16 kHz. + DescribeTable("hands the whole turn to the input transcription with its own rate", + func(vad bool, sessionRate, wantRate int) { + stream := newFakeS2SStream() + stream.script = replyScript + m := baseModel() + session, tr, conv := newHarness(stream, m) + session.InputSampleRate = sessionRate + utt := make([]byte, 2*sessionRate) // one second + for i := range utt { + utt[i] = byte(i) + } + + session.s2sEndTurn() + if vad { + slot := &commitSlot{done: make(chan struct{}), source: respcoord.SourceVAD} + s2sCommitTurn(context.Background(), utt, "", session, conv, tr, slot) + } else { + session.issueClientCommit(utt, conv, tr) + } + session.respSink.wait() + Eventually(func() int { m.transcribedMu.Lock(); defer m.transcribedMu.Unlock(); return len(m.transcribedWAV) }, 2*time.Second).Should(Equal(1)) + + m.transcribedMu.Lock() + wav := m.transcribedWAV[0] + m.transcribedMu.Unlock() + Expect(len(wav)).To(BeNumerically(">", 44)) + Expect(binary.LittleEndian.Uint32(wav[24:28])).To(Equal(uint32(wantRate)), "sample rate in the WAV header") + Expect(wav[44:]).To(Equal(utt), "the transcription sees the whole turn") + }, + Entry("a client commit, at the session rate", false, 24000, 24000), + Entry("a VAD turn, at 16 kHz", true, 24000, 16000), + ) + + It("reports a failed input transcription without breaking the response", func() { + stream := newFakeS2SStream() + stream.script = replyScript + m := baseModel() + m.transcribeFinal = nil + m.transcribeErr = errors.New("asr down") + session, tr, conv := newHarness(stream, m) + + session.s2sEndTurn() + session.issueClientCommit([]byte{1, 2, 3, 4}, conv, tr) + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + session.respSink.wait() + + // The transcript runs detached from the response. + Eventually(func() []string { return eventTypeNames(tr.events()) }, 2*time.Second). + Should(ContainElement("conversation.item.input_audio_transcription.failed")) + names := eventTypeNames(tr.events()) + Expect(names).ToNot(ContainElement("conversation.item.input_audio_transcription.completed")) + Expect(names).To(ContainElement("response.done")) + Expect(tr.countEvents(types.ServerEventTypeResponseOutputAudioTranscriptDone)).To(Equal(1)) + }) + + It("round-trips a tool call and its output", func() { + stream := newFakeS2SStream() + creates := 0 + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + if !isControl(req, "response.create") { + return + } + creates++ + if creates == 1 { + f.push("response.function_call_arguments.done", nil, map[string]any{"call_id": "x", "name": "get_weather", "arguments": `{"city":"Paris"}`}) + f.push("response.done", nil, map[string]any{"status": "completed"}) + return + } + replyScript(f, req) + } + session, tr, conv := newHarness(stream, baseModel()) + + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + session.respSink.wait() + + Expect(fromResponseCreated(tr.events())).To(Equal([]string{ + "response.created", + "response.output_item.added", + "response.function_call_arguments.delta", + "response.function_call_arguments.done", + "response.output_item.done", + "response.done", + })) + var callID string + for _, e := range tr.events() { + if d, ok := e.(types.ResponseFunctionCallArgumentsDoneEvent); ok { + Expect(d.Name).To(Equal("get_weather")) + Expect(d.Arguments).To(Equal(`{"city":"Paris"}`)) + callID = d.CallID + } + } + Expect(callID).ToNot(BeEmpty()) + + // What the item.create handler does for the client's tool result. + conv.Lock.Lock() + conv.Items = append(conv.Items, &types.MessageItemUnion{FunctionCallOutput: &types.MessageItemFunctionCallOutput{ + ID: generateItemID(), CallID: callID, Output: `{"temperature_c":21}`, + }}) + conv.Lock.Unlock() + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + session.respSink.wait() + + var outputCtl *proto.AudioToAudioControl + reqs := stream.requests() + for i, r := range reqs { + if isControl(r, "conversation.item.create") { + outputCtl = r.GetControl() + Expect(isControl(reqs[i+1], "response.create")).To(BeTrue(), "the result precedes the response") + } + } + Expect(outputCtl).ToNot(BeNil()) + var payload map[string]string + Expect(json.Unmarshal(outputCtl.GetPayload(), &payload)).To(Succeed()) + Expect(payload).To(HaveKeyWithValue("type", "function_call_output")) + Expect(payload).To(HaveKeyWithValue("call_id", callID)) + Expect(tr.countEvents(types.ServerEventTypeResponseDone)).To(Equal(2)) + Expect(tr.countEvents(types.ServerEventTypeResponseOutputAudioDelta)).To(Equal(2)) + }) + + It("finishes a response whose tool call runs on the server", func() { + stream := newFakeS2SStream() + creates := 0 + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + if !isControl(req, "response.create") { + return + } + creates++ + if creates == 1 { + f.push("response.function_call_arguments.done", nil, map[string]any{"name": "list_models", "arguments": `{}`}) + f.push("response.done", nil, map[string]any{"status": "completed"}) + return + } + replyScript(f, req) + } + session, tr, conv := newHarness(stream, baseModel()) + session.AssistantExecutor = &fakeToolExecutor{names: map[string]bool{"list_models": true}, output: "[]"} + + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + // The follow-up response needs the stream the first one used. A hang + // here means the first response still held it. + Eventually(func() int { return tr.countEvents(types.ServerEventTypeResponseDone) }, 3*time.Second).Should(Equal(1)) + session.respSink.wait() + + Expect(creates).To(Equal(2)) + Expect(tr.countEvents(types.ServerEventTypeResponseCreated)).To(Equal(1)) + Expect(tr.countEvents(types.ServerEventTypeResponseOutputAudioTranscriptDone)).To(Equal(1)) + names := eventTypeNames(tr.events()) + Expect(names[len(names)-1]).To(Equal("response.done")) + }) + + Describe("tool calls in the raw text of a response", func() { + const rawCall = `{"name":"get_time","arguments":{"tz":"UTC"}}` + weather := []types.ToolUnion{{Function: &types.ToolFunction{Name: "get_weather"}}, {Function: &types.ToolFunction{Name: "get_time"}}} + + type scenario struct { + tools []types.ToolUnion + choice *types.ToolChoiceUnion // of the session + params types.ResponseCreateParams + preset string + delta string + callEvent map[string]any + done map[string]any + } + // play answers the first response with the given events and returns + // what the client saw and the conversation. + play := func(sc scenario) (names []string, tr *fakeTransport, conv *Conversation) { + stream := newFakeS2SStream() + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + if !isControl(req, "response.create") { + return + } + f.push("response.audio_transcript.delta", nil, map[string]any{"delta": sc.delta}) + if sc.callEvent != nil { + f.push("response.function_call_arguments.done", nil, sc.callEvent) + } + f.push("response.done", nil, sc.done) + } + m := baseModel() + m.cfg.FunctionsConfig.XMLFormatPreset = sc.preset + var session *Session + session, tr, conv = newHarness(stream, m) + session.Tools = sc.tools + session.ToolChoice = sc.choice + session.issueClientResponse(sc.params, conv, tr) + session.respSink.wait() + for _, e := range tr.events() { + if d, ok := e.(types.ResponseFunctionCallArgumentsDoneEvent); ok { + names = append(names, d.Name) + } + } + return names, tr, conv + } + run := func(tools []types.ToolUnion, callEvent map[string]any, done map[string]any) (names []string) { + names, _, _ = play(scenario{tools: tools, delta: "One moment. ", callEvent: callEvent, done: done}) + return names + } + + It("prefers the calls the backend parsed", func() { + names := run(weather, + map[string]any{"call_id": "x", "name": "get_weather", "arguments": `{"city":"Paris"}`}, + map[string]any{"status": "completed", "text": "One moment. ", "raw_text": "One moment. " + rawCall}) + Expect(names).To(Equal([]string{"get_weather"})) + }) + + It("parses the raw text when the backend found no call", func() { + names := run(weather, nil, + map[string]any{"status": "completed", "text": "One moment. " + rawCall, "raw_text": "One moment. " + rawCall}) + Expect(names).To(Equal([]string{"get_time"})) + }) + + It("does not parse the raw text when tool_choice is none", func() { + none := &types.ToolChoiceUnion{Mode: types.ToolChoiceModeNone} + done := map[string]any{"status": "completed", "text": "One moment. " + rawCall, "raw_text": "One moment. " + rawCall} + + names, _, _ := play(scenario{tools: weather, choice: none, delta: "One moment. ", done: done}) + Expect(names).To(BeEmpty(), "none on the session") + + // the value of the response wins over the one of the session + required := &types.ToolChoiceUnion{Mode: types.ToolChoiceModeRequired} + names, _, _ = play(scenario{tools: weather, choice: required, params: types.ResponseCreateParams{ToolChoice: none}, delta: "One moment. ", done: done}) + Expect(names).To(BeEmpty(), "none on the response") + + names, _, _ = play(scenario{tools: weather, choice: none, params: types.ResponseCreateParams{ToolChoice: required}, delta: "One moment. ", done: done}) + Expect(names).To(Equal([]string{"get_time"}), "required on the response") + }) + + It("uses the tools of the session when a response lists none", func() { + done := map[string]any{"status": "completed", "text": "One moment. " + rawCall, "raw_text": "One moment. " + rawCall} + names, _, _ := play(scenario{tools: weather, params: types.ResponseCreateParams{Tools: []types.ToolUnion{}}, delta: "One moment. ", done: done}) + Expect(names).To(Equal([]string{"get_time"})) + }) + + Context("with the lfm3 preset and no capture regex", func() { + const call = "\n\n\n" + + transcriptDone := func(tr *fakeTransport) (texts []string) { + for _, e := range tr.events() { + if d, ok := e.(types.ResponseOutputAudioTranscriptDoneEvent); ok { + texts = append(texts, d.Transcript) + } + } + return texts + } + + It("keeps the call out of the transcript of a reply that is only a call", func() { + names, tr, _ := play(scenario{tools: weather, preset: "lfm3", delta: call, + done: map[string]any{"status": "completed", "text": call, "raw_text": call}}) + Expect(names).To(Equal([]string{"get_time"})) + Expect(transcriptDone(tr)).To(Equal([]string{""})) + }) + + It("keeps the text around the call", func() { + raw := "One moment.\n" + call + names, tr, _ := play(scenario{tools: weather, preset: "lfm3", delta: raw, + done: map[string]any{"status": "completed", "text": raw, "raw_text": raw}}) + Expect(names).To(Equal([]string{"get_time"})) + Expect(transcriptDone(tr)).To(Equal([]string{"One moment."})) + }) + }) + + It("does not look for calls when the request had no tools", func() { + names := run(nil, nil, + map[string]any{"status": "completed", "text": "One moment. " + rawCall, "raw_text": "One moment. " + rawCall}) + Expect(names).To(BeEmpty()) + }) + }) + + Describe("s2sResponseToolCalls", func() { + parsed := []functions.FuncCallResults{{Name: "a", Arguments: "{}"}} + + It("returns the parsed calls whatever the text says", func() { + calls, content, _, fromText := s2sResponseToolCalls(parsed, `{"name":"b","arguments":{}}`, true, functions.FunctionsConfig{}) + Expect(calls).To(Equal(parsed)) + Expect(content).To(BeEmpty()) + Expect(fromText).To(BeFalse()) + }) + + It("finds a call in the text and returns the content around it", func() { + fc := functions.FunctionsConfig{CaptureLLMResult: []string{`(?s)^(.*?)\s*\{"name"`}} + calls, content, known, fromText := s2sResponseToolCalls(nil, `Sure. {"name":"b","arguments":{"x":1}}`, true, fc) + Expect(calls).To(HaveLen(1)) + Expect(calls[0].Name).To(Equal("b")) + Expect(content).To(Equal("Sure.")) + Expect(known).To(BeTrue()) + Expect(fromText).To(BeTrue()) + + // without a capture regex the content is unknown, as in the chained pipeline + calls, content, known, fromText = s2sResponseToolCalls(nil, `Sure. {"name":"b","arguments":{"x":1}}`, true, functions.FunctionsConfig{}) + Expect(calls).To(HaveLen(1)) + Expect(content).To(BeEmpty()) + Expect(known).To(BeFalse()) + Expect(fromText).To(BeTrue()) + }) + + It("cuts the lfm3 call markup out of the content without a capture regex", func() { + fc := functions.FunctionsConfig{XMLFormatPreset: "lfm3"} + const call = "\n\n\n" + + calls, content, known, fromText := s2sResponseToolCalls(nil, call, true, fc) + Expect(calls).To(HaveLen(1)) + Expect(fromText).To(BeTrue()) + Expect(known).To(BeTrue()) + Expect(content).To(BeEmpty()) + + calls, content, known, _ = s2sResponseToolCalls(nil, "One moment.\n"+call, true, fc) + Expect(calls).To(HaveLen(1)) + Expect(known).To(BeTrue()) + Expect(content).To(Equal("One moment.")) + }) + + It("finds nothing in plain text or without tools", func() { + calls, _, _, fromText := s2sResponseToolCalls(nil, "just words", true, functions.FunctionsConfig{}) + Expect(calls).To(BeEmpty()) + Expect(fromText).To(BeFalse()) + calls, _, _, fromText = s2sResponseToolCalls(nil, `{"name":"b","arguments":{}}`, false, functions.FunctionsConfig{}) + Expect(calls).To(BeEmpty()) + Expect(fromText).To(BeFalse()) + }) + }) + + Describe("a named function as tool_choice", func() { + named := &types.ToolChoiceUnion{Function: &types.ToolChoiceFunction{Name: "f"}} + + It("is refused in session.update", func() { + err := checkS2SUpdate(&Session{}, &types.RealtimeSession{ToolChoice: named}) + Expect(err).To(HaveOccurred()) + Expect(checkS2SUpdate(&Session{}, &types.RealtimeSession{ToolChoice: &types.ToolChoiceUnion{Mode: types.ToolChoiceModeRequired}})).To(Succeed()) + }) + + It("is refused in response.create without a request to the backend", func() { + stream := newFakeS2SStream() + creates := 0 + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + if isControl(req, "response.create") { + creates++ + } + } + session, tr, conv := newHarness(stream, baseModel()) + session.issueClientResponse(types.ResponseCreateParams{ToolChoice: named}, conv, tr) + session.respSink.wait() + Expect(creates).To(Equal(0)) + Expect(tr.countEvents(types.ServerEventTypeError)).To(BeNumerically(">=", 1)) + }) + }) + + Describe("tool_choice on the wire", func() { + It("maps a mode and drops a named function", func() { + Expect(s2sToolChoiceMode(nil)).To(BeEmpty()) + Expect(s2sToolChoiceMode(&types.ToolChoiceUnion{Mode: types.ToolChoiceModeRequired})).To(Equal("required")) + Expect(s2sToolChoiceMode(&types.ToolChoiceUnion{Mode: types.ToolChoiceModeNone})).To(Equal("none")) + Expect(s2sToolChoiceMode(&types.ToolChoiceUnion{Function: &types.ToolChoiceFunction{Name: "f"}})).To(BeEmpty()) + }) + + It("sends the session mode in the config and a response mode with response.create", func() { + stream := newFakeS2SStream() + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + if req.GetConfig() != nil { + f.push("session.created", nil, map[string]any{"input_sample_rate": rate, "output_sample_rate": 24000}) + } + if isControl(req, "response.create") { + f.push("response.done", nil, map[string]any{"status": "completed"}) + } + } + session := &Session{ToolChoice: &types.ToolChoiceUnion{Mode: types.ToolChoiceModeRequired}} + Expect(openS2SForSession(session, openerFunc(func(context.Context) (S2SStream, error) { return stream, nil }), &fakeTransport{})).To(Succeed()) + stream.owner = session.s2s + DeferCleanup(session.closeS2S) + Expect(stream.requests()[0].GetConfig().GetParams()).To(HaveKeyWithValue("tool_choice", "required")) + }) + }) + + It("cancels on barge-in before the new speech and forwards no audio after it", func() { + stream := newFakeS2SStream() + cancelled := make(chan struct{}) + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + switch { + case isControl(req, "response.create"): + go func() { + for { + select { + case <-cancelled: + return + case <-time.After(5 * time.Millisecond): + f.push("response.audio.delta", []byte{1, 1}, nil) + } + } + }() + case isControl(req, "response.cancel"): + close(cancelled) + // In-flight audio the backend had already produced. + for range 3 { + f.push("response.audio.delta", []byte{0xEE, 0xEE}, nil) + } + f.push("response.done", nil, map[string]any{"status": "cancelled"}) + } + } + session, tr, conv := newHarness(stream, baseModel()) + session.InputAudioBuffer = make([]byte, rate*2) // one second of the new speech + + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + Eventually(func() int { return tr.countEvents(types.ServerEventTypeResponseOutputAudioDelta) }, 2*time.Second).Should(BeNumerically(">=", 2)) + + // What the VAD does on speech onset: start the turn, then cancel. + session.s2sBeginTurn(0) + session.respSink.cancel(respcoord.SourceVAD) + session.respSink.wait() + + reqs := stream.requests() + cancelAt, frameAt := -1, -1 + for i, r := range reqs { + if isControl(r, "response.cancel") && cancelAt < 0 { + cancelAt = i + } + if r.GetFrame() != nil && frameAt < 0 { + frameAt = i + } + } + Expect(cancelAt).To(BeNumerically(">=", 0)) + Expect(frameAt).To(BeNumerically(">", cancelAt), "the cancel goes before the first frame") + Expect(stream.controls()).To(ContainElement("response.cancel")) + Expect(countCancels(stream)).To(Equal(1)) + + for _, e := range tr.events() { + if d, ok := e.(types.ResponseOutputAudioDeltaEvent); ok { + Expect(d.Delta).ToNot(Equal("7u4="), "audio produced after the cancel reached the client") + } + } + names := eventTypeNames(tr.events()) + Expect(names[len(names)-1]).To(Equal("response.done")) + var done types.ResponseDoneEvent + for _, e := range tr.events() { + if d, ok := e.(types.ResponseDoneEvent); ok { + done = d + } + } + Expect(done.Response.Status).To(Equal(types.ResponseStatusCancelled)) + }) + + It("forwards a VAD turn from speech start minus the pre-roll", func() { + stream := newFakeS2SStream() + session, _, _ := newHarness(stream, baseModel()) + session.InputAudioBuffer = make([]byte, 3*rate*2) // 3 s + + // Before the turn opens nothing goes out. + session.s2sOnAppendLocked(make([]byte, 640)) + Expect(stream.requests()).To(BeEmpty()) + + session.s2sBeginTurn(2.0) // forwards from 1.7 s: 1.3 s of the buffer + Expect(framesBytes(stream)).To(Equal(int(1.3 * rate * 2))) + + session.s2sOnAppendLocked(make([]byte, 640)) + Expect(framesBytes(stream)).To(Equal(int(1.3*rate*2) + 640)) + + session.s2sEndTurn() + Expect(isControl(stream.requests()[len(stream.requests())-1], "input_audio_buffer.commit")).To(BeTrue()) + session.s2sOnAppendLocked(make([]byte, 640)) + Expect(framesBytes(stream)).To(Equal(int(1.3*rate*2)+640), "forwarding stops at the commit") + }) + + It("holds manual-commit audio back until the commit", func() { + stream := newFakeS2SStream() + session, _, conv := newHarness(stream, baseModel()) + session.TurnDetection = nil + chunk := make([]byte, 640) + session.AudioBufferLock.Lock() + session.InputAudioBuffer = append(session.InputAudioBuffer, chunk...) + session.s2sOnAppendLocked(chunk) + session.AudioBufferLock.Unlock() + Expect(stream.requests()).To(BeEmpty(), "idle audio must not reach the model's context") + + session.s2sCommitAudio(chunk, conv) + Expect(framesBytes(stream)).To(Equal(640)) + Expect(stream.controls()).To(Equal([]string{"input_audio_buffer.commit"})) + }) + + It("discards cleared manual-commit audio for good", func() { + stream := newFakeS2SStream() + session, tr, conv := newHarness(stream, baseModel()) + session.TurnDetection = nil + session.AudioBufferLock.Lock() + session.InputAudioBuffer = make([]byte, 640) + session.s2sOnAppendLocked(make([]byte, 640)) + session.AudioBufferLock.Unlock() + + clearInputAudio(session) + commitClientAudio(session, conv, tr) + session.respSink.wait() + Expect(framesBytes(stream)).To(BeZero()) + }) + + It("commits uncommitted audio before a response.create asks for an answer", func() { + stream := newFakeS2SStream() + stream.script = replyScript + session, tr, conv := newHarness(stream, baseModel()) + session.TurnDetection = nil + session.InputAudioBuffer = make([]byte, 640) + Expect(session.hasBufferedInput()).To(BeTrue()) + + commitClientAudio(session, conv, tr) + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + session.respSink.wait() + + Expect(session.hasBufferedInput()).To(BeFalse()) + Expect(stream.controls()).To(Equal([]string{"input_audio_buffer.commit", "response.create"})) + }) + + It("keeps a text item ahead of the audio committed after it", func() { + stream := newFakeS2SStream() + session, _, conv := newHarness(stream, baseModel()) + session.TurnDetection = nil + conv.Items = append(conv.Items, &types.MessageItemUnion{User: &types.MessageItemUser{ + ID: "t1", + Content: []types.MessageContentInput{{Type: types.MessageContentTypeInputText, Text: "hello"}}, + }}) + + session.s2sCommitAudio(make([]byte, 640), conv) + + var kinds []string + for _, r := range stream.requests() { + switch { + case r.GetFrame() != nil: + kinds = append(kinds, "frame") + case r.GetControl() != nil: + kinds = append(kinds, r.GetControl().GetEvent()) + } + } + Expect(kinds).To(Equal([]string{"conversation.item.create", "frame", "input_audio_buffer.commit"})) + }) + + It("ends the open VAD turn when turn detection is switched off", func() { + stream := newFakeS2SStream() + session, _, conv := newHarness(stream, baseModel()) + session.InputAudioBuffer = make([]byte, 640) + session.s2sBeginTurn(0) + Expect(session.s2sUserTurnOpen()).To(BeTrue()) + + session.s2sStopVAD() + Expect(session.s2sUserTurnOpen()).To(BeFalse()) + + // Manual mode: new audio is buffered, not forwarded, and a commit + // sends only that. + session.TurnDetection = nil + chunk := make([]byte, 320) + session.InputAudioBuffer = append(session.InputAudioBuffer, chunk...) + session.s2sOnAppendLocked(chunk) + session.s2sCommitAudio(session.InputAudioBuffer, conv) + + pcm := 0 + for _, r := range stream.requests() { + if f := r.GetFrame(); f != nil { + pcm += len(f.GetPcm()) + } + } + Expect(pcm).To(Equal(640+320), "each sample reaches the backend once") + Expect(stream.controls()).To(Equal([]string{"input_audio_buffer.commit", "input_audio_buffer.commit"})) + }) + + It("holds a tool result back while a user turn is open and sends it after", func() { + stream := newFakeS2SStream() + session, _, conv := newHarness(stream, baseModel()) + conv.Items = append(conv.Items, &types.MessageItemUnion{FunctionCallOutput: &types.MessageItemFunctionCallOutput{ + ID: "fo1", CallID: "call_1", Output: "{}", + }}) + session.InputAudioBuffer = make([]byte, 640) + session.s2sBeginTurn(0) + + // The user speaks before the result is forwarded: the backend would + // refuse it, so it must stay pending and not be marked as sent. + session.s2s.forwardPendingItems(conv, &session.AudioBufferLock) + Expect(stream.controls()).To(BeEmpty()) + + // A turn that was dropped after audio went out is still open. + session.s2sAbortTurn() + session.s2s.forwardPendingItems(conv, &session.AudioBufferLock) + Expect(stream.controls()).To(BeEmpty()) + + session.s2sEndTurn() + session.s2s.forwardPendingItems(conv, &session.AudioBufferLock) + session.s2s.forwardPendingItems(conv, &session.AudioBufferLock) + Expect(stream.controls()).To(Equal([]string{"input_audio_buffer.commit", "conversation.item.create"})) + }) + + It("resamples a committed turn to the backend input rate", func() { + stream := newFakeS2SStream() + session, _, conv := newHarness(stream, baseModel()) + session.InputSampleRate = 24000 + session.s2sCommitAudio(make([]byte, 24000*2/10), conv) // 100 ms at 24 kHz + Expect(framesBytes(stream)).To(BeNumerically("~", rate*2/10, 8)) + }) + + It("reports an open VAD turn so a tool result can be refused", func() { + stream := newFakeS2SStream() + session, _, _ := newHarness(stream, baseModel()) + Expect(session.s2sUserTurnOpen()).To(BeFalse()) + session.s2sBeginTurn(0) + Expect(session.s2sUserTurnOpen()).To(BeTrue()) + session.s2sEndTurn() + Expect(session.s2sUserTurnOpen()).To(BeFalse()) + }) + + It("ends the session when the stream dies during a response", func() { + stream := newFakeS2SStream() + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + if isControl(req, "response.create") { + _ = f.Close() // the backend goes away mid-response + } + } + session, tr, conv := newHarness(stream, baseModel()) + var fatal atomic.Int32 + session.s2s.setOnFatal(func() { fatal.Add(1) }) + + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + session.respSink.wait() + + Eventually(fatal.Load, 2*time.Second).Should(Equal(int32(1))) + Expect(tr.countEvents(types.ServerEventTypeError)).To(Equal(1)) + // Nothing more is accepted or queued for the dead stream. + Expect(session.s2s.forwardPCM(make([]byte, 640), rate)).To(BeZero()) + Expect(session.s2s.control("input_audio_buffer.commit", nil)).To(MatchError(errS2SClosed)) + session.s2s.outMu.Lock() + queued := len(session.s2s.outQ) + session.s2s.outMu.Unlock() + Expect(queued).To(BeZero()) + }) + + It("drops what is queued after the session is closed", func() { + stream := newFakeS2SStream() + session, _, _ := newHarness(stream, baseModel()) + session.s2s.close() + for range 50 { + session.s2s.sendAsync(&proto.AudioToAudioRequest{}) + } + session.s2s.outMu.Lock() + queued := len(session.s2s.outQ) + session.s2s.outMu.Unlock() + Expect(queued).To(BeZero()) + }) + + It("closes the stream only when no write is in progress", func() { + stream := &overlapStream{fakeS2SStream: fakeS2SStream{recv: make(chan *proto.AudioToAudioResponse, 8), closing: make(chan struct{})}} + st := newS2SSession(stream, nil, nil) + for range 5 { + st.sendAsync(&proto.AudioToAudioRequest{}) + } + time.Sleep(30 * time.Millisecond) // a Send is under way + st.close() + Eventually(func() bool { + select { + case _, ok := <-stream.recv: + return !ok + default: + return false + } + }, 2*time.Second).Should(BeTrue()) + Expect(stream.overlap.Load()).To(BeFalse()) + }) + + It("calls onDead when the stream ends outside a response", func() { + stream := newFakeS2SStream() + st := newS2SSession(stream, nil, nil) + got := make(chan error, 1) + st.setOnDead(func(err error) { got <- err }) + _ = stream.Close() + Eventually(got).Should(Receive()) + st.close() + }) + + It("does not call onDead for a deliberate close", func() { + stream := newFakeS2SStream() + st := newS2SSession(stream, nil, nil) + got := make(chan error, 1) + st.setOnDead(func(err error) { got <- err }) + st.close() + Consistently(got, 200*time.Millisecond).ShouldNot(Receive()) + }) + + It("surfaces the real status when Send fails with a bare EOF", func() { + stream := &eofStream{status: status.Error(codes.Unimplemented, "no such method"), failed: make(chan struct{})} + st := newS2SSession(stream, nil, nil) + DeferCleanup(st.close) + err := st.open(context.Background(), s2sConfig{}) + Expect(err).To(HaveOccurred()) + Expect(status.Code(errors.Unwrap(err))).To(Equal(codes.Unimplemented)) + }) + + It("reports a backend error outside a response", func() { + stream := newFakeS2SStream() + got := make(chan string, 1) + st := newS2SSession(stream, nil, func(typ, msg string) { got <- typ + ": " + msg }) + DeferCleanup(st.close) + stream.push("error", nil, map[string]any{"type": "invalid_request_error", "message": "instructions cannot change"}) + Eventually(got).Should(Receive(Equal("invalid_request_error: instructions cannot change"))) + }) + + Describe("backend errors during a response", func() { + errorCodes := func(tr *fakeTransport) []string { + var out []string + for _, e := range tr.events() { + if ev, ok := e.(types.ErrorEvent); ok { + out = append(out, ev.Error.Code) + } + } + return out + } + + It("keeps the response going after an error that is not about it", func() { + stream := newFakeS2SStream() + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + if isControl(req, "response.create") { + f.push("response.audio_transcript.delta", nil, map[string]any{"delta": "hi "}) + f.push("error", nil, map[string]any{"type": "invalid_request_error", "message": "session.update: voice cannot change after the first audio or message"}) + f.push("response.audio_transcript.delta", nil, map[string]any{"delta": "there"}) + f.push("response.done", nil, map[string]any{"status": "completed", "text": "hi there"}) + } + } + session, tr, conv := newHarness(stream, baseModel()) + + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + session.respSink.wait() + + Expect(tr.transcriptDeltaText()).To(Equal("hi there")) + names := eventTypeNames(tr.events()) + Expect(names[len(names)-1]).To(Equal("response.done")) + Expect(errorCodes(tr)).To(Equal([]string{"invalid_request_error"}), "the client still hears about the error") + }) + + It("ends the response on an error that answers its response.create", func() { + stream := newFakeS2SStream() + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + if isControl(req, "response.create") { + f.push("error", nil, map[string]any{"type": "conversation_empty", "message": "commit audio or create a message first", "response_scoped": true}) + } + } + session, tr, conv := newHarness(stream, baseModel()) + + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + session.respSink.wait() + + Expect(errorCodes(tr)).To(Equal([]string{"conversation_empty"})) + Expect(fromResponseCreated(tr.events())).To(Equal([]string{"response.created", "error"})) + }) + + It("does not take events of an earlier response for the next one", func() { + stream := newFakeS2SStream() + stream.script = replyScript + session, tr, conv := newHarness(stream, baseModel()) + // What an earlier response left in the buffer. + session.s2s.events <- &proto.AudioToAudioResponse{Event: "response.audio_transcript.delta", Meta: []byte(`{"delta":"stale "}`)} + session.s2s.events <- &proto.AudioToAudioResponse{Event: "response.done", Meta: []byte(`{"status":"completed","text":"stale"}`)} + + session.issueClientResponse(types.ResponseCreateParams{}, conv, tr) + session.respSink.wait() + + Expect(tr.transcriptDeltaText()).To(Equal("hi there")) + }) + + It("sends no commit when no audio was forwarded for the turn", func() { + stream := newFakeS2SStream() + session, _, _ := newHarness(stream, baseModel()) + + session.s2sEndTurn() + Expect(stream.controls()).To(BeEmpty()) + + session.InputAudioBuffer = make([]byte, 640) + session.s2sBeginTurn(0) + session.s2sEndTurn() + Expect(stream.controls()).To(Equal([]string{"input_audio_buffer.commit"})) + }) + }) + + It("ends the reader when nobody reads the events and the session closes", func() { + stream := newFakeS2SStream() + st := newS2SSession(stream, nil, nil) + st.active.Store(true) + for range 100 { + stream.push("response.audio.delta", []byte{1, 2}, nil) + } + Eventually(func() int { return len(st.events) }).Should(Equal(cap(st.events))) + + st.close() + Eventually(st.dead, 2*time.Second).Should(BeClosed()) + }) + + It("gives up on a cancel the backend never takes, even when the write queue is stuck", func() { + old := s2sCancelGrace + s2sCancelGrace = 100 * time.Millisecond + DeferCleanup(func() { s2sCancelGrace = old }) + + stream := newFakeS2SStream() + release := make(chan struct{}) + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + switch { + case isControl(req, "response.create"): + f.push("response.audio_transcript.delta", nil, map[string]any{"delta": "hi"}) + case isControl(req, "response.cancel"): + <-release // backpressure: the write never completes + } + } + DeferCleanup(func() { close(release) }) + session, tr, conv := newHarness(stream, baseModel()) + + ctx, cancel := context.WithCancel(context.Background()) + r := &liveResponse{id: "resp_1"} + finished := make(chan struct{}) + go func() { + defer close(finished) + s2sResponseTurn(ctx, session, conv, tr, nil, 0, r) + }() + Eventually(func() string { return tr.transcriptDeltaText() }).Should(Equal("hi")) + + cancel() + Eventually(finished, 3*time.Second).Should(BeClosed()) + Expect(r.outcome).To(Equal(outcomeFailed)) + }) + + Describe("the model that transcribes the input of an any-to-any session", func() { + It("is none by default, another model, or the model itself when named", func() { + yes := true + for name, tc := range map[string]struct{ transcription, want string }{ + "default": {"", ""}, + "other model": {"asr", "asr"}, + "own model": {"s2s", "s2s"}, + } { + cfg := &config.ModelConfig{Name: "s2s", Pipeline: config.Pipeline{AnyToAny: &yes, Transcription: tc.transcription}} + _, _, ok := prepareRealtimeConfig(cfg) + Expect(ok).To(BeTrue(), name) + Expect(s2sInputTranscriptionModel(cfg)).To(Equal(tc.want), name) + } + }) + }) + + Describe("the input transcription on the model of the stream", func() { + sessionOn := func(name, transcription string) *Session { + st := newS2SSession(newFakeS2SStream(), nil, nil) + DeferCleanup(st.close) + return &Session{s2s: st, ModelConfig: &config.ModelConfig{Name: name, Pipeline: config.Pipeline{Transcription: transcription}}} + } + + It("is told apart from a transcription model of its own", func() { + Expect(s2sTranscribesOnStreamModel(sessionOn("s2s", "s2s"))).To(BeTrue()) + Expect(s2sTranscribesOnStreamModel(sessionOn("s2s", "asr"))).To(BeFalse()) + Expect(s2sTranscribesOnStreamModel(&Session{ModelConfig: &config.ModelConfig{Name: "s2s"}})).To(BeFalse()) + }) + + It("runs only while no response holds the stream, and keeps responses out meanwhile", func() { + st := sessionOn("s2s", "s2s").s2s + st.sem <- struct{}{} // a response is running + + ran := make(chan struct{}) + release := make(chan struct{}) + done := make(chan bool, 1) + go func() { + done <- st.exclusive(context.Background(), func() { + close(ran) + <-release + }) + }() + Consistently(ran, 100*time.Millisecond).ShouldNot(BeClosed()) + + <-st.sem // the response ends + Eventually(ran).Should(BeClosed()) + select { + case st.sem <- struct{}{}: + Fail("a response took the stream while the transcription ran") + case <-time.After(50 * time.Millisecond): + } + close(release) + Eventually(done).Should(Receive(BeTrue())) + Expect(st.sem).To(BeEmpty()) + }) + + It("gives up when its context ends before the stream is free", func() { + st := sessionOn("s2s", "s2s").s2s + st.sem <- struct{}{} + ctx, cancel := context.WithCancel(context.Background()) + cancel() + Expect(st.exclusive(ctx, func() { Fail("ran") })).To(BeFalse()) + }) + }) + + Describe("the output mode on the wire", func() { + openSession := func(tools []types.ToolUnion, mods []types.Modality) (*Session, *fakeS2SStream) { + stream := newFakeS2SStream() + stream.script = func(f *fakeS2SStream, req *proto.AudioToAudioRequest) { + if req.GetConfig() != nil { + f.push("session.created", nil, map[string]any{"input_sample_rate": rate, "output_sample_rate": 24000}) + } + } + session := &Session{Tools: tools, OutputModalities: mods} + Expect(openS2SForSession(session, openerFunc(func(context.Context) (S2SStream, error) { return stream, nil }), &fakeTransport{})).To(Succeed()) + stream.owner = session.s2s + DeferCleanup(session.closeS2S) + return session, stream + } + + It("leaves the mode to the backend when the client chose none", func() { + session, stream := openSession(nil, nil) + cfg := stream.requests()[0].GetConfig() + Expect(cfg.GetParams()).ToNot(HaveKey("output_modalities")) + + // Tools added before the first turn must not meet a mode fixed at open. + session.Tools = []types.ToolUnion{{Function: &types.ToolFunction{Name: "get_time"}}} + session.s2sSyncConfig() + var update map[string]any + for _, r := range stream.requests() { + if isControl(r, "session.update") { + Expect(json.Unmarshal(r.GetControl().GetPayload(), &update)).To(Succeed()) + } + } + Expect(update).To(HaveKey("tools")) + Expect(update).ToNot(HaveKey("output_modalities")) + }) + + It("forwards the flat tools of a client session.update to the backend in the nested form", func() { + session, stream := openSession(nil, nil) + // The event as the real-weights e2e spec sends it: the Responses + // API tool shape, no session type, tools before the first audio. + var update types.SessionUpdateEvent + Expect(json.Unmarshal([]byte(`{"type":"session.update","session":{"tools":[{"type":"function","name":"get_weather","description":"Current weather for a city.","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}]}}`), &update)).To(Succeed()) + Expect(updateSession(session, &update.Session, nil, nil, nil, nil, nil, nil)).To(Succeed()) + session.s2sSyncConfig() + + var payload struct { + Tools []struct { + Type string `json:"type"` + Function struct { + Name string `json:"name"` + Parameters map[string]any `json:"parameters"` + } `json:"function"` + } `json:"tools"` + } + var seen bool + for _, r := range stream.requests() { + if isControl(r, "session.update") { + seen = true + Expect(json.Unmarshal(r.GetControl().GetPayload(), &payload)).To(Succeed()) + } + } + Expect(seen).To(BeTrue(), "no session.update control reached the backend") + Expect(payload.Tools).To(HaveLen(1)) + Expect(payload.Tools[0].Type).To(Equal("function")) + Expect(payload.Tools[0].Function.Name).To(Equal("get_weather")) + Expect(payload.Tools[0].Function.Parameters).To(HaveKey("properties")) + }) + + It("sends the modalities the client chose", func() { + _, stream := openSession(nil, []types.Modality{types.ModalityText}) + Expect(stream.requests()[0].GetConfig().GetParams()).To(HaveKeyWithValue("output_modalities", `["text"]`)) + }) + }) + + Describe("prepareRealtimeConfig for any-to-any models", func() { + yes := true + newCfg := func() *config.ModelConfig { + return &config.ModelConfig{Name: "s2s", Pipeline: config.Pipeline{AnyToAny: &yes}} + } + It("fills llm, leaves transcription, VAD and TTS empty", func() { + cfg := newCfg() + _, _, ok := prepareRealtimeConfig(cfg) + Expect(ok).To(BeTrue()) + Expect(cfg.Pipeline.LLM).To(Equal("s2s")) + // no input transcription unless the config asks for one + Expect(cfg.Pipeline.Transcription).To(BeEmpty()) + Expect(s2sInputTranscriptionModel(cfg)).To(BeEmpty()) + Expect(cfg.Pipeline.VAD).To(BeEmpty()) + Expect(cfg.Pipeline.TTS).To(BeEmpty()) + }) + It("keeps a configured VAD and transcription model", func() { + cfg := newCfg() + cfg.Pipeline.VAD, cfg.Pipeline.Transcription = "silero", "asr" + _, _, ok := prepareRealtimeConfig(cfg) + Expect(ok).To(BeTrue()) + Expect(cfg.Pipeline.VAD).To(Equal("silero")) + Expect(cfg.Pipeline.Transcription).To(Equal("asr")) + }) + DescribeTable("rejects features that need a separate LLM call", + func(mutate func(*config.ModelConfig), want string) { + cfg := newCfg() + mutate(cfg) + code, msg, ok := prepareRealtimeConfig(cfg) + Expect(ok).To(BeFalse()) + Expect(code).To(Equal("invalid_pipeline")) + Expect(msg).To(ContainSubstring(want)) + }, + Entry("classifier", func(c *config.ModelConfig) { c.Pipeline.Classifier = &config.PipelineClassifier{Enabled: true} }, "classifier"), + Entry("compaction", func(c *config.ModelConfig) { c.Pipeline.Compaction = &config.PipelineCompaction{Enabled: true} }, "compaction"), + Entry("streaming llm", func(c *config.ModelConfig) { c.Pipeline.Streaming.LLM = &yes }, "streaming.llm"), + Entry("voice recognition", func(c *config.ModelConfig) { + c.Pipeline.VoiceRecognition = &config.PipelineVoiceRecognition{Model: "spk"} + }, "voice_recognition"), + Entry("sound detection", func(c *config.ModelConfig) { c.Pipeline.SoundDetection = "ced" }, "sound_detection"), + ) + }) + + It("returns errS2SOnly for chained-pipeline operations", func() { + m := &anyToAnyModel{} + _, err := m.Predict(context.Background(), nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) + Expect(errors.Is(err, errS2SOnly)).To(BeTrue()) + _, _, err = m.TTS(context.Background(), "x", "", "") + Expect(errors.Is(err, errS2SOnly)).To(BeTrue()) + _, err = m.VAD(context.Background(), &schema.VADRequest{}) + Expect(err).To(HaveOccurred(), "no VAD model configured") + }) +}) + +type openerFunc func(ctx context.Context) (S2SStream, error) + +func (f openerFunc) OpenS2S(ctx context.Context) (S2SStream, error) { return f(ctx) } + +func countCancels(f *fakeS2SStream) int { + n := 0 + for _, c := range f.controls() { + if c == "response.cancel" { + n++ + } + } + return n +} + +// framesBytes totals the PCM bytes of the frames sent so far. +func framesBytes(f *fakeS2SStream) int { + n := 0 + for _, r := range f.requests() { + if fr := r.GetFrame(); fr != nil { + n += len(fr.GetPcm()) + } + } + return n +} diff --git a/core/http/endpoints/openai/realtime_semantic_vad.go b/core/http/endpoints/openai/realtime_semantic_vad.go index ac5e6b4d8..981a6d15d 100644 --- a/core/http/endpoints/openai/realtime_semantic_vad.go +++ b/core/http/endpoints/openai/realtime_semantic_vad.go @@ -367,6 +367,16 @@ func turnDetectionActive(td *types.TurnDetectionUnion) bool { return td != nil && (td.ServerVad != nil || td.SemanticVad != nil) } +// defaultSessionTurnDetection is the turn detection a new session starts with. +// A pipeline that names no VAD model has nothing to detect turns with, so the +// client commits them (turn_detection none). +func defaultSessionTurnDetection(cfg *config.ModelConfig) *types.TurnDetectionUnion { + if cfg == nil || cfg.Pipeline.VAD == "" { + return nil + } + return defaultTurnDetection(cfg) +} + // defaultTurnDetection seeds a new session's turn detection from the // pipeline's server-side default: semantic_vad pipelines start sessions in // semantic mode (clients can still override via session.update); everything diff --git a/core/http/endpoints/openai/realtime_stream_test.go b/core/http/endpoints/openai/realtime_stream_test.go index 32cefe342..461853748 100644 --- a/core/http/endpoints/openai/realtime_stream_test.go +++ b/core/http/endpoints/openai/realtime_stream_test.go @@ -3,6 +3,7 @@ package openai import ( "context" "fmt" + "time" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" @@ -225,6 +226,35 @@ var _ = Describe("streamLLMResponse", func() { }) var _ = Describe("triggerResponse", func() { + // A barge-in cancels the response context while the LLM is still generating. + // The call then fails with the context error. That is a cancel, not a + // failure: the client must get response.done{cancelled} and no error, or it + // waits for a terminal that never comes. + It("ends as cancelled when the context is cancelled during the LLM call", func() { + m := &fakeModel{cfg: &config.ModelConfig{}, predictUntilCancel: true} + session := &Session{OutputSampleRate: 24000, ModelInterface: m, ModelConfig: &config.ModelConfig{}} + t := &fakeTransport{} + ctx, cancel := context.WithCancel(context.Background()) + + done := make(chan struct{}) + go func() { + defer close(done) + triggerResponse(ctx, session, &Conversation{}, t, nil) + }() + cancel() + Eventually(done, 2*time.Second).Should(BeClosed()) + + Expect(t.countEvents(types.ServerEventTypeResponseDone)).To(Equal(1)) + Expect(t.countEvents(types.ServerEventTypeError)).To(BeZero()) + var status types.ResponseStatus + for _, e := range t.events() { + if d, ok := e.(types.ResponseDoneEvent); ok { + status = d.Response.Status + } + } + Expect(status).To(Equal(types.ResponseStatusCancelled)) + }) + It("emits exactly one response.created and one response.done with output and usage", func() { m := &fakeModel{ cfg: &config.ModelConfig{}, diff --git a/core/http/react-ui/e2e/talk.spec.js b/core/http/react-ui/e2e/talk.spec.js index 073eaddc3..d051b03ff 100644 --- a/core/http/react-ui/e2e/talk.spec.js +++ b/core/http/react-ui/e2e/talk.spec.js @@ -304,6 +304,83 @@ test.describe('Talk session settings', () => { }) }) +test.describe('Talk with a speech-to-speech model', () => { + // The model list the server returns for an any-to-any model: no slots of + // its own, and a VAD only when the model config names one. + async function s2sModel(page, { vad = '' } = {}) { + await page.route('**/api/pipeline-models', route => route.fulfill({ + json: [{ name: 'lfm3-audio', voice: 'female', vad, transcription: 'lfm3-audio', llm: 'lfm3-audio', tts: 'lfm3-audio', any_to_any: true, self_contained: true }], + })) + } + + test('the sheet names one speech-to-speech model, with no slots and no voice detection', async ({ page }) => { + await fakeRealtime(page) + await s2sModel(page) + await page.goto('/app/talk') + await page.getByTestId('talk-settings-button').click() + const pipeline = page.getByTestId('talk-pipeline') + await expect(pipeline).toContainText('Speech-to-speech model') + await expect(pipeline).not.toContainText('Transcription') + await expect(pipeline).not.toContainText('Language model') + await expect(pipeline).toContainText('No voice detection model is set.') + await expect(pipeline.getByRole('button', { name: 'Edit model config' })).toBeVisible() + await expect(page.getByTestId('talk-settings').getByLabel('Voice', { exact: true })).toHaveAttribute('list', 'talk-voice-options') + }) + + test('without voice detection, Done speaking ends the turn: commit, then response.create', async ({ page }) => { + await fakeRealtime(page) + await s2sModel(page) + await startSession(page) + await expect(status(page)).toHaveAttribute('data-state', 'listening') + await expect(status(page)).toContainText('Say what you need, then press Done speaking.') + await page.getByTestId('talk-end-turn').click() + const types = await page.evaluate(() => window.__talk.sent.map(m => m.type)) + expect(types.slice(-2)).toEqual(['input_audio_buffer.commit', 'response.create']) + }) + + test('without voice detection, the input buffer is emptied when the session starts and when a reply ends', async ({ page }) => { + await fakeRealtime(page) + await s2sModel(page) + await startSession(page) + const sentTypes = () => page.evaluate(() => window.__talk.sent.map(m => m.type)) + expect(await sentTypes()).toContain('input_audio_buffer.clear') + const before = (await sentTypes()).filter(t => t === 'input_audio_buffer.clear').length + await emit(page, { type: 'response.done', response: { status: 'completed' } }) + const after = (await sentTypes()).filter(t => t === 'input_audio_buffer.clear').length + expect(after).toBe(before + 1) + }) + + test('idle note says to press Done speaking when no voice detection model is set', async ({ page }) => { + await fakeRealtime(page) + await s2sModel(page) + await page.goto('/app/talk') + await expect(page.getByText('press Done speaking when you finish a turn')).toBeVisible() + await expect(page.getByText('The server notices when you stop talking')).toHaveCount(0) + }) + + test('with a voice detection model, the turns are detected and Done speaking is not offered', async ({ page }) => { + await fakeRealtime(page) + await s2sModel(page, { vad: 'silero-vad' }) + await startSession(page) + await expect(status(page)).toHaveAttribute('data-state', 'listening') + await expect(page.getByTestId('talk-end-turn')).toHaveCount(0) + await expect(page.getByTestId('talk-settings-button')).toBeVisible() + }) + + test('text-only replies stream into the transcript', async ({ page }) => { + await fakeRealtime(page) + await s2sModel(page) + await startSession(page) + await emit(page, { type: 'response.output_text.delta', item_id: 'a1', delta: 'Hello ' }) + await emit(page, { type: 'response.output_text.delta', item_id: 'a1', delta: 'there' }) + await emit(page, { type: 'response.output_text.done', item_id: 'a1', text: 'Hello there.' }) + await emit(page, { type: 'response.done', response: { status: 'completed' } }) + const lines = page.getByTestId('talk-transcript').locator('.talk-turn') + await expect(lines).toHaveCount(1) + await expect(lines.nth(0)).toContainText('Hello there.') + }) +}) + test.describe('Talk on a phone', () => { test.use({ viewport: { width: 390, height: 844 } }) diff --git a/core/http/react-ui/public/locales/en/tools.json b/core/http/react-ui/public/locales/en/tools.json index 67617c7d9..2f1ef9c5f 100644 --- a/core/http/react-ui/public/locales/en/tools.json +++ b/core/http/react-ui/public/locales/en/tools.json @@ -243,10 +243,6 @@ "rewardsHint": "GRPO needs at least one reward function. Pick built-in ones or add your own.", "resume": "Resume from checkpoint", "clearCheckpoint": "Clear the checkpoint", - "liquidHint": "The dataset must be preprocessed by LFM2AudioChatMapper: a directory of arrow files the LFM2 data loader can read.", - "liquidVoice": "Speech voice", - "liquidInherit": "Inherit from the system prompt", - "liquidVal": "Validation dataset", "extra": "Extra options", "extraHint": "Backend-specific key and value pairs.", "summaryFull": "full model", diff --git a/core/http/react-ui/src/components/tools/tools.css b/core/http/react-ui/src/components/tools/tools.css index eb8bc6f47..4cdf71170 100644 --- a/core/http/react-ui/src/components/tools/tools.css +++ b/core/http/react-ui/src/components/tools/tools.css @@ -80,8 +80,8 @@ .bt-kv__row { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1fr) auto; gap: var(--dk-space-2); width: 100%; align-items: center; } .bt-banner { display: flex; align-items: center; gap: var(--dk-space-3); padding: var(--dk-space-3) var(--dk-space-4); border-radius: var(--dk-radius-2); background: var(--dk-accent-wash); color: var(--dk-text); font-size: var(--dk-text-13); flex-wrap: wrap; } .bt-banner code { overflow-wrap: anywhere; } -.bt-rewards, .bt-liquid { display: flex; flex-direction: column; gap: var(--dk-space-3); padding-top: var(--dk-space-3); border-top: 1px solid var(--dk-edge); } -.bt-rewards .dk-hint, .bt-liquid .dk-hint { margin: 0; } +.bt-rewards { display: flex; flex-direction: column; gap: var(--dk-space-3); padding-top: var(--dk-space-3); border-top: 1px solid var(--dk-edge); } +.bt-rewards .dk-hint { margin: 0; } .bt-reward { display: flex; flex-direction: column; gap: var(--dk-space-2); padding: var(--dk-space-3); border: 1px solid var(--dk-edge); border-radius: var(--dk-radius-2); background: var(--dk-card); min-width: 0; } .bt-reward[data-on="true"] { background: var(--dk-accent-wash); border-color: transparent; } .bt-reward__name { display: block; font-size: var(--dk-text-13); font-weight: var(--dk-weight-semibold); } diff --git a/core/http/routes/ui.go b/core/http/routes/ui.go index 9053bd31a..f0fbfa919 100644 --- a/core/http/routes/ui.go +++ b/core/http/routes/ui.go @@ -26,10 +26,11 @@ func RegisterUIRoutes(app *echo.Echo, app.GET("/api/canvas/preview/:id", localai.GetCanvasPreviewEndpoint(canvasPreviews)) // Pipeline models API (for the Talk page WebRTC interface). - // A model qualifies when it either declares an explicit VAD+STT+LLM+TTS - // pipeline (legacy/composed) or carries the realtime_audio usecase (a - // self-contained any-to-any audio backend like liquid-audio that owns the - // full loop in a single AudioToAudioStream RPC). + // A model qualifies when it either declares an explicit STT+LLM+TTS (the VAD is optional) + // pipeline (legacy/composed) or is an any-to-any speech-to-speech model + // (realtime_audio plus pipeline.any_to_any). The second one runs the whole + // loop in a single AudioToAudioStream RPC, so it needs no slots: its VAD + // is optional and the other slots carry its own name. app.GET("/api/pipeline-models", func(c echo.Context) error { type pipelineModelInfo struct { Name string `json:"name"` @@ -38,19 +39,20 @@ func RegisterUIRoutes(app *echo.Echo, LLM string `json:"llm"` TTS string `json:"tts"` Voice string `json:"voice"` - // SelfContained is true for any-to-any audio models — the four - // pipeline slots are populated with the model's own name so the - // UI can render them, but the Realtime API routes the session - // directly to the backend's AudioToAudioStream RPC. + // AnyToAny is true for a speech-to-speech model. The Talk page shows + // it as one model, with no four-slot view. + AnyToAny bool `json:"any_to_any,omitempty"` + // SelfContained is the older name of AnyToAny, kept for clients + // built before the field existed. SelfContained bool `json:"self_contained,omitempty"` } pipelineModels := cl.GetModelConfigsByFilter(func(_ string, cfg *config.ModelConfig) bool { - if cfg.HasUsecases(config.FLAG_REALTIME_AUDIO) { + if cfg.Pipeline.IsAnyToAny() && cfg.HasUsecases(config.FLAG_REALTIME_AUDIO) { return true } p := cfg.Pipeline - return p.VAD != "" && p.Transcription != "" && p.LLM != "" && p.TTS != "" + return p.Transcription != "" && p.LLM != "" && p.TTS != "" }) slices.SortFunc(pipelineModels, func(a, b config.ModelConfig) int { @@ -59,14 +61,15 @@ func RegisterUIRoutes(app *echo.Echo, models := make([]pipelineModelInfo, 0, len(pipelineModels)) for _, cfg := range pipelineModels { - if cfg.HasUsecases(config.FLAG_REALTIME_AUDIO) { + if cfg.Pipeline.IsAnyToAny() && cfg.HasUsecases(config.FLAG_REALTIME_AUDIO) { models = append(models, pipelineModelInfo{ Name: cfg.Name, - VAD: cfg.Name, + VAD: cfg.Pipeline.VAD, Transcription: cfg.Name, LLM: cfg.Name, TTS: cfg.Name, Voice: cfg.TTSConfig.Voice, + AnyToAny: true, SelfContained: true, }) continue diff --git a/core/http/routes/ui_pipeline_models_test.go b/core/http/routes/ui_pipeline_models_test.go index 24f4c3099..db6462ca0 100644 --- a/core/http/routes/ui_pipeline_models_test.go +++ b/core/http/routes/ui_pipeline_models_test.go @@ -84,60 +84,95 @@ pipeline: Expect(models[0]["vad"]).To(Equal("silero")) Expect(models[0]["llm"]).To(Equal("llama")) Expect(models[0]["voice"]).To(Equal("en-amy")) - // self_contained is omitempty — absent for legacy pipelines. + // any_to_any and self_contained are omitempty — absent for legacy pipelines. _, hasFlag := models[0]["self_contained"] Expect(hasFlag).To(BeFalse()) + _, hasAnyToAny := models[0]["any_to_any"] + Expect(hasAnyToAny).To(BeFalse()) }) - It("surfaces self-contained any-to-any models tagged with realtime_audio", func() { + It("lists a chained pipeline that has no VAD", func() { + writeConfig("no-vad-pipeline", ` +name: no-vad-pipeline +backend: llama-cpp +pipeline: + transcription: whisper + llm: llama + tts: piper +`) + // Still partial without a speech slot. + writeConfig("no-tts-pipeline", ` +name: no-tts-pipeline +backend: llama-cpp +pipeline: + transcription: whisper + llm: llama +`) + + models := queryPipelineModels() + Expect(models).To(HaveLen(1)) + Expect(models[0]["name"]).To(Equal("no-vad-pipeline")) + Expect(models[0]["vad"]).To(BeEmpty()) + Expect(models[0]["tts"]).To(Equal("piper")) + }) + + It("surfaces any-to-any models as one speech-to-speech model with an optional VAD", func() { writeConfig("lfm-realtime", ` name: lfm-realtime -backend: liquid-audio +backend: llama-cpp known_usecases: - realtime_audio - chat - tts - transcript +pipeline: + any_to_any: true tts: - voice: us_female + voice: female `) models := queryPipelineModels() Expect(models).To(HaveLen(1)) Expect(models[0]["name"]).To(Equal("lfm-realtime")) - // All four pipeline slots are populated with the model's own name so - // the Talk page UI has something to render. - Expect(models[0]["vad"]).To(Equal("lfm-realtime")) - Expect(models[0]["transcription"]).To(Equal("lfm-realtime")) + Expect(models[0]["any_to_any"]).To(BeTrue()) + Expect(models[0]["self_contained"]).To(BeTrue()) + Expect(models[0]["voice"]).To(Equal("female")) + // No VAD configured: server VAD stays off and the client commits turns. + Expect(models[0]["vad"]).To(BeEmpty()) + // The model answers every role, so the UI can address it by name. Expect(models[0]["llm"]).To(Equal("lfm-realtime")) Expect(models[0]["tts"]).To(Equal("lfm-realtime")) - Expect(models[0]["voice"]).To(Equal("us_female")) - Expect(models[0]["self_contained"]).To(BeTrue()) }) - It("includes both legacy and self-contained models in the same response", func() { - writeConfig("legacy", ` -name: legacy + It("reports the configured VAD of an any-to-any model", func() { + writeConfig("lfm-vad", ` +name: lfm-vad backend: llama-cpp +known_usecases: + - realtime_audio pipeline: + any_to_any: true vad: silero - transcription: whisper - llm: llama - tts: piper `) - writeConfig("realtime", ` -name: realtime -backend: liquid-audio + + models := queryPipelineModels() + Expect(models).To(HaveLen(1)) + Expect(models[0]["any_to_any"]).To(BeTrue()) + Expect(models[0]["vad"]).To(Equal("silero")) + }) + + It("does not list a realtime_audio model that is not any-to-any and has no pipeline", func() { + // Without any_to_any the session takes the chained path, which needs + // the four slots. A bare realtime_audio model has none, so it is not + // a selectable realtime model. + writeConfig("chained-bare", ` +name: chained-bare +backend: llama-cpp known_usecases: - realtime_audio `) - models := queryPipelineModels() - Expect(models).To(HaveLen(2)) - // Sorted by name → legacy, realtime. - Expect(models[0]["name"]).To(Equal("legacy")) - Expect(models[1]["name"]).To(Equal("realtime")) - Expect(models[1]["self_contained"]).To(BeTrue()) + Expect(queryPipelineModels()).To(BeEmpty()) }) It("excludes models that have neither a pipeline nor realtime_audio", func() { diff --git a/docs/content/advanced/model-configuration.md b/docs/content/advanced/model-configuration.md index accd57ba5..df10b628a 100644 --- a/docs/content/advanced/model-configuration.md +++ b/docs/content/advanced/model-configuration.md @@ -157,7 +157,7 @@ behavior. The initially migrated backend families are `transformers` and its aliases, `diffusers`, `qwen-asr`, `fish-speech`, `nemo`, `voxcpm`, `qwen-tts`, -`liquid-audio`, `vllm`, `vllm-omni`, and `sglang`. Automatic imports add +`vllm`, `vllm-omni`, and `sglang`. Automatic imports add artifact declarations only for this set. Compatible external backends may opt in by declaring the artifact explicitly. @@ -915,6 +915,7 @@ Configure how the model handles function/tool calls: | `function.replace_function_results` | array | | Replace function call results with patterns | | `function.replace_llm_results` | array | | Replace LLM results with patterns | | `function.capture_llm_results` | array | | Capture LLM results as text (e.g., for "thinking" blocks) | +| `function.xml_format_preset` | string | | Force an XML tool-call format: `qwen3-coder`, `qwen3.5`, `lfm3`, `glm-4.5`, `minimax-m2`, `kimi-k2` and others. Empty tries all of them | ### Grammar Configuration @@ -1135,6 +1136,7 @@ Define pipelines for audio-to-audio processing and the [Realtime API]({{%relref | `pipeline.llm` | string | LLM model name | | `pipeline.transcription` | string | Transcription model name | | `pipeline.vad` | string | Voice activity detection model name | +| `pipeline.any_to_any` | bool | When `true`, a `realtime_audio` model runs as one speech-to-speech model (audio in, audio or text out) instead of the chained VAD, transcription, LLM and TTS path. Experimental. See [Speech-to-speech models]({{%relref "features/openai-realtime#speech-to-speech-models-experimental" %}}) | | `pipeline.turn_detection` | object | Realtime turn-detection defaults. Keys: `type` (`server_vad`/`semantic_vad`), `eagerness` (`low`/`medium`/`high`/`auto`), `retranscribe`, `vad_window_sec` (widen the per-tick VAD scan window; values below the automatic floor are ignored). See [Realtime turn detection]({{%relref "features/openai-realtime" %}}) | | `pipeline.classifier` | object | Realtime classifier mode: prefill-scored option selection instead of generation. Keys: `enabled`, `threshold`, `normalization` (`raw`/`mean`), `history_items`, `fallback` (`mode`: `none`/`reply`/`generate`, `reply`), `options` (list of `id`, `description`, `reply`, `tool` `{name, arguments}`), `address` (wake-word gate: `names`, `mode`: `ignore`/`reply`, `reply`), `model` (optional separate scoring config). See [Realtime classifier mode]({{%relref "features/openai-realtime#classifier-mode-localai-extension" %}}) | diff --git a/docs/content/features/audio-to-text.md b/docs/content/features/audio-to-text.md index 8e78f9bcf..96458b12b 100644 --- a/docs/content/features/audio-to-text.md +++ b/docs/content/features/audio-to-text.md @@ -176,6 +176,10 @@ curl http://localhost:8080/v1/audio/transcriptions \ -F model="qwen3-asr" ``` +### LFM3-Audio (experimental) + +The gallery entries `lfm3-audio-2b` and `lfm3-audio-2b-q8` also transcribe through the `llama-cpp` backend, using the same model as speech-to-speech and TTS. When a `language` is given, the request asks for transcription in that language with the system prompt `Perform ASR in .`. Without one, the model is asked to infer the language. Transcription uses the second slot (`parallel:2`), so it can run next to a realtime session. A realtime session does not use it for the input transcript by default; see [Realtime]({{% relref "openai-realtime" %}}). This needs a llama.cpp build with LFM3-Audio support. + ## Using the parakeet-cpp backend [parakeet.cpp](https://github.com/mudler/parakeet.cpp) is a C++/ggml port of NVIDIA NeMo Parakeet that matches the upstream PyTorch models on CPU. GGUF weights for every model and quant are published in a single repo, [`mudler/parakeet-cpp-gguf`](https://huggingface.co/mudler/parakeet-cpp-gguf). F16 is the recommended default, and Q4_K stays near-lossless on the small models. The easiest path is to import directly (the GGUFs auto-detect to this backend): diff --git a/docs/content/features/audio-transform.md b/docs/content/features/audio-transform.md index 58733f51f..04877e975 100644 --- a/docs/content/features/audio-transform.md +++ b/docs/content/features/audio-transform.md @@ -174,7 +174,7 @@ PCM frames; server emits binary PCM frames at the same cadence. This endpoint accepts models with the `audio_transform` use case. It does not accept any-to-any models with the `realtime_audio` use case. For example, -`liquid-audio` uses the [OpenAI Realtime API]({{% relref "features/openai-realtime" %}}) +a realtime any-to-any model uses the [OpenAI Realtime API]({{% relref "features/openai-realtime" %}}) instead. ### Wire format diff --git a/docs/content/features/openai-realtime.md b/docs/content/features/openai-realtime.md index 662fe9aa1..390290a7a 100644 --- a/docs/content/features/openai-realtime.md +++ b/docs/content/features/openai-realtime.md @@ -465,6 +465,111 @@ Registering an option list (pipeline seed or `session.update`) prewarms the scor The concrete scoring model must declare `score` in `known_usecases`. A single llama.cpp model can serve ordinary inference and classification concurrently by declaring multiple use cases, for example `known_usecases: [chat, completion, score]`; LocalAI reserves the scoring slots only when `score` is present. Scoring also requires the unified KV cache, which is enabled by default, so a score-enabled model cannot set `kv_unified:false`. +## Speech-to-speech models (experimental) + +A speech-to-speech (S2S) model takes audio in and answers with audio, text or both, in one model. Instead of chaining VAD, transcription, an LLM and TTS, the session sends audio to the model and gets the reply back from it. The llama-cpp backend serves these models. + +{{% notice warning %}} +Speech-to-speech over the Realtime API is experimental. It needs a llama.cpp build with LFM3-Audio and realtime support, which the LocalAI llama.cpp pin must include. It has not been measured on real hardware, and its behaviour may change. +{{% /notice %}} + +### Configuration + +The gallery has [LiquidAI LFM3-Audio-2B](https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF) in two quantizations (Q4_K_M or Q8_0 weights, both with the Q8_0 audio projector) and four shapes each. Every entry is independent: installing `lfm3-audio-2b` always gets the Q4_K_M build, never Q8_0. + +| Entry (Q4_K_M) | Entry (Q8_0) | Shape | VAD | +|---|---|---|---| +| `lfm3-audio-2b` | `lfm3-audio-2b-q8` | One speech-to-speech model | none, the client commits turns | +| `lfm3-audio-2b-vad` | `lfm3-audio-2b-q8-vad` | One speech-to-speech model | `silero-vad-ggml` | +| `lfm3-audio-2b-pipeline` | `lfm3-audio-2b-q8-pipeline` | Chained pipeline on the same model | none, the client commits turns | +| `lfm3-audio-2b-pipeline-vad` | `lfm3-audio-2b-q8-pipeline-vad` | Chained pipeline on the same model | `silero-vad-ggml` | + +The gallery cannot install one model as a dependency of another. The `-vad` entries name `silero-vad-ggml` in `pipeline.vad`, so install it first with `local-ai models install silero-vad-ggml`. Without it, the session fails to start with a model-not-found error. + +Install one with `local-ai models install lfm3-audio-2b`. Its model configuration looks like this: + +```yaml +name: lfm3-audio-2b +backend: llama-cpp +context_size: 8192 +mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf +parameters: + model: LFM3-Audio-2B-Q4_K_M.gguf +options: + - parallel:2 +known_usecases: [realtime_audio, chat, transcript, tts] +pipeline: + any_to_any: true +template: + use_tokenizer_template: true +``` + +- **`pipeline.any_to_any: true`** runs the model as one speech-to-speech model. Without it, a `realtime_audio` model keeps the chained path (VAD, transcription, LLM, TTS). +- **`parallel:2`** gives the model two slots. One is held by the session. The other serves transcription, TTS or chat calls at the same time. + +### Chained pipeline on LFM3 + +The `-pipeline` entries do not use the speech-to-speech stream. They run the classic realtime pipeline (VAD, transcription, LLM, TTS), with the transcription, chat and speech stages all served by the one llama.cpp model: speech recognition through the transcription path, the reply through the chat path (with its templates, tools and streaming), and speech synthesis through the TTS path with the named voice `female` (or `male`). Choose it when you want the regular chat path or the text of each stage; choose the speech-to-speech entries for the lowest latency. The model configuration names itself in each slot and is not a `realtime_audio` model: + +```yaml +name: lfm3-audio-2b-pipeline-vad +backend: llama-cpp +mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf +parameters: + model: LFM3-Audio-2B-Q4_K_M.gguf +known_usecases: [chat, transcript, tts] +tts: + voice: female +pipeline: + vad: silero-vad-ggml + transcription: lfm3-audio-2b-pipeline-vad + llm: lfm3-audio-2b-pipeline-vad + tts: lfm3-audio-2b-pipeline-vad +``` + +Keep the installed model name, because the slots refer to it. The `-pipeline` entries have no `vad` line. A chained pipeline without a VAD starts with turn detection off, refuses `turn_detection` of type `server_vad` in `session.update`, and expects `input_audio_buffer.commit` followed by `response.create`. + +### Migrating from the liquid-audio backend + +The Python `liquid-audio` backend is removed, together with its importer and its gallery entries. It served LFM2-Audio models. Use the `llama-cpp` backend with `lfm3-audio-2b` or `lfm3-audio-2b-q8` instead, and set `pipeline.any_to_any: true` as shown above. A model with `backend: llama-cpp` and `realtime_audio` but without `pipeline.any_to_any` is refused with `invalid_pipeline`, because llama.cpp has no VAD, transcription or TTS of its own. + +LFM2.5-Audio is not covered: llama.cpp does not run it, so there is no replacement for it. Remove the old model configurations that name `backend: liquid-audio`. + +### Voice activity detection + +Speech-to-speech models do not have a VAD of their own. Turn detection depends on the pipeline: + +- **With a VAD model** (`pipeline.vad`, for example `silero-vad-ggml`, installed separately), the server detects the start and end of each turn. Use `turn_detection` with `server_vad` as in the rest of this page. Audio is forwarded from 300 ms before speech starts, so silence before the user speaks is not added to the model context. +- **Without a VAD model** (the plain `lfm3-audio-2b` entries), server VAD is off. The client ends each turn with `input_audio_buffer.commit` and then sends `response.create`. Set `turn_detection` to `null` in `session.update` to make this explicit. + +### Tools + +Define tools in `session.update` as usual. When the model calls one, the session sends `response.output_item.added` with a `function_call` item, then `response.function_call_arguments.delta` and `response.function_call_arguments.done`, and then `response.done`. To return the result, send a `conversation.item.create` with `type: function_call_output`, the same `call_id` and the `output` string, and then `response.create`. The model answers with the result in the next reply. A reply that is only a tool call has no audio. + +How a call is read depends on the model: + +- The backend reads the text of the reply with the parser and grammar of the model's chat template, as the llama.cpp server does for `/v1/realtime`. The call is cut out of the transcript. The audio is sent as the model produces it, also after a call started. `tool_choice` (`auto`, `none` or `required`) works in `session.update` and in `response.create`, and the response value wins. A named function cannot be forced: `session.update` and `response.create` refuse it with an error. The backend knows only the tools of the session, so a `tools` list on `response.create` does not change what it reads. With `tool_choice` set to `none`, a call the model writes anyway stays text. +- If the backend found no call and the request had tools, LocalAI parses the raw text of the reply with the `function` settings of the model, the way the chained pipeline does. Set `function.xml_format_preset` (the LFM3-Audio gallery entries set it to `lfm3`) or the other `function` options for a format the backend parser does not know. In this case the call text can already be in the streamed transcript, and the transcript is replaced only if `function.capture_llm_results` captures the content. + +### Transcripts + +- **Reply transcript.** `response.output_audio_transcript.delta` and `response.output_audio_transcript.done` carry the text of the spoken reply. In a text-only response, `response.output_text.delta` is used instead. +- **User transcript.** The stream answers from the audio, so an any-to-any model does not transcribe the input unless `pipeline.transcription` is set. When it names another model (a sherpa-onnx ASR, for example), `conversation.item.input_audio_transcription.completed` is sent for the user item, in parallel with the reply. When it names the model itself, the transcript is made on a second slot before the reply starts, so it arrives before the first output of that reply (for a manual commit and for a VAD turn), and it delays the reply by its own run time. Both share one llama.cpp context, and while they run together llama.cpp rebuilds its buffers on every step, which slows the reply to a few tokens per second; this is why the default is no transcript. The transcript of the reply always comes from the stream. If transcription fails, `conversation.item.input_audio_transcription.failed` is sent and the reply continues. + +### Barge-in + +Interruption works as described in [Controlling response interruption](#controlling-response-interruption). When the user starts speaking during a reply, the server stops the reply, sends `response.done` with `status: cancelled`, and handles the new speech as the next turn. The part of the reply that was already decoded stays in the model context, as in llama.cpp. + +### Limits + +- **Audio format.** The model listens to 16 kHz mono audio. LocalAI resamples the session input to 16 kHz. Output audio is 24 kHz mono PCM. +- **Voice.** The `voice` setting is not applied to speech-to-speech output. llama.cpp's realtime slot has no voice selection. The `voice` setting still applies to the [text-to-speech endpoint]({{%relref "features/text-to-audio" %}}). +- **Context.** A session holds one slot for its whole life, and all its audio stays in the context. A long session reaches `context_size`. The session then sends an error and closes. There is no compaction for speech-to-speech sessions. +- **Session updates after the first turn.** Once the first audio or message has been added, `instructions`, `tools` and `voice` cannot change. A `session.update` that tries to change them gets an `error`. `output_modalities` and `max_output_tokens` can still change. +- **Not supported.** `conversation.item.truncate` and `conversation.item.delete` return an `error`, because the context cannot be cut by audio time. `input_audio_buffer.clear` drops audio that has not been committed. Without a VAD model, the server keeps appended audio in its buffer and sends it to the model only at `input_audio_buffer.commit` (or at a `response.create` that finds uncommitted audio), so a clear discards it for good. With server VAD, audio is streamed to the model during a turn, and what was already sent stays in the context. +- **Pipeline features.** The pipeline features that need a separate LLM call are rejected with `invalid_pipeline`: classifier mode, conversation compaction with a summary, `streaming.llm`, and voice recognition gating. +- **Distributed mode.** The conversation state lives in one backend slot, so a session stays on the instance that opened it until it ends. See [distributed mode]({{%relref "features/distributed-mode" %}}) for how instances are chosen. + ## Transports The Realtime API supports two transports: **WebSocket** and **WebRTC**. @@ -751,7 +856,7 @@ This event is a LocalAI extension to the OpenAI Realtime API and is server-emitt ## Talk in the web UI -The **Talk** page of the web UI is a client for this API over WebRTC. Pick a pipeline model with the chip at the top, then press **Start session**. The microphone streams while the session runs. The server detects when you stop talking and answers on its own, and you can speak over a reply to interrupt it. The page has no push-to-talk or hands-free switch. +The **Talk** page of the web UI is a client for this API over WebRTC. Pick a pipeline model with the chip at the top, then press **Start session**. The microphone streams while the session runs. The server detects when you stop talking and answers on its own, and you can speak over a reply to interrupt it. The page has no push-to-talk or hands-free switch. A speech-to-speech model with no VAD model set has no turn detection: the page shows **Done speaking**, which ends the turn and asks for the reply. The heading under the orb says what is happening: connecting, listening, thinking (also while a tool runs), speaking, or interrupted after you cut a reply off. The orb follows the real microphone level while it listens and the reply's audio while it speaks. These states have their own screen: diff --git a/docs/content/features/text-to-audio.md b/docs/content/features/text-to-audio.md index 180d4eaea..56da7b23f 100644 --- a/docs/content/features/text-to-audio.md +++ b/docs/content/features/text-to-audio.md @@ -1079,6 +1079,10 @@ prose style, so `voice` selects a speaker index or a voice name and `instruction equivalent. See the [NeMo-Speech.cpp backend]({{%relref "features/nemo-speech-cpp" %}}) page for the full option list and the conversion steps. +### LFM3-Audio (llama.cpp, experimental) + +[LFM3-Audio-2B](https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF) synthesises speech through the `llama-cpp` backend. Install the gallery entry `lfm3-audio-2b` (or `lfm3-audio-2b-q8`). Its voices are `female` (the default) and `male`. The `language` field takes a language code, and the default is `en`. The model uses the sampling defaults stored in its GGUF files. This needs a llama.cpp build with LFM3-Audio support. + ## Response format To provide some compatibility with OpenAI API regarding `response_format`, ffmpeg must be installed (or a docker image including ffmpeg used) to leverage converting the generated wav file before the api provide its response. diff --git a/docs/content/reference/compatibility-table.md b/docs/content/reference/compatibility-table.md index 9585a1e6d..69d042c3d 100644 --- a/docs/content/reference/compatibility-table.md +++ b/docs/content/reference/compatibility-table.md @@ -77,10 +77,15 @@ All backends listed here can be installed on demand from the [Backend Gallery]({ | [Kitten TTS](https://github.com/KittenML/KittenTTS) | Kitten TTS model | CPU, Metal | | [Supertonic](https://github.com/supertone-inc/supertonic) | Lightning-fast on-device multilingual TTS via ONNX | CPU | | [MLX-Audio](https://github.com/Blaizzy/mlx-audio) | Audio models on Apple Silicon | CPU, CUDA 12/13, Metal, Jetson L4T | -| [liquid-audio](https://github.com/Liquid4All/liquid-audio) | LFM2 end-to-end speech-to-speech, ASR, and TTS | CPU, CUDA 12/13, ROCm, Intel SYCL, Jetson L4T | | [audio.cpp](https://github.com/0xShug0/audio.cpp) | Multi-family GGML audio engine: TTS, voice cloning and voice design (`supertonic`, `vibevoice`, `qwen3_tts`, `chatterbox` and more). See [audio.cpp backend](/features/audio-cpp/) | CPU, CUDA 12/13, Metal | | [NeMo-Speech.cpp](https://github.com/NVIDIA/NeMo-Speech.cpp) | NVIDIA MagpieTTS over NanoCodec, served by the same backend as Nemotron ASR. See [NeMo-Speech.cpp backend](/features/nemo-speech-cpp/) | CPU, CUDA 12/13, Vulkan, Metal, Jetson L4T | +## Speech-to-Speech + +| Backend | Description | Acceleration | +|---------|-------------|-------------| +| [llama.cpp](https://github.com/ggml-org/llama.cpp) | Any-to-any audio models (for example [LFM3-Audio]({{%relref "features/openai-realtime" %}}), experimental) served through the Realtime API, with tool calls and transcripts. See [Speech-to-speech models]({{%relref "features/openai-realtime#speech-to-speech-models-experimental" %}}) | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T | + ## Music & Sound Generation | Backend | Description | Acceleration | diff --git a/gallery/backends/index.yaml b/gallery/backends/index.yaml index d9d0d0eb4..82af69fd5 100644 --- a/gallery/backends/index.yaml +++ b/gallery/backends/index.yaml @@ -3491,40 +3491,6 @@ urls: - https://github.com/microsoft/VibeVoice version: 0.0.0-latest -- alias: liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: latest - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: liquid-audio - kind: backend - license: LFM-Open-License-v1.0 - name: liquid-audio - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-latest - alias: qwen-tts capabilities: amd: rocm-qwen-tts @@ -35500,600 +35466,6 @@ urls: - https://github.com/microsoft/VibeVoice version: 0.0.0-development -- alias: liquid-audio - capabilities: - amd: rocm-liquid-audio-development - default: cpu-liquid-audio-development - intel: intel-liquid-audio-development - metal: metal-liquid-audio-development - nvidia: cuda12-liquid-audio-development - nvidia-cuda-12: cuda12-liquid-audio-development - nvidia-cuda-13: cuda13-liquid-audio-development - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio-development - channel: development - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: liquid-audio-development - kind: backend - license: LFM-Open-License-v1.0 - name: liquid-audio-development - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-development -- alias: liquid-audio - arch: amd64 - artifact: oci://quay.io/go-skynet/local-ai-backends:latest-cpu-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: latest - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - digest: sha256:ad9798b82dae6f2e8d12b077bf1e029c586f44480cea2ff2428671fbbbd4a870 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: cpu-liquid-audio - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:latest-cpu-liquid-audio - name: cpu-liquid-audio - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-latest.ad9798b82dae -- alias: liquid-audio - arch: amd64 - artifact: oci://quay.io/go-skynet/local-ai-backends:master-cpu-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: development - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: cpu-liquid-audio-development - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:master-cpu-liquid-audio - mutable: true - name: cpu-liquid-audio-development - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-development -- alias: liquid-audio - arch: arm64 - artifact: oci://quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: latest - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - digest: sha256:f744bafbc2026d852d109161c9aad3efd99d5b339fcdea0856d058e1a50e6d78 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: metal-liquid-audio - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:latest-metal-darwin-arm64-liquid-audio - name: metal-liquid-audio - os: darwin - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-latest.f744bafbc202 -- alias: liquid-audio - arch: arm64 - artifact: oci://quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: development - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: metal-liquid-audio-development - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:master-metal-darwin-arm64-liquid-audio - mutable: true - name: metal-liquid-audio-development - os: darwin - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-development -- alias: liquid-audio - arch: amd64 - artifact: oci://quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: latest - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - digest: sha256:ca554511dd390a5ffcef6d02164e27f1c22226d51598365bb6126219c0c870b8 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: cuda12-liquid-audio - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:latest-gpu-nvidia-cuda-12-liquid-audio - name: cuda12-liquid-audio - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-latest.ca554511dd39 -- alias: liquid-audio - arch: amd64 - artifact: oci://quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: development - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: cuda12-liquid-audio-development - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:master-gpu-nvidia-cuda-12-liquid-audio - mutable: true - name: cuda12-liquid-audio-development - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-development -- alias: liquid-audio - arch: amd64 - artifact: oci://quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: latest - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - digest: sha256:d1ec28a025abf0b7397d5dae2bfd08b5a864dc7c80a87359bd9071bdc8b6b063 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: cuda13-liquid-audio - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:latest-gpu-nvidia-cuda-13-liquid-audio - name: cuda13-liquid-audio - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-latest.d1ec28a025ab -- alias: liquid-audio - arch: amd64 - artifact: oci://quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: development - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: cuda13-liquid-audio-development - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:master-gpu-nvidia-cuda-13-liquid-audio - mutable: true - name: cuda13-liquid-audio-development - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-development -- alias: liquid-audio - arch: amd64 - artifact: oci://quay.io/go-skynet/local-ai-backends:latest-gpu-intel-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: latest - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - digest: sha256:e331eab911829e84d23634483beeef813e0fb334bc58743f95e3a9eba9db360b - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: intel-liquid-audio - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:latest-gpu-intel-liquid-audio - name: intel-liquid-audio - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-latest.e331eab91182 -- alias: liquid-audio - arch: amd64 - artifact: oci://quay.io/go-skynet/local-ai-backends:master-gpu-intel-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: development - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: intel-liquid-audio-development - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:master-gpu-intel-liquid-audio - mutable: true - name: intel-liquid-audio-development - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-development -- alias: liquid-audio - arch: amd64 - artifact: oci://quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: latest - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - digest: sha256:6fafe2ba12f0810292899fc37f35553ebdada0e073d7b5d7fcde2c8fcc3b50d4 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: rocm-liquid-audio - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:latest-gpu-rocm-hipblas-liquid-audio - name: rocm-liquid-audio - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-latest.6fafe2ba12f0 -- alias: liquid-audio - arch: amd64 - artifact: oci://quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: development - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: rocm-liquid-audio-development - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:master-gpu-rocm-hipblas-liquid-audio - mutable: true - name: rocm-liquid-audio-development - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-development -- alias: liquid-audio - arch: arm64 - artifact: oci://quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: latest - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - digest: sha256:fb521024116a1c2c69fcdc241922a25631c46555e42b3fcf66aa2d23da9506f2 - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: cuda13-nvidia-l4t-arm64-liquid-audio - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-liquid-audio - name: cuda13-nvidia-l4t-arm64-liquid-audio - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-latest.fb521024116a -- alias: liquid-audio - arch: arm64 - artifact: oci://quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-liquid-audio - capabilities: - amd: rocm-liquid-audio - default: cpu-liquid-audio - intel: intel-liquid-audio - metal: metal-liquid-audio - nvidia: cuda12-liquid-audio - nvidia-cuda-12: cuda12-liquid-audio - nvidia-cuda-13: cuda13-liquid-audio - nvidia-l4t-cuda-13: cuda13-nvidia-l4t-arm64-liquid-audio - channel: development - description: | - LiquidAI LFM2 / LFM2.5 Audio Python backend. End-to-end speech-to-speech, ASR, - TTS (4 baked voices), and text chat from a single 1.5B model. Wraps the - upstream `liquid-audio` package; supports fine-tuning via LocalAI's - /v1/fine-tuning/jobs endpoint. - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - id: cuda13-nvidia-l4t-arm64-liquid-audio-development - kind: backend - license: LFM-Open-License-v1.0 - mirrors: - - oci://localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-liquid-audio - mutable: true - name: cuda13-nvidia-l4t-arm64-liquid-audio-development - os: linux - schemaVersion: 1 - tags: - - speech-to-speech - - any-to-any - - text-to-speech - - speech-to-text - - TTS - - ASR - - realtime - urls: - - https://github.com/Liquid4All/liquid-audio - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - version: 0.0.0-development - alias: qwen-tts capabilities: amd: rocm-qwen-tts-development diff --git a/gallery/index.yaml b/gallery/index.yaml index 056ffd9b1..c518f20ca 100644 --- a/gallery/index.yaml +++ b/gallery/index.yaml @@ -29718,103 +29718,6 @@ model: llama-cpp/models/LFM2.5-1.2B-Nova-Function-Calling.Q4_K_M.gguf template: use_tokenizer_template: true -- url: github:mudler/LocalAI/gallery/liquid-audio.yaml@master - name: lfm2.5-audio-1.5b-realtime - description: | - LFM2.5-Audio-1.5B is LiquidAI's any-to-any audio foundation model. The - 1.2B LFM2.5 backbone plus a FastConformer audio encoder and an LFM2-based - audio detokenizer give real-time speech-to-speech with text + audio output - interleaved at 12.5 Hz / 24 kHz. This entry runs in S2S (speech-to-speech) - mode and is the model the LocalAI realtime API any-to-any path consumes. - Switch to ASR, TTS, or chat by picking the sibling gallery entries. - license: LFM-Open-License-v1.0 - urls: - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - tags: - - lfm2 - - liquid - - audio - - speech-to-speech - - any-to-any - - realtime - - 1.5b - overrides: - backend: liquid-audio - known_usecases: - - realtime_audio - - chat - - transcript - - tts - - vad - options: - - mode:s2s -- url: github:mudler/LocalAI/gallery/liquid-audio.yaml@master - name: lfm2.5-audio-1.5b-chat - description: | - LFM2.5-Audio-1.5B in text-only chat mode. The model runs `generate_sequential` - with no audio modality, behaving like a small LFM2 chat model. Pick this - entry for tool-calling experiments without the audio overhead. - license: LFM-Open-License-v1.0 - urls: - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - lfm2 - - liquid - - audio - - chat - - 1.5b - overrides: - backend: liquid-audio - known_usecases: - - chat - options: - - mode:chat -- url: github:mudler/LocalAI/gallery/liquid-audio.yaml@master - name: lfm2.5-audio-1.5b-asr - description: | - LFM2.5-Audio-1.5B in ASR mode. System prompt `Perform ASR.` is prepended; - output is capitalised and punctuated. Wire this entry as a transcription - model on the /v1/audio/transcriptions endpoint. - license: LFM-Open-License-v1.0 - urls: - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - lfm2 - - liquid - - audio - - asr - - speech-to-text - - 1.5b - overrides: - backend: liquid-audio - known_usecases: - - transcript - options: - - mode:asr -- url: github:mudler/LocalAI/gallery/liquid-audio.yaml@master - name: lfm2.5-audio-1.5b-tts - description: | - LFM2.5-Audio-1.5B in TTS mode. Four baked voices: us_male, us_female, - uk_male, uk_female — pick the default at load time via `voice:` option, - or override per-request via the OpenAI `/v1/audio/speech` `voice` field. - license: LFM-Open-License-v1.0 - urls: - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - lfm2 - - liquid - - audio - - tts - - text-to-speech - - 1.5b - overrides: - backend: liquid-audio - known_usecases: - - tts - options: - - mode:tts - - voice:us_female - url: github:mudler/LocalAI/gallery/virtual.yaml@master name: mistral-nemo-instruct-2407-12b-thinking-m-claude-opus-high-reasoning-i1 description: | @@ -32044,6 +31947,448 @@ mmproj: mmproj-LFM2-VL-1.6B-F16.gguf parameters: model: LFM2-VL-1.6B-F16.gguf +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: lfm3-audio-2b + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q4_K_M weights with the Q8_0 audio projector). An any-to-any model: one llama.cpp session takes speech or text + and answers with speech, text or both, and the same model also transcribes speech (ASR, with a language + hint) and synthesises speech with the named voices female or male. + No VAD model is set: the client commits each turn with input_audio_buffer.commit. Use the -vad entry + for automatic turn detection. + Speech-to-speech over /v1/realtime is experimental and needs a LocalAI build whose llama.cpp pin includes the + LFM3 audio and realtime support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - any-to-any + - cpu + - gpu + files: + - filename: LFM3-Audio-2B-Q4_K_M.gguf + sha256: 9a773289bd321ed879cd3a76545f6aca506e403851a45fb315420aee45625be2 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q4_K_M.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - realtime_audio + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q4_K_M.gguf + pipeline: + any_to_any: true + template: + use_tokenizer_template: true + tts: + voice: female +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: lfm3-audio-2b-q8 + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q8_0 weights with the Q8_0 audio projector). An any-to-any model: one llama.cpp session takes speech or text + and answers with speech, text or both, and the same model also transcribes speech (ASR, with a language + hint) and synthesises speech with the named voices female or male. + No VAD model is set: the client commits each turn with input_audio_buffer.commit. Use the -vad entry + for automatic turn detection. + Speech-to-speech over /v1/realtime is experimental and needs a LocalAI build whose llama.cpp pin includes the + LFM3 audio and realtime support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - any-to-any + - cpu + - gpu + files: + - filename: LFM3-Audio-2B-Q8_0.gguf + sha256: d4e36902ba8f0ee7b333596030a59d650f88ac76660b4d9ba5ff77c3c2258fc0 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q8_0.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - realtime_audio + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q8_0.gguf + pipeline: + any_to_any: true + template: + use_tokenizer_template: true + tts: + voice: female +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: lfm3-audio-2b-vad + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q4_K_M weights with the Q8_0 audio projector). An any-to-any model: one llama.cpp session takes speech or text + and answers with speech, text or both, and the same model also transcribes speech (ASR, with a language + hint) and synthesises speech with the named voices female or male. + Server-side turn detection is on: the entry sets pipeline.vad to silero-vad-ggml, so install the + silero-vad-ggml model first (the gallery does not install it for you). The VAD only decides when a turn + ends; the model itself answers the speech. + Speech-to-speech over /v1/realtime is experimental and needs a LocalAI build whose llama.cpp pin includes the + LFM3 audio and realtime support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - any-to-any + - vad + - cpu + - gpu + files: + - filename: LFM3-Audio-2B-Q4_K_M.gguf + sha256: 9a773289bd321ed879cd3a76545f6aca506e403851a45fb315420aee45625be2 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q4_K_M.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - realtime_audio + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q4_K_M.gguf + pipeline: + any_to_any: true + vad: silero-vad-ggml + template: + use_tokenizer_template: true + tts: + voice: female +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: lfm3-audio-2b-q8-vad + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q8_0 weights with the Q8_0 audio projector). An any-to-any model: one llama.cpp session takes speech or text + and answers with speech, text or both, and the same model also transcribes speech (ASR, with a language + hint) and synthesises speech with the named voices female or male. + Server-side turn detection is on: the entry sets pipeline.vad to silero-vad-ggml, so install the + silero-vad-ggml model first (the gallery does not install it for you). The VAD only decides when a turn + ends; the model itself answers the speech. + Speech-to-speech over /v1/realtime is experimental and needs a LocalAI build whose llama.cpp pin includes the + LFM3 audio and realtime support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - any-to-any + - vad + - cpu + - gpu + files: + - filename: LFM3-Audio-2B-Q8_0.gguf + sha256: d4e36902ba8f0ee7b333596030a59d650f88ac76660b4d9ba5ff77c3c2258fc0 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q8_0.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - realtime_audio + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q8_0.gguf + pipeline: + any_to_any: true + vad: silero-vad-ggml + template: + use_tokenizer_template: true + tts: + voice: female +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: lfm3-audio-2b-pipeline + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q4_K_M weights with the Q8_0 audio projector), used as a chained realtime pipeline: transcription, + chat and speech synthesis (voice female) all run on this one llama.cpp model, one stage after the other, + instead of as one speech-to-speech session. Pick it when you want per-stage text, the regular chat + path (tools, templates, streaming) or the transcript before the answer. + No VAD model is set: the client commits each turn with input_audio_buffer.commit. Use the + -pipeline-vad entry for automatic turn detection. + The pipeline slots name this entry, so keep the installed model name as it is. + Needs a LocalAI build whose llama.cpp pin includes the LFM3 audio support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - pipeline + - cpu + - gpu + files: + - filename: LFM3-Audio-2B-Q4_K_M.gguf + sha256: 9a773289bd321ed879cd3a76545f6aca506e403851a45fb315420aee45625be2 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q4_K_M.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q4_K_M.gguf + pipeline: + llm: lfm3-audio-2b-pipeline + transcription: lfm3-audio-2b-pipeline + tts: lfm3-audio-2b-pipeline + template: + use_tokenizer_template: true + tts: + voice: female +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: lfm3-audio-2b-q8-pipeline + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q8_0 weights with the Q8_0 audio projector), used as a chained realtime pipeline: transcription, + chat and speech synthesis (voice female) all run on this one llama.cpp model, one stage after the other, + instead of as one speech-to-speech session. Pick it when you want per-stage text, the regular chat + path (tools, templates, streaming) or the transcript before the answer. + No VAD model is set: the client commits each turn with input_audio_buffer.commit. Use the + -pipeline-vad entry for automatic turn detection. + The pipeline slots name this entry, so keep the installed model name as it is. + Needs a LocalAI build whose llama.cpp pin includes the LFM3 audio support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - pipeline + - cpu + - gpu + files: + - filename: LFM3-Audio-2B-Q8_0.gguf + sha256: d4e36902ba8f0ee7b333596030a59d650f88ac76660b4d9ba5ff77c3c2258fc0 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q8_0.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q8_0.gguf + pipeline: + llm: lfm3-audio-2b-q8-pipeline + transcription: lfm3-audio-2b-q8-pipeline + tts: lfm3-audio-2b-q8-pipeline + template: + use_tokenizer_template: true + tts: + voice: female +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: lfm3-audio-2b-pipeline-vad + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q4_K_M weights with the Q8_0 audio projector), used as a chained realtime pipeline: transcription, + chat and speech synthesis (voice female) all run on this one llama.cpp model, one stage after the other, + instead of as one speech-to-speech session. Pick it when you want per-stage text, the regular chat + path (tools, templates, streaming) or the transcript before the answer. + Server-side turn detection is on: the entry sets pipeline.vad to silero-vad-ggml, so install the + silero-vad-ggml model first (the gallery does not install it for you). + The pipeline slots name this entry, so keep the installed model name as it is. + Needs a LocalAI build whose llama.cpp pin includes the LFM3 audio support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - pipeline + - vad + - cpu + - gpu + files: + - filename: LFM3-Audio-2B-Q4_K_M.gguf + sha256: 9a773289bd321ed879cd3a76545f6aca506e403851a45fb315420aee45625be2 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q4_K_M.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q4_K_M.gguf + pipeline: + llm: lfm3-audio-2b-pipeline-vad + transcription: lfm3-audio-2b-pipeline-vad + tts: lfm3-audio-2b-pipeline-vad + vad: silero-vad-ggml + template: + use_tokenizer_template: true + tts: + voice: female +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: lfm3-audio-2b-q8-pipeline-vad + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q8_0 weights with the Q8_0 audio projector), used as a chained realtime pipeline: transcription, + chat and speech synthesis (voice female) all run on this one llama.cpp model, one stage after the other, + instead of as one speech-to-speech session. Pick it when you want per-stage text, the regular chat + path (tools, templates, streaming) or the transcript before the answer. + Server-side turn detection is on: the entry sets pipeline.vad to silero-vad-ggml, so install the + silero-vad-ggml model first (the gallery does not install it for you). + The pipeline slots name this entry, so keep the installed model name as it is. + Needs a LocalAI build whose llama.cpp pin includes the LFM3 audio support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - pipeline + - vad + - cpu + - gpu + files: + - filename: LFM3-Audio-2B-Q8_0.gguf + sha256: d4e36902ba8f0ee7b333596030a59d650f88ac76660b4d9ba5ff77c3c2258fc0 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q8_0.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q8_0.gguf + pipeline: + llm: lfm3-audio-2b-q8-pipeline-vad + transcription: lfm3-audio-2b-q8-pipeline-vad + tts: lfm3-audio-2b-q8-pipeline-vad + vad: silero-vad-ggml + template: + use_tokenizer_template: true + tts: + voice: female - url: github:mudler/LocalAI/gallery/lfm.yaml@master name: lfm2.5-vl-1.6b description: | diff --git a/gallery/liquid-audio.yaml b/gallery/liquid-audio.yaml deleted file mode 100644 index c2e41b644..000000000 --- a/gallery/liquid-audio.yaml +++ /dev/null @@ -1,46 +0,0 @@ ---- -name: "liquid-audio" - -description: | - LiquidAI LFM2 / LFM2.5 Audio models served by the Python `liquid-audio` backend. - Supports four roles via the `mode:` option: - - chat text-only chat completion (generate_sequential, no audio) - - asr speech-to-text (Perform ASR. system prompt) - - tts text-to-speech in 4 baked voices (us_male/us_female/uk_male/uk_female) - - s2s interleaved speech-to-speech (the realtime any-to-any path) - -license: "LFM Open License v1.0" - -urls: - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - - https://github.com/Liquid4All/liquid-audio - -config_file: | - backend: liquid-audio - context_size: 32768 - f16: true - mmap: true - # realtime_audio surfaces the model on the Talk page; chat/tts/transcript - # let it also serve the standalone /v1/chat/completions, /v1/audio/speech, - # and /v1/audio/transcriptions endpoints (backend implements all three). - known_usecases: - - realtime_audio - - chat - - tts - - transcript - - vad - parameters: - model: LiquidAI/LFM2.5-Audio-1.5B - artifacts: - - name: model - target: model - source: - type: huggingface - repo: LiquidAI/LFM2.5-Audio-1.5B - # Special tokens emitted in the text track during interleaved generation. - # Included so a future client-side parser can spot them; the LFM2 tool-call - # format itself is auto-detected by the upstream llama.cpp parser when the - # model loads under that backend. - stopwords: - - <|im_end|> - - <|endoftext|> diff --git a/gallery/models/index.yaml b/gallery/models/index.yaml index 6ef5d31a1..73ca28fc0 100644 --- a/gallery/models/index.yaml +++ b/gallery/models/index.yaml @@ -34831,135 +34831,6 @@ model: llama-cpp/models/LFM2.5-1.2B-Nova-Function-Calling.Q4_K_M.gguf template: use_tokenizer_template: true -- kind: model - id: lfm2.5-audio-1.5b-realtime - name: lfm2.5-audio-1.5b-realtime - version: 0.0.0-gallery.5cec0853616f - description: | - LFM2.5-Audio-1.5B is LiquidAI's any-to-any audio foundation model. The - 1.2B LFM2.5 backbone plus a FastConformer audio encoder and an LFM2-based - audio detokenizer give real-time speech-to-speech with text + audio output - interleaved at 12.5 Hz / 24 kHz. This entry runs in S2S (speech-to-speech) - mode and is the model the LocalAI realtime API any-to-any path consumes. - Switch to ASR, TTS, or chat by picking the sibling gallery entries. - license: LFM-Open-License-v1.0 - urls: - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - lfm2 - - liquid - - audio - - speech-to-speech - - any-to-any - - realtime - - 1.5b - icon: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/7_6D7rWrLxp2hb6OHSV1p.png - channel: gallery - policy: - requireSignature: false - schemaVersion: 1 - model: - url: github:mudler/LocalAI/gallery/liquid-audio.yaml@master - overrides: - backend: liquid-audio - known_usecases: - - realtime_audio - - chat - - transcript - - tts - - vad - options: - - mode:s2s -- kind: model - id: lfm2.5-audio-1.5b-chat - name: lfm2.5-audio-1.5b-chat - version: 0.0.0-gallery.bbebd03a511b - description: | - LFM2.5-Audio-1.5B in text-only chat mode. The model runs `generate_sequential` - with no audio modality, behaving like a small LFM2 chat model. Pick this - entry for tool-calling experiments without the audio overhead. - license: LFM-Open-License-v1.0 - urls: - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - lfm2 - - liquid - - audio - - chat - - 1.5b - channel: gallery - policy: - requireSignature: false - schemaVersion: 1 - model: - url: github:mudler/LocalAI/gallery/liquid-audio.yaml@master - overrides: - backend: liquid-audio - known_usecases: - - chat - options: - - mode:chat -- kind: model - id: lfm2.5-audio-1.5b-asr - name: lfm2.5-audio-1.5b-asr - version: 0.0.0-gallery.cb76eb7d6450 - description: | - LFM2.5-Audio-1.5B in ASR mode. System prompt `Perform ASR.` is prepended; - output is capitalised and punctuated. Wire this entry as a transcription - model on the /v1/audio/transcriptions endpoint. - license: LFM-Open-License-v1.0 - urls: - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - lfm2 - - liquid - - audio - - asr - - speech-to-text - - 1.5b - channel: gallery - policy: - requireSignature: false - schemaVersion: 1 - model: - url: github:mudler/LocalAI/gallery/liquid-audio.yaml@master - overrides: - backend: liquid-audio - known_usecases: - - transcript - options: - - mode:asr -- kind: model - id: lfm2.5-audio-1.5b-tts - name: lfm2.5-audio-1.5b-tts - version: 0.0.0-gallery.737be4078fe1 - description: | - LFM2.5-Audio-1.5B in TTS mode. Four baked voices: us_male, us_female, - uk_male, uk_female — pick the default at load time via `voice:` option, - or override per-request via the OpenAI `/v1/audio/speech` `voice` field. - license: LFM-Open-License-v1.0 - urls: - - https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B - tags: - - lfm2 - - liquid - - audio - - tts - - text-to-speech - - 1.5b - channel: gallery - policy: - requireSignature: false - schemaVersion: 1 - model: - url: github:mudler/LocalAI/gallery/liquid-audio.yaml@master - overrides: - backend: liquid-audio - known_usecases: - - tts - options: - - mode:tts - - voice:us_female - kind: model id: mistral-nemo-instruct-2407-12b-thinking-m-claude-opus-high-reasoning-i1 name: mistral-nemo-instruct-2407-12b-thinking-m-claude-opus-high-reasoning-i1 @@ -37685,6 +37556,512 @@ mmproj: mmproj-LFM2-VL-1.6B-F16.gguf parameters: model: LFM2-VL-1.6B-F16.gguf +- kind: model + id: lfm3-audio-2b + name: lfm3-audio-2b + version: 0.0.0-gallery.000000000000 + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q4_K_M weights with the Q8_0 audio projector). An any-to-any model: one llama.cpp session takes speech or text + and answers with speech, text or both, and the same model also transcribes speech (ASR, with a language + hint) and synthesises speech with the named voices female or male. + No VAD model is set: the client commits each turn with input_audio_buffer.commit. Use the -vad entry + for automatic turn detection. + Speech-to-speech over /v1/realtime is experimental and needs a LocalAI build whose llama.cpp pin includes the + LFM3 audio and realtime support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - any-to-any + - cpu + - gpu + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: LFM3-Audio-2B-Q4_K_M.gguf + sha256: 9a773289bd321ed879cd3a76545f6aca506e403851a45fb315420aee45625be2 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q4_K_M.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - realtime_audio + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q4_K_M.gguf + pipeline: + any_to_any: true + template: + use_tokenizer_template: true + tts: + voice: female +- kind: model + id: lfm3-audio-2b-q8 + name: lfm3-audio-2b-q8 + version: 0.0.0-gallery.000000000000 + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q8_0 weights with the Q8_0 audio projector). An any-to-any model: one llama.cpp session takes speech or text + and answers with speech, text or both, and the same model also transcribes speech (ASR, with a language + hint) and synthesises speech with the named voices female or male. + No VAD model is set: the client commits each turn with input_audio_buffer.commit. Use the -vad entry + for automatic turn detection. + Speech-to-speech over /v1/realtime is experimental and needs a LocalAI build whose llama.cpp pin includes the + LFM3 audio and realtime support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - any-to-any + - cpu + - gpu + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: LFM3-Audio-2B-Q8_0.gguf + sha256: d4e36902ba8f0ee7b333596030a59d650f88ac76660b4d9ba5ff77c3c2258fc0 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q8_0.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - realtime_audio + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q8_0.gguf + pipeline: + any_to_any: true + template: + use_tokenizer_template: true + tts: + voice: female +- kind: model + id: lfm3-audio-2b-vad + name: lfm3-audio-2b-vad + version: 0.0.0-gallery.000000000000 + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q4_K_M weights with the Q8_0 audio projector). An any-to-any model: one llama.cpp session takes speech or text + and answers with speech, text or both, and the same model also transcribes speech (ASR, with a language + hint) and synthesises speech with the named voices female or male. + Server-side turn detection is on: the entry sets pipeline.vad to silero-vad-ggml, so install the + silero-vad-ggml model first (the gallery does not install it for you). The VAD only decides when a turn + ends; the model itself answers the speech. + Speech-to-speech over /v1/realtime is experimental and needs a LocalAI build whose llama.cpp pin includes the + LFM3 audio and realtime support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - any-to-any + - vad + - cpu + - gpu + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: LFM3-Audio-2B-Q4_K_M.gguf + sha256: 9a773289bd321ed879cd3a76545f6aca506e403851a45fb315420aee45625be2 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q4_K_M.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - realtime_audio + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q4_K_M.gguf + pipeline: + any_to_any: true + vad: silero-vad-ggml + template: + use_tokenizer_template: true + tts: + voice: female +- kind: model + id: lfm3-audio-2b-q8-vad + name: lfm3-audio-2b-q8-vad + version: 0.0.0-gallery.000000000000 + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q8_0 weights with the Q8_0 audio projector). An any-to-any model: one llama.cpp session takes speech or text + and answers with speech, text or both, and the same model also transcribes speech (ASR, with a language + hint) and synthesises speech with the named voices female or male. + Server-side turn detection is on: the entry sets pipeline.vad to silero-vad-ggml, so install the + silero-vad-ggml model first (the gallery does not install it for you). The VAD only decides when a turn + ends; the model itself answers the speech. + Speech-to-speech over /v1/realtime is experimental and needs a LocalAI build whose llama.cpp pin includes the + LFM3 audio and realtime support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - any-to-any + - vad + - cpu + - gpu + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: LFM3-Audio-2B-Q8_0.gguf + sha256: d4e36902ba8f0ee7b333596030a59d650f88ac76660b4d9ba5ff77c3c2258fc0 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q8_0.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - realtime_audio + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q8_0.gguf + pipeline: + any_to_any: true + vad: silero-vad-ggml + template: + use_tokenizer_template: true + tts: + voice: female +- kind: model + id: lfm3-audio-2b-pipeline + name: lfm3-audio-2b-pipeline + version: 0.0.0-gallery.000000000000 + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q4_K_M weights with the Q8_0 audio projector), used as a chained realtime pipeline: transcription, + chat and speech synthesis (voice female) all run on this one llama.cpp model, one stage after the other, + instead of as one speech-to-speech session. Pick it when you want per-stage text, the regular chat + path (tools, templates, streaming) or the transcript before the answer. + No VAD model is set: the client commits each turn with input_audio_buffer.commit. Use the + -pipeline-vad entry for automatic turn detection. + The pipeline slots name this entry, so keep the installed model name as it is. + Needs a LocalAI build whose llama.cpp pin includes the LFM3 audio support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - pipeline + - cpu + - gpu + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: LFM3-Audio-2B-Q4_K_M.gguf + sha256: 9a773289bd321ed879cd3a76545f6aca506e403851a45fb315420aee45625be2 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q4_K_M.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q4_K_M.gguf + pipeline: + llm: lfm3-audio-2b-pipeline + transcription: lfm3-audio-2b-pipeline + tts: lfm3-audio-2b-pipeline + template: + use_tokenizer_template: true + tts: + voice: female +- kind: model + id: lfm3-audio-2b-q8-pipeline + name: lfm3-audio-2b-q8-pipeline + version: 0.0.0-gallery.000000000000 + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q8_0 weights with the Q8_0 audio projector), used as a chained realtime pipeline: transcription, + chat and speech synthesis (voice female) all run on this one llama.cpp model, one stage after the other, + instead of as one speech-to-speech session. Pick it when you want per-stage text, the regular chat + path (tools, templates, streaming) or the transcript before the answer. + No VAD model is set: the client commits each turn with input_audio_buffer.commit. Use the + -pipeline-vad entry for automatic turn detection. + The pipeline slots name this entry, so keep the installed model name as it is. + Needs a LocalAI build whose llama.cpp pin includes the LFM3 audio support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - pipeline + - cpu + - gpu + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: LFM3-Audio-2B-Q8_0.gguf + sha256: d4e36902ba8f0ee7b333596030a59d650f88ac76660b4d9ba5ff77c3c2258fc0 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q8_0.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q8_0.gguf + pipeline: + llm: lfm3-audio-2b-q8-pipeline + transcription: lfm3-audio-2b-q8-pipeline + tts: lfm3-audio-2b-q8-pipeline + template: + use_tokenizer_template: true + tts: + voice: female +- kind: model + id: lfm3-audio-2b-pipeline-vad + name: lfm3-audio-2b-pipeline-vad + version: 0.0.0-gallery.000000000000 + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q4_K_M weights with the Q8_0 audio projector), used as a chained realtime pipeline: transcription, + chat and speech synthesis (voice female) all run on this one llama.cpp model, one stage after the other, + instead of as one speech-to-speech session. Pick it when you want per-stage text, the regular chat + path (tools, templates, streaming) or the transcript before the answer. + Server-side turn detection is on: the entry sets pipeline.vad to silero-vad-ggml, so install the + silero-vad-ggml model first (the gallery does not install it for you). + The pipeline slots name this entry, so keep the installed model name as it is. + Needs a LocalAI build whose llama.cpp pin includes the LFM3 audio support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - pipeline + - vad + - cpu + - gpu + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: LFM3-Audio-2B-Q4_K_M.gguf + sha256: 9a773289bd321ed879cd3a76545f6aca506e403851a45fb315420aee45625be2 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q4_K_M.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q4_K_M.gguf + pipeline: + llm: lfm3-audio-2b-pipeline-vad + transcription: lfm3-audio-2b-pipeline-vad + tts: lfm3-audio-2b-pipeline-vad + vad: silero-vad-ggml + template: + use_tokenizer_template: true + tts: + voice: female +- kind: model + id: lfm3-audio-2b-q8-pipeline-vad + name: lfm3-audio-2b-q8-pipeline-vad + version: 0.0.0-gallery.000000000000 + description: | + LiquidAI LFM3-Audio-2B in GGUF (Q8_0 weights with the Q8_0 audio projector), used as a chained realtime pipeline: transcription, + chat and speech synthesis (voice female) all run on this one llama.cpp model, one stage after the other, + instead of as one speech-to-speech session. Pick it when you want per-stage text, the regular chat + path (tools, templates, streaming) or the transcript before the answer. + Server-side turn detection is on: the entry sets pipeline.vad to silero-vad-ggml, so install the + silero-vad-ggml model first (the gallery does not install it for you). + The pipeline slots name this entry, so keep the installed model name as it is. + Needs a LocalAI build whose llama.cpp pin includes the LFM3 audio support. + Licensed under the LFM Open License v1.0 (https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/blob/main/LICENSE). + license: lfm1.0 + urls: + - https://huggingface.co/LiquidAI/LFM3-Audio-2B + - https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF + tags: + - gguf + - audio + - speech + - tts + - asr + - realtime + - pipeline + - vad + - cpu + - gpu + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: LFM3-Audio-2B-Q8_0.gguf + sha256: d4e36902ba8f0ee7b333596030a59d650f88ac76660b4d9ba5ff77c3c2258fc0 + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/LFM3-Audio-2B-Q8_0.gguf + - filename: mmproj-LFM3-Audio-2B-Q8_0.gguf + sha256: 26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e + uri: https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c/mmproj-LFM3-Audio-2B-Q8_0.gguf + overrides: + backend: llama-cpp + context_size: 8192 + function: + xml_format_preset: lfm3 + known_usecases: + - chat + - transcript + - tts + mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf + options: + - parallel:2 + parameters: + model: LFM3-Audio-2B-Q8_0.gguf + pipeline: + llm: lfm3-audio-2b-q8-pipeline-vad + transcription: lfm3-audio-2b-q8-pipeline-vad + tts: lfm3-audio-2b-q8-pipeline-vad + vad: silero-vad-ggml + template: + use_tokenizer_template: true + tts: + voice: female - kind: model id: lfm2.5-vl-1.6b name: lfm2.5-vl-1.6b diff --git a/pkg/functions/iterative_parser.go b/pkg/functions/iterative_parser.go index cfcf46812..f21fe5538 100644 --- a/pkg/functions/iterative_parser.go +++ b/pkg/functions/iterative_parser.go @@ -44,6 +44,17 @@ type ChatMsgParser struct { content strings.Builder reasoning strings.Builder toolCalls []FuncCallResults + + // checkpoint is the state after the last complete scope block, set once + // at least one call was parsed. A later block that does not parse is + // dropped back to it: the model may mention the tag in prose. + checkpoint *parseCheckpoint +} + +type parseCheckpoint struct { + pos int + calls int + content string } // NewChatMsgParser creates a new iterative parser @@ -628,7 +639,36 @@ func removeHealingMarkerFromJSONAny(value any, marker string) any { // TryConsumeXMLToolCalls attempts to parse XML tool calls using the iterative parser // Returns true if tool calls were found and parsed, false otherwise // Similar to llama.cpp's parse_xml_tool_calls +// +// Once a call was parsed, text that follows it and looks like a further block +// but is not one (a prose mention of the opening tag, a malformed block) does +// not discard the calls already found. They are kept, the parser stops before +// that text and the text stays in the content. func (p *ChatMsgParser) TryConsumeXMLToolCalls(format *XMLToolCallFormat) (bool, error) { + p.checkpoint = nil + ok, err := p.tryConsumeXMLToolCalls(format) + cp := p.checkpoint + p.checkpoint = nil + if cp == nil || len(p.toolCalls) == 0 { + return ok, err + } + if err != nil { + // A partial exception while streaming is a block still being + // written: keep it so the caller sees the call grow. + if _, partial := err.(*ChatMsgPartialException); partial && p.isPartial { + return ok, err + } + } else if ok { + return ok, err + } + p.toolCalls = p.toolCalls[:cp.calls] + p.content.Reset() + p.content.WriteString(cp.content) + p.pos = cp.pos + return len(p.toolCalls) > 0, nil +} + +func (p *ChatMsgParser) tryConsumeXMLToolCalls(format *XMLToolCallFormat) (bool, error) { if format == nil { return false, errors.New("format is required") } @@ -753,6 +793,9 @@ func (p *ChatMsgParser) TryConsumeXMLToolCalls(format *XMLToolCallFormat) (bool, // Parse multiple scopes (for formats like qwen3-coder that can have multiple blocks) // Continue parsing until no more scopes are found for { + if format.ScopeStart != "" && len(p.toolCalls) > 0 { + p.checkpoint = &parseCheckpoint{pos: p.pos, calls: len(p.toolCalls), content: p.content.String()} + } // Parse scope_start if present if format.ScopeStart != "" && !AllSpace(format.ScopeStart) { tc := p.TryFindLiteral(format.ScopeStart) @@ -760,11 +803,20 @@ func (p *ChatMsgParser) TryConsumeXMLToolCalls(format *XMLToolCallFormat) (bool, // No more scopes found, break break } - if !allSpaceOrEscapedNewlines(tc.Prelude) { - // Non-whitespace before scope_start, stop parsing + if !allSpaceOrEscapedNewlines(tc.Prelude) && len(p.ToolCalls()) == 0 { + // Non-whitespace before the first scope_start, stop parsing. + // Text between two blocks is skipped: a model may comment on + // its calls, and each block is a call of its own. p.MoveTo(tc.Groups[0].Begin - len(tc.Prelude)) break } + if text := strings.TrimSpace(tc.Prelude); text != "" && !allSpaceOrEscapedNewlines(tc.Prelude) { + // Keep the skipped text in the content, in order. + if p.content.Len() > 0 { + p.content.WriteString(" ") + } + p.content.WriteString(text) + } // Validate size match (partial detection) if len(tc.Groups) > 0 { matchedSize := tc.Groups[0].End - tc.Groups[0].Begin diff --git a/pkg/functions/parse.go b/pkg/functions/parse.go index c5dc4d406..4d3f7973c 100644 --- a/pkg/functions/parse.go +++ b/pkg/functions/parse.go @@ -104,7 +104,7 @@ type FunctionsConfig struct { FunctionNameKey string `yaml:"function_name_key,omitempty" json:"function_name_key,omitempty"` FunctionArgumentsKey string `yaml:"function_arguments_key,omitempty" json:"function_arguments_key,omitempty"` - // XMLFormatPreset is an optional preset format name to force (e.g., "qwen3-coder", "glm-4.5", "minimax-m2") + // XMLFormatPreset is an optional preset format name to force (e.g., "qwen3-coder", "glm-4.5", "minimax-m2", "lfm3") // If empty, auto-detection will try all formats XMLFormatPreset string `yaml:"xml_format_preset,omitempty" json:"xml_format_preset,omitempty"` // XMLFormat is an optional custom XML format configuration @@ -478,6 +478,23 @@ func getAllXMLFormats() []xmlFormatPreset { TrimRawArgVal: true, }, }, + { + // LFM3: value, + // the format its chat template writes. Strings are raw text, other + // values are JSON, which is what the raw-or-JSON argument parser reads. + name: "lfm3", + format: &XMLToolCallFormat{ + ScopeStart: "", + ToolStart: "", + KeyStart: "", + ValEnd: "", + ToolEnd: "", + ScopeEnd: "", + TrimRawArgVal: true, + }, + }, { name: "glm-4.5", format: &XMLToolCallFormat{ diff --git a/pkg/functions/peg/chat.go b/pkg/functions/peg/chat.go index 927a83c73..02a2e7117 100644 --- a/pkg/functions/peg/chat.go +++ b/pkg/functions/peg/chat.go @@ -738,9 +738,10 @@ func (m *ChatPegMapper) mapNode(node *AstNode) { // Try to parse as JSON var parsed json.RawMessage if err := json.Unmarshal([]byte(content), &parsed); err == nil { - // Check if it's a string + // Check if it's a string. A null also unmarshals into a + // string without an error, so look at the first byte. var s string - if err2 := json.Unmarshal(parsed, &s); err2 == nil { + if err2 := json.Unmarshal(parsed, &s); err2 == nil && parsed[0] == '"' { // It's a string — strip closing quote for monotonic streaming escaped, _ := json.Marshal(s) str := string(escaped) diff --git a/pkg/functions/peg_integration_test.go b/pkg/functions/peg_integration_test.go index 581195524..f320365bc 100644 --- a/pkg/functions/peg_integration_test.go +++ b/pkg/functions/peg_integration_test.go @@ -299,3 +299,190 @@ var _ = Describe("PEG Integration", func() { }) }) }) + +// The LFM3 chat template writes a call as +// +// \n\n\nVALUE\n\n\n +// +// A string value is raw text. Anything else is rendered as JSON, booleans as +// true and false, null as null. +var _ = Describe("LFM3 tool calls", func() { + lfm3 := FunctionsConfig{XMLFormatPreset: "lfm3"} + + call := func(name string, params ...string) string { + s := "\n\n" + for i := 0; i+1 < len(params); i += 2 { + s += "\n" + params[i+1] + "\n\n" + } + return s + "\n" + } + + It("has a preset of its own", func() { + Expect(GetXMLFormatPreset("lfm3")).NotTo(BeNil()) + }) + + It("parses one call", func() { + results := ParseFunctionCall(call("get_time", "tz", "UTC"), lfm3) + Expect(results).To(HaveLen(1)) + Expect(results[0].Name).To(Equal("get_time")) + Expect(results[0].Arguments).To(MatchJSON(`{"tz":"UTC"}`)) + }) + + It("parses a call without arguments", func() { + results := ParseFunctionCall(call("list_models"), lfm3) + Expect(results).To(HaveLen(1)) + Expect(results[0].Name).To(Equal("list_models")) + Expect(results[0].Arguments).To(MatchJSON(`{}`)) + }) + + It("parses two calls, separated by the blank line the template writes", func() { + results := ParseFunctionCall(call("get_time", "tz", "UTC")+"\n\n"+call("get_date", "tz", "CET"), lfm3) + Expect(results).To(HaveLen(2)) + Expect(results[0].Name).To(Equal("get_time")) + Expect(results[0].Arguments).To(MatchJSON(`{"tz":"UTC"}`)) + Expect(results[1].Name).To(Equal("get_date")) + Expect(results[1].Arguments).To(MatchJSON(`{"tz":"CET"}`)) + }) + + It("keeps the order of consecutive blocks and of blocks with text between them, with the preset and without", func() { + inputs := map[string]string{ + "back to back": call("first", "n", "1") + call("second", "n", "2") + call("third", "n", "3"), + "with text": call("first", "n", "1") + "\nThen the next one.\n" + call("second", "n", "2") + "\nAnd the last.\n" + call("third", "n", "3"), + } + for name, input := range inputs { + for _, cfg := range []FunctionsConfig{lfm3, {}} { + results := ParseFunctionCall(input, cfg) + Expect(results).To(HaveLen(3), name) + for i, want := range []string{"first", "second", "third"} { + Expect(results[i].Name).To(Equal(want), name) + } + Expect(results[2].Arguments).To(MatchJSON(`{"n": 3}`), name) + } + } + }) + + // The parser has no tool schema, so a value that reads as JSON is taken as + // JSON. llama.cpp keeps a parameter the schema types as a string raw. + It("takes a value that reads as JSON for JSON, whatever the schema types it as", func() { + results := ParseFunctionCall(call("f", "code", "42", "flag", "true", "name", "Bob"), lfm3) + Expect(results).To(HaveLen(1)) + Expect(results[0].Arguments).To(MatchJSON(`{"code": 42, "flag": true, "name": "Bob"}`)) + }) + + It("keeps a string as text and reads the other values as JSON", func() { + results := ParseFunctionCall(call("book", + "title", "The Hobbit", + "count", "3", + "ratio", "0.5", + "urgent", "true", + "note", "null", + "tags", `["a", "b"]`, + "opts", `{"x": 1}`, + ), lfm3) + Expect(results).To(HaveLen(1)) + Expect(results[0].Arguments).To(MatchJSON(`{ + "title": "The Hobbit", + "count": 3, + "ratio": 0.5, + "urgent": true, + "note": null, + "tags": ["a", "b"], + "opts": {"x": 1} + }`)) + }) + + It("reads a value of digits as a number", func() { + results := ParseFunctionCall(call("f", "zip", "10115"), lfm3) + Expect(results).To(HaveLen(1)) + Expect(results[0].Arguments).To(MatchJSON(`{"zip": 10115}`)) + }) + + It("keeps a multi-line string value", func() { + results := ParseFunctionCall(call("write", "text", "line one\nline two"), lfm3) + Expect(results).To(HaveLen(1)) + Expect(results[0].Arguments).To(MatchJSON(`{"text": "line one\nline two"}`)) + }) + + Context("when text after a valid call is not another call", func() { + inputs := map[string]string{ + "a mention of the opening tag": call("a") + "\nI used the tag to call a.", + "a markdown example": call("a") + "\nThe format is:\n```\n\n\n```\nDone.", + "a malformed second block": call("a") + "\nThen\n\n\n\n1\n", + "a second block without name": call("a") + "\n\nnothing here", + } + for name, input := range inputs { + It("keeps the first call after "+name, func() { + for _, cfg := range []FunctionsConfig{lfm3, {}, {XMLFormatPreset: "qwen3-coder"}} { + results := ParseFunctionCall(input, cfg) + Expect(results).To(HaveLen(1), name) + Expect(results[0].Name).To(Equal("a"), name) + } + }) + } + + It("keeps that text in the content", func() { + input := call("a") + "\nI used the tag to call a." + Expect(StripToolCallMarkup(input)).To(ContainSubstring("I used the")) + Expect(StripToolCallMarkup(input)).To(ContainSubstring("to call a.")) + }) + }) + + It("keeps the text between two blocks when the markup is stripped", func() { + input := "Checking.\n" + call("a") + "\nNow the second one.\n" + call("b") + "\nDone." + Expect(StripToolCallMarkup(input)).To(Equal("Checking. Now the second one. Done.")) + }) + + It("finds a call between text", func() { + input := "Let me check.\n" + call("get_time", "tz", "UTC") + "\nDone." + results := ParseFunctionCall(input, lfm3) + Expect(results).To(HaveLen(1)) + Expect(results[0].Name).To(Equal("get_time")) + }) + + It("finds nothing in text without a call", func() { + Expect(ParseFunctionCall("It is noon.", lfm3)).To(BeEmpty()) + }) + + It("is found without the preset", func() { + results := ParseFunctionCall(call("get_time", "tz", "UTC"), FunctionsConfig{}) + Expect(results).To(HaveLen(1)) + Expect(results[0].Name).To(Equal("get_time")) + }) + + Context("while the text is still coming", func() { + It("reports an unfinished call as partial", func() { + full := call("get_time", "tz", "UTC") + format := GetXMLFormatPreset("lfm3") + // every cut inside the block is partial, and no cut gives a wrong name + for cut := len("\n\n\nUTC\n\n"); cut < len(full); cut++ { + parser := NewChatMsgParser(full[:cut], true) + _, err := parser.TryConsumeXMLToolCalls(format) + if err != nil { + _, isPartial := err.(*ChatMsgPartialException) + Expect(isPartial).To(BeTrue(), "cut at %d: %v", cut, err) + } + for _, tc := range parser.ToolCalls() { + Expect(tc.Name).To(Equal("get_time")) + } + } + }) + + It("parses the first call while the second is unfinished", func() { + input := call("get_time", "tz", "UTC") + "\n\n\n\n\nCE" + parser := NewChatMsgParser(input, true) + _, _ = parser.TryConsumeXMLToolCalls(GetXMLFormatPreset("lfm3")) + Expect(parser.ToolCalls()).NotTo(BeEmpty()) + Expect(parser.ToolCalls()[0].Name).To(Equal("get_time")) + }) + + It("parses the full text once it is complete", func() { + full := call("get_time", "tz", "UTC") + parser := NewChatMsgParser(full, false) + ok, err := parser.TryConsumeXMLToolCalls(GetXMLFormatPreset("lfm3")) + Expect(err).NotTo(HaveOccurred()) + Expect(ok).To(BeTrue()) + Expect(parser.ToolCalls()).To(HaveLen(1)) + Expect(parser.ToolCalls()[0].Name).To(Equal("get_time")) + }) + }) +}) diff --git a/swagger/docs.go b/swagger/docs.go index c682fdcb0..8e4531fd0 100644 --- a/swagger/docs.go +++ b/swagger/docs.go @@ -5691,7 +5691,7 @@ const docTemplate = `{ }, "/audio/transformations/stream": { "get": { - "description": "Streams binary PCM frames in (interleaved stereo: ch0=audio, ch1=reference) and out (mono). The model must support the audio_transform use case. Any-to-any models such as liquid-audio use the OpenAI Realtime API instead. The first message must be a JSON ` + "`" + `session.update` + "`" + ` envelope describing model + sample format + frame size + backend params. Server emits binary PCM on the same cadence.", + "description": "Streams binary PCM frames in (interleaved stereo: ch0=audio, ch1=reference) and out (mono). The model must support the audio_transform use case. Any-to-any realtime_audio models use the OpenAI Realtime API instead. The first message must be a JSON ` + "`" + `session.update` + "`" + ` envelope describing model + sample format + frame size + backend params. Server emits binary PCM on the same cadence.", "tags": [ "audio" ], diff --git a/swagger/swagger.json b/swagger/swagger.json index d2665ae6d..7e1fafadd 100644 --- a/swagger/swagger.json +++ b/swagger/swagger.json @@ -5688,7 +5688,7 @@ }, "/audio/transformations/stream": { "get": { - "description": "Streams binary PCM frames in (interleaved stereo: ch0=audio, ch1=reference) and out (mono). The model must support the audio_transform use case. Any-to-any models such as liquid-audio use the OpenAI Realtime API instead. The first message must be a JSON `session.update` envelope describing model + sample format + frame size + backend params. Server emits binary PCM on the same cadence.", + "description": "Streams binary PCM frames in (interleaved stereo: ch0=audio, ch1=reference) and out (mono). The model must support the audio_transform use case. Any-to-any realtime_audio models use the OpenAI Realtime API instead. The first message must be a JSON `session.update` envelope describing model + sample format + frame size + backend params. Server emits binary PCM on the same cadence.", "tags": [ "audio" ], diff --git a/swagger/swagger.yaml b/swagger/swagger.yaml index 77b4c6541..cd3b76743 100644 --- a/swagger/swagger.yaml +++ b/swagger/swagger.yaml @@ -9815,10 +9815,9 @@ paths: get: description: 'Streams binary PCM frames in (interleaved stereo: ch0=audio, ch1=reference) and out (mono). The model must support the audio_transform use case. Any-to-any - models such as liquid-audio use the OpenAI Realtime API instead. The first - message must be a JSON `session.update` envelope describing model + sample - format + frame size + backend params. Server emits binary PCM on the same - cadence.' + realtime_audio models use the OpenAI Realtime API instead. The first message + must be a JSON `session.update` envelope describing model + sample format + + frame size + backend params. Server emits binary PCM on the same cadence.' responses: {} summary: Bidirectional realtime audio transform over WebSocket. tags: diff --git a/tests/e2e-backends/backend_test.go b/tests/e2e-backends/backend_test.go index 4afda646a..05891803b 100644 --- a/tests/e2e-backends/backend_test.go +++ b/tests/e2e-backends/backend_test.go @@ -3,6 +3,7 @@ package e2ebackends_test import ( "context" "encoding/base64" + "encoding/json" "fmt" "io" "net" @@ -13,7 +14,9 @@ import ( "time" "unicode/utf8" + laudio "github.com/mudler/LocalAI/pkg/audio" pb "github.com/mudler/LocalAI/pkg/grpc/proto" + "github.com/mudler/LocalAI/pkg/sound" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" "github.com/phayes/freeport" @@ -93,6 +96,20 @@ import ( // BACKEND_TEST_TTS_TEXT Override the text synthesized by the tts/ttsstream // specs (default: "The quick brown fox jumps over the // lazy dog."). +// BACKEND_TEST_TTS_MIN_SECONDS, BACKEND_TEST_TTS_MAX_SECONDS, +// BACKEND_TEST_TTS_MIN_RMS +// When any is set, the tts spec also decodes the WAV +// it wrote and asserts the duration is inside the +// range and the s16 RMS is at least the floor. This +// separates real speech from a silent or runaway +// (missing EOS) output. +// BACKEND_TEST_TRANSCRIPT_CONTAINS +// Comma-separated words; the transcription spec asserts +// the text contains at least one (case-insensitive), +// e.g. "country" for the jfk.wav sample. +// The "s2s" capability exercises AudioToAudioStream +// with BACKEND_TEST_AUDIO_* as the spoken input +// (a text turn with a tool, then its result). // // The suite is intentionally model-format-agnostic: it only ever passes the // file path to LoadModel, so GGUF, ONNX, safetensors, .bin etc. all work so @@ -107,6 +124,7 @@ const ( capTools = "tools" capTranscription = "transcription" capTTS = "tts" + capS2S = "s2s" capImage = "image" capFaceDetect = "face_detect" capFaceEmbed = "face_embed" @@ -826,6 +844,10 @@ var _ = Describe("Backend container", Ordered, func() { "AudioTranscription returned empty text") GinkgoWriter.Printf("AudioTranscription: text=%q language=%q duration=%v\n", res.GetText(), res.GetLanguage(), res.GetDuration()) + if want := os.Getenv("BACKEND_TEST_TRANSCRIPT_CONTAINS"); want != "" { + Expect(containsAnyFold(res.GetText(), strings.Split(want, ",")...)).To(BeTrue(), + "transcript %q contains none of %q", res.GetText(), want) + } }) It("streams audio transcription via AudioTranscriptionStream", func() { @@ -1186,6 +1208,7 @@ var _ = Describe("Backend container", Ordered, func() { Expect(info.Size()).To(BeNumerically(">", int64(1024)), "TTS output too small: %d bytes", info.Size()) GinkgoWriter.Printf("TTS: wrote %s (%d bytes)\n", dst, info.Size()) + assertTTSAudio(dst) }) It("streams PCM via TTSStream", func() { @@ -1222,6 +1245,167 @@ var _ = Describe("Backend container", Ordered, func() { GinkgoWriter.Printf("TTSStream: %d chunks, %d bytes\n", chunks, totalBytes) }) + It("runs a speech-to-speech session via AudioToAudioStream", func() { + if !caps[capS2S] { + Skip("s2s capability not enabled") + } + Expect(audioFile).NotTo(BeEmpty(), + "BACKEND_TEST_AUDIO_FILE or BACKEND_TEST_AUDIO_URL must be set when s2s cap is enabled") + raw, err := os.ReadFile(audioFile) + Expect(err).NotTo(HaveOccurred()) + pcm, rate := laudio.ParseWAV(raw) + Expect(rate).To(BeNumerically(">", 0), "%s is not a WAV file", audioFile) + samples := sound.ResampleInt16(sound.BytesToInt16sLE(pcm), rate, 16000) + input := sound.Int16toBytesLE(samples) + + ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute) + defer cancel() + stream, err := client.AudioToAudioStream(ctx) + Expect(err).NotTo(HaveOccurred()) + + tools, err := json.Marshal([]map[string]any{{ + "type": "function", + "function": map[string]any{ + "name": "get_time", + "description": "Returns the current time.", + "parameters": map[string]any{"type": "object", "properties": map[string]any{}}, + }, + }}) + Expect(err).NotTo(HaveOccurred()) + send := func(req *pb.AudioToAudioRequest) { + GinkgoHelper() + Expect(stream.Send(req)).To(Succeed()) + } + control := func(event string, payload any) { + GinkgoHelper() + b, err := json.Marshal(payload) + Expect(err).NotTo(HaveOccurred()) + send(&pb.AudioToAudioRequest{Payload: &pb.AudioToAudioRequest_Control{ + Control: &pb.AudioToAudioControl{Event: event, Payload: b}}}) + } + + // A single reader goroutine, so Recv is never called concurrently. + type s2sEvent struct { + event string + pcm []byte + meta map[string]any + err error + } + events := make(chan s2sEvent, 1024) + go func() { + defer GinkgoRecover() + defer close(events) + for { + r, err := stream.Recv() + if err != nil { + if err != io.EOF { + events <- s2sEvent{err: err} + } + return + } + e := s2sEvent{event: r.GetEvent(), pcm: r.GetPcm()} + if len(r.GetMeta()) > 0 { + _ = json.Unmarshal(r.GetMeta(), &e.meta) + } + events <- e + } + }() + // collect reads until response.done and returns the events of that response. + collect := func() []s2sEvent { + GinkgoHelper() + var got []s2sEvent + timeout := time.After(10 * time.Minute) + for { + select { + case e, ok := <-events: + Expect(ok).To(BeTrue(), "stream closed before response.done") + Expect(e.err).NotTo(HaveOccurred()) + Expect(e.event).NotTo(Equal("error"), "backend error: %v", e.meta) + got = append(got, e) + if e.event == "response.done" { + return got + } + case <-timeout: + Fail("timed out waiting for response.done") + } + } + } + + send(&pb.AudioToAudioRequest{Payload: &pb.AudioToAudioRequest_Config{Config: &pb.AudioToAudioConfig{ + InputSampleRate: 16000, + OutputSampleRate: 24000, + Tools: string(tools), + }}}) + created := <-events + Expect(created.err).NotTo(HaveOccurred()) + Expect(created.event).To(Equal("session.created")) + Expect(created.meta["input_sample_rate"]).To(BeNumerically("==", 16000)) + Expect(created.meta["output_sample_rate"]).To(BeNumerically("==", 24000)) + + // Turn 1: spoken input in 20 ms frames, commit, then ask for a bounded reply. + frame := 16000 * 20 / 1000 * 2 + for off := 0; off < len(input); off += frame { + end := min(off+frame, len(input)) + send(&pb.AudioToAudioRequest{Payload: &pb.AudioToAudioRequest_Frame{ + Frame: &pb.AudioToAudioFrame{Pcm: input[off:end]}}}) + } + control("input_audio_buffer.commit", map[string]any{}) + control("response.create", map[string]any{"max_output_tokens": 200}) + turn1 := collect() + var gotCommit bool + var audio []byte + for _, e := range turn1 { + switch e.event { + case "input_audio_buffer.committed": + gotCommit = true + case "response.audio.delta": + audio = append(audio, e.pcm...) + } + } + Expect(gotCommit).To(BeTrue(), "no input_audio_buffer.committed") + Expect(turn1[len(turn1)-1].meta["status"]).To(Equal("completed")) + Expect(audio).NotTo(BeEmpty(), "no response.audio.delta in turn 1") + Expect(sound.CalculateRMS16(sound.BytesToInt16sLE(audio))).To(BeNumerically(">", 100), + "turn 1 audio is silent") + + // Turn 2: a text item that needs the tool. + control("conversation.item.create", map[string]any{"item": map[string]any{ + "type": "message", "role": "user", + "content": []map[string]any{{"type": "input_text", "text": "Call the get_time tool."}}, + }}) + control("response.create", map[string]any{"max_output_tokens": 200}) + turn2 := collect() + var callID string + var calls int + for _, e := range turn2 { + if e.event == "response.function_call_arguments.done" { + calls++ + Expect(e.meta["name"]).To(Equal("get_time")) + callID, _ = e.meta["call_id"].(string) + } + } + Expect(calls).To(Equal(1), "expected one get_time call") + Expect(callID).NotTo(BeEmpty()) + + // Turn 3: the tool result, then a spoken answer. + control("conversation.item.create", map[string]any{"item": map[string]any{ + "type": "function_call_output", "call_id": callID, "output": `{"time":"12:00"}`, + }}) + control("response.create", map[string]any{"max_output_tokens": 200}) + turn3 := collect() + var out3 []byte + for _, e := range turn3 { + if e.event == "response.audio.delta" { + out3 = append(out3, e.pcm...) + } + } + Expect(turn3[len(turn3)-1].meta["status"]).To(Equal("completed")) + Expect(out3).NotTo(BeEmpty(), "no audio after the tool result") + GinkgoWriter.Printf("s2s: turn1 audio=%d bytes, tool call %s, turn3 audio=%d bytes\n", + len(audio), callID, len(out3)) + Expect(stream.CloseSend()).To(Succeed()) + }) + It("transforms audio via AudioTransform (batch)", func() { if !caps[capAudioTransform] { Skip("audio_transform capability not enabled") @@ -1655,3 +1839,56 @@ func keys(m map[string]bool) []string { } return out } + +// containsAnyFold reports whether text contains any of words, ignoring case +// and surrounding spaces. +func containsAnyFold(text string, words ...string) bool { + lower := strings.ToLower(text) + for _, w := range words { + if w = strings.ToLower(strings.TrimSpace(w)); w != "" && strings.Contains(lower, w) { + return true + } + } + return false +} + +func envFloat(name string) (float64, bool) { + raw := os.Getenv(name) + if raw == "" { + return 0, false + } + var v float64 + if _, err := fmt.Sscanf(raw, "%g", &v); err != nil { + return 0, false + } + return v, true +} + +// assertTTSAudio applies the optional duration and RMS bounds from +// BACKEND_TEST_TTS_{MIN,MAX}_SECONDS and BACKEND_TEST_TTS_MIN_RMS to a WAV. +func assertTTSAudio(path string) { + GinkgoHelper() + minSec, hasMin := envFloat("BACKEND_TEST_TTS_MIN_SECONDS") + maxSec, hasMax := envFloat("BACKEND_TEST_TTS_MAX_SECONDS") + minRMS, hasRMS := envFloat("BACKEND_TEST_TTS_MIN_RMS") + if !hasMin && !hasMax && !hasRMS { + return + } + raw, err := os.ReadFile(path) + Expect(err).NotTo(HaveOccurred()) + pcm, rate := laudio.ParseWAV(raw) + Expect(rate).To(BeNumerically(">", 0), "TTS output is not a WAV") + samples := sound.BytesToInt16sLE(pcm) + seconds := float64(len(samples)) / float64(rate) + rms := sound.CalculateRMS16(samples) + GinkgoWriter.Printf("TTS audio: %.2f s, rms %.0f\n", seconds, rms) + if hasMin { + Expect(seconds).To(BeNumerically(">=", minSec), "TTS audio too short") + } + if hasMax { + Expect(seconds).To(BeNumerically("<=", maxSec), "TTS audio too long (missing EOS?)") + } + if hasRMS { + Expect(rms).To(BeNumerically(">=", minRMS), "TTS audio is silent") + } +} diff --git a/tests/e2e/REALTIME_S2S.md b/tests/e2e/REALTIME_S2S.md new file mode 100644 index 000000000..6c93e3602 --- /dev/null +++ b/tests/e2e/REALTIME_S2S.md @@ -0,0 +1,79 @@ +# Realtime speech-to-speech E2E + +`realtime_s2s_test.go` drives `/v1/realtime` with the four shapes of the LFM3 +gallery entries: an any-to-any model or a chained pipeline, each with a VAD or +without one. A fake client speaks audio that a TTS generated, streams it as +`input_audio_buffer.append` chunks, and checks session setup, the turn +detection the model allows, server VAD and manual-commit turns, audio deltas, +transcripts, barge-in and a tool-call round trip. + +| Shape | Mock model | Gallery entry | +|---|---|---| +| any-to-any, VAD | `realtime-s2s-mock` | `lfm3-audio-2b-vad`, `-q8-vad` | +| any-to-any, no VAD | `realtime-s2s-mock-novad` | `lfm3-audio-2b`, `-q8` | +| pipeline, VAD | `realtime-pipeline` | `lfm3-audio-2b-pipeline-vad`, `-q8-pipeline-vad` | +| pipeline, no VAD | `realtime-pipeline-novad` | `lfm3-audio-2b-pipeline`, `-q8-pipeline` | + +Specs that need a VAD (server VAD turn, barge-in) run only for the VAD shapes. +Barge-in runs for the chained mock pipeline in the real variant only, because +the mock LLM answers at once. A VAD-less shape also checks that it starts with +turn detection off and that `server_vad` is refused. + +## Mock variant (CI) + +Uses the mock backend. No weights. This is what CI runs. + +```sh +make test-realtime +``` + +## Real-weights variant (manual, GPU recommended) + +Runs the same specs against LFM3-Audio on the llama-cpp backend, once for each +shape in the table above, with configs that mirror the gallery entries (the VAD +is the sherpa silero model its extracted backend already has, in place of +`silero-vad-ggml`). It is never run in CI: the label filter of +`make test-realtime` excludes `real-models`. + +The client renders its input speech with a sherpa-onnx VITS voice. A separate +sherpa-onnx ASR transcribes the reply audio, so the checks do not rely on the +model's own transcript. Assertions are robust to sampling: reply duration and +loudness, keywords found in the transcript or the ASR text, and event order. + +An any-to-any model does not transcribe its input by default (that would share +the llama.cpp context with the reply and slow it down). The script sets +`pipeline.transcription` of the any-to-any configs to the sherpa ASR, so the +input transcript path stays covered. The mock configs name `mock-stt` the same way. + +```sh +# Build the backend images once (pick the build type for the host). +BUILD_TYPE=cublas CUDA_MAJOR_VERSION=13 make docker-build-llama-cpp +make docker-build-sherpa-onnx + +# Run. Downloads about 3 GB on the first run; KEEP_WORK keeps it for the next. +KEEP_WORK=1 WORK_DIR=$HOME/lfm3-e2e LFM3_GPU_LAYERS=99 make test-realtime-s2s-real-nobuild +``` + +`make test-realtime-s2s-real` also builds the two images. Set `LFM3_VARIANT=q8` +to test the Q8_0 entries instead of the Q4_K_M ones. Set `LFM3_MATRIX` to run +some shapes only, as `kind:vad` words, for example: + +```sh +LFM3_MATRIX="pipeline:novad pipeline:vad" make test-realtime-s2s-real-nobuild +``` + +The default runs `any-to-any:vad any-to-any:novad pipeline:vad pipeline:novad`. + +To run against a different setup, set the variables yourself and call ginkgo: + +| Variable | Meaning | +|---|---| +| `REALTIME_S2S_MODEL` | Model name in the models path. Selects the real variant. | +| `REALTIME_S2S_KIND` | `any-to-any` (default) or `pipeline`. | +| `REALTIME_S2S_VAD` | `1` (default) when the model has a VAD, `0` when it has none. | +| `REALTIME_S2S_TTS` | TTS model that renders the input utterances. | +| `REALTIME_S2S_ASR` | ASR model that transcribes the reply audio. Optional. | +| `REALTIME_MODELS_PATH`, `REALTIME_BACKENDS_PATH` | Models and extracted backends. | + +The gRPC-level check for the same backend is `make test-extra-backend-llama-cpp-lfm3` +(TTS, TTS stream, ASR and the `s2s` capability). diff --git a/tests/e2e/e2e_suite_test.go b/tests/e2e/e2e_suite_test.go index 6cfe19f28..ba3cc9a91 100644 --- a/tests/e2e/e2e_suite_test.go +++ b/tests/e2e/e2e_suite_test.go @@ -5,6 +5,7 @@ import ( "fmt" "net/http" "os" + "os/exec" "path/filepath" "testing" "time" @@ -275,6 +276,58 @@ var _ = BeforeSuite(func() { Expect(err).ToNot(HaveOccurred()) Expect(os.WriteFile(filepath.Join(modelsPath, "realtime-pipeline.yaml"), pipelineData, 0644)).To(Succeed()) + // Speech-to-speech model for the realtime S2S specs (realtime_s2s_test.go): + // one any-to-any model served by the mock backend, with server VAD from + // mock-vad. The specs stay pending until /v1/realtime routes S2S models. + s2sCfg := map[string]any{ + "name": "realtime-s2s-mock", + "backend": "mock-backend", + "known_usecases": []string{"FLAG_REALTIME_AUDIO", "FLAG_CHAT", "FLAG_TRANSCRIPT", "FLAG_TTS"}, + "parameters": map[string]any{ + "model": "mock-s2s.bin", + }, + "pipeline": map[string]any{ + "any_to_any": true, + "vad": "mock-vad", + "transcription": "mock-stt", // the input transcript comes from a model of its own + }, + } + s2sData, err := yaml.Marshal(s2sCfg) + Expect(err).ToNot(HaveOccurred()) + Expect(os.WriteFile(filepath.Join(modelsPath, "realtime-s2s-mock.yaml"), s2sData, 0644)).To(Succeed()) + + // The same any-to-any model without a VAD model: nothing detects turns, so + // the client commits them (the lfm3-audio-2b gallery shape). + s2sNoVADCfg := map[string]any{ + "name": "realtime-s2s-mock-novad", + "backend": "mock-backend", + "known_usecases": []string{"FLAG_REALTIME_AUDIO", "FLAG_CHAT", "FLAG_TRANSCRIPT", "FLAG_TTS"}, + "parameters": map[string]any{ + "model": "mock-s2s.bin", + }, + "pipeline": map[string]any{ + "any_to_any": true, + "transcription": "mock-stt", + }, + } + s2sNoVADData, err := yaml.Marshal(s2sNoVADCfg) + Expect(err).ToNot(HaveOccurred()) + Expect(os.WriteFile(filepath.Join(modelsPath, "realtime-s2s-mock-novad.yaml"), s2sNoVADData, 0644)).To(Succeed()) + + // Chained pipeline without a VAD slot (the lfm3-audio-2b-pipeline gallery + // shape). realtime-pipeline above is the chained form with a VAD. + pipelineNoVADCfg := map[string]any{ + "name": "realtime-pipeline-novad", + "pipeline": map[string]any{ + "transcription": "mock-stt", + "llm": "mock-llm", + "tts": "mock-tts", + }, + } + pipelineNoVADData, err := yaml.Marshal(pipelineNoVADCfg) + Expect(err).ToNot(HaveOccurred()) + Expect(os.WriteFile(filepath.Join(modelsPath, "realtime-pipeline-novad.yaml"), pipelineNoVADData, 0644)).To(Succeed()) + // Realtime pipelines whose LLM is a failover chain: target 0 always fails // to load, target 1 is mock-llm. rt-failover skips the warm-up so the // switch happens on the first turn, mid-session; rt-failover-warm keeps it @@ -526,6 +579,14 @@ var _ = BeforeSuite(func() { } } + // The real S2S run converts the input audio with ffmpeg; fail early with a + // clear message instead of a decode error in the middle of a response. + if os.Getenv("REALTIME_S2S_MODEL") != "" { + _, err := exec.LookPath("ffmpeg") + Expect(err).ToNot(HaveOccurred(), "the real S2S run needs ffmpeg on PATH (the input audio converter runs it)") + Expect(os.Getenv("REALTIME_BACKENDS_PATH")).ToNot(BeEmpty(), "the real S2S run needs REALTIME_BACKENDS_PATH to point at the extracted llama-cpp and sherpa-onnx backends") + } + // Set up system state. When REALTIME_BACKENDS_PATH is set, use it so the // application can discover real backend binaries for real-model tests. systemOpts := []system.SystemStateOptions{ @@ -726,7 +787,12 @@ var _ = BeforeSuite(func() { // Register mock backend (always available for non-realtime tests). localAIApp.ModelLoader().SetExternalBackend("mock-backend", mockBackendPath) localAIApp.ModelLoader().SetExternalBackend("opus", mockBackendPath) - localAIApp.ModelLoader().SetExternalBackend("llama-cpp", mockBackendPath) + // The real S2S run (REALTIME_S2S_MODEL) must reach the real llama-cpp + // backend found under REALTIME_BACKENDS_PATH, so the mock only stands in + // for it when that run is off. + if os.Getenv("REALTIME_S2S_MODEL") == "" { + localAIApp.ModelLoader().SetExternalBackend("llama-cpp", mockBackendPath) + } if cloudProxyPath != "" { localAIApp.ModelLoader().SetExternalBackend("cloud-proxy", cloudProxyPath) } diff --git a/tests/e2e/mock-backend/main.go b/tests/e2e/mock-backend/main.go index 05e31c9d7..744fa43bb 100644 --- a/tests/e2e/mock-backend/main.go +++ b/tests/e2e/mock-backend/main.go @@ -1016,16 +1016,57 @@ func (m *MockBackend) VAD(ctx context.Context, in *pb.VADRequest) (*pb.VADRespon return &pb.VADResponse{}, nil } - // Audio has signal — return a single segment covering the duration. - duration := float64(len(in.Audio)) / 16000.0 - return &pb.VADResponse{ - Segments: []*pb.VADSegment{ - { - Start: 0.0, - End: float32(duration), - }, - }, - }, nil + // Audio has signal. Report the loud stretches as segments, so audio with + // silence after the speech lets the realtime turn detector see the end of + // the turn. Audio that is loud throughout is one segment over its whole + // duration, as before. + return &pb.VADResponse{Segments: mockVADSegments(in.Audio)}, nil +} + +// mockVADSegments finds the loud stretches of 16 kHz float32 audio: 10 ms +// windows above a small RMS floor, with gaps under 200 ms merged. +func mockVADSegments(audio []float32) []*pb.VADSegment { + const ( + rate = 16000 + window = rate / 100 + floor = 0.01 + mergeGap = 20 // windows + ) + var segs []*pb.VADSegment + start, last := -1, -1 // window indexes of the open segment + flush := func() { + if start < 0 { + return + } + segs = append(segs, &pb.VADSegment{ + Start: float32(start*window) / rate, + End: float32(min((last+1)*window, len(audio))) / rate, + }) + start = -1 + } + for w := 0; w*window < len(audio); w++ { + chunk := audio[w*window : min((w+1)*window, len(audio))] + var sumSq float64 + for _, v := range chunk { + sumSq += float64(v) * float64(v) + } + if math.Sqrt(sumSq/float64(len(chunk))) < floor { + continue + } + if start >= 0 && w-last > mergeGap { + flush() + } + if start < 0 { + start = w + } + last = w + } + flush() + if len(segs) == 0 { + // Faint audio above the overall floor but under the window floor. + segs = append(segs, &pb.VADSegment{Start: 0, End: float32(len(audio)) / rate}) + } + return segs } // Diarize returns a deterministic two-speaker layout that exercises the diff --git a/tests/e2e/mock-backend/s2s.go b/tests/e2e/mock-backend/s2s.go new file mode 100644 index 000000000..3cee2315d --- /dev/null +++ b/tests/e2e/mock-backend/s2s.go @@ -0,0 +1,537 @@ +// SPDX-License-Identifier: MIT +package main + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "math" + "strings" + "sync" + "sync/atomic" + "time" + + pb "github.com/mudler/LocalAI/pkg/grpc/proto" +) + +// The mock AudioToAudioStream is a deterministic stand-in for an any-to-any +// speech model. It follows the contract in backend/backend.proto and the +// llama-cpp realtime session, so the realtime endpoint can be tested without +// weights. Behaviour, per turn: +// +// - config: session.created with the fixed rates and voices. Tools are kept. +// - frame / input_audio_buffer.commit: the input is kept until the commit, +// which records its RMS and replies input_audio_buffer.committed. +// - response.create: when tools are set and no function_call_output has +// arrived yet, the first tool is called with {"city":"Paris"}. Otherwise the +// reply is "mock reply rms=" as transcript pieces, followed by +// s2sReplyFrames 20 ms tone frames (default 50 = 1 s) at 24 kHz, one every +// 20 ms of wall time, so a barge-in can land mid-response. +// With tools, response.done also carries raw_text: the call markup of the +// LFM3 template for a tool call, the transcript otherwise. +// - response.cancel: stops the reply at once and sends response.done with +// status cancelled. +// +// Where it matters, it keeps the limits of the llama-cpp session: +// +// - output_modalities from the config is a fixed session mode. Without one, +// each response picks its mode from the tools it has then: the model +// chooses with tools, text plus audio without. A forced text plus audio +// mode with tools makes the model read the tool call out loud. +// - after the first commit or message, session.update of instructions, voice +// or tools is refused with an error that does not belong to a response, and +// a running reply goes on. +// - committing an empty input buffer is an error that does not end a reply. +// - response.create while a reply runs is refused. +// - an error that ends the request of a response.create carries +// "response_scoped": true in its meta; others do not. +// +// Everything is driven from one receive loop. Each response runs on its own +// goroutine and shares no state with the loop beyond its plan, so only the +// gRPC send is locked. + +const ( + mockS2SInputRate = 16000 + mockS2SOutputRate = 24000 + mockS2SFrameDuration = 20 * time.Millisecond + mockS2SToneHz = 440 + mockS2SToneAmplitude = 8000 + mockS2SDefaultFrames = 50 +) + +// s2sReplyFrames is how many tone frames a reply has. The specs of this package +// change it to get a short or a long reply; the mock reads it once per reply. +var s2sReplyFrames atomic.Int32 + +func init() { s2sReplyFrames.Store(mockS2SDefaultFrames) } + +// mockS2SFrames returns how many tone frames a reply has. +func mockS2SFrames() int { return int(s2sReplyFrames.Load()) } + +type s2sStream struct { + stream pb.Backend_AudioToAudioStreamServer + + sendMu sync.Mutex + seq int64 + + configured bool + tools []string + outputModalities []string + started bool + turnPCM []byte + lastRMS float64 + toolOutputSeen bool + calls int + resp *s2sResponse +} + +// s2sResponse is one reply in flight. cancel is closed by the receive loop; +// done is closed by the response goroutine once its last event is sent. +type s2sResponse struct { + cancel chan struct{} + done chan struct{} +} + +// s2sPlan is everything a response goroutine needs, copied at response.create. +type s2sPlan struct { + toolName string + callID string + rms int + textOnly bool + frames int + // spoken is the text and audio of a model that reads a tool call out loud. + spoken string + // hasTools is true if the session has tools. Then response.done carries + // raw_text, what the model wrote with the call markup, like the llama-cpp + // session does. + hasTools bool +} + +// AudioToAudioStream implements the bidirectional S2S RPC. +func (m *MockBackend) AudioToAudioStream(stream pb.Backend_AudioToAudioStreamServer) error { + s := &s2sStream{stream: stream} + defer s.stopResponse() + + for { + req, err := stream.Recv() + if errors.Is(err, io.EOF) { + return nil + } + if err != nil { + return err + } + if !s.configured && req.GetConfig() == nil { + s.sendError("invalid_request_error", "the first message must be a config") + return nil + } + switch p := req.GetPayload().(type) { + case *pb.AudioToAudioRequest_Config: + if !s.configure(p.Config) { + return nil + } + case *pb.AudioToAudioRequest_Frame: + s.frame(p.Frame) + case *pb.AudioToAudioRequest_Control: + s.control(p.Control) + default: + s.sendError("invalid_request_error", "empty request payload") + } + } +} + +// configure starts a session, or restarts it on reset. A later config without +// reset only replaces tools and params. It returns false after an error event, +// which ends the stream. +func (s *s2sStream) configure(c *pb.AudioToAudioConfig) bool { + tools, err := parseS2STools(c.GetTools()) + if err != nil { + s.sendError("invalid_request_error", "tools: "+err.Error()) + return false + } + s.stopResponse() + first := !s.configured || c.GetReset_() + if first { + s.started = false + s.outputModalities = nil + s.turnPCM = nil + s.lastRMS = 0 + s.toolOutputSeen = false + s.calls = 0 + s.configured = true + s.send("session.created", nil, 0, map[string]any{ + "input_sample_rate": mockS2SInputRate, + "output_sample_rate": mockS2SOutputRate, + "voices": []string{"female", "male"}, + }) + } + s.tools = tools + if mods := parseS2SModalities(c.GetParams()["output_modalities"]); mods != nil { + s.outputModalities = mods + } + return true +} + +func (s *s2sStream) frame(f *pb.AudioToAudioFrame) { + s.turnPCM = append(s.turnPCM, f.GetPcm()...) + if f.GetEndOfInput() { + s.commit() + } +} + +func (s *s2sStream) commit() { + if len(s.turnPCM) == 0 { + s.sendError("input_audio_buffer_commit_empty", "input_audio_buffer.commit: the buffer is empty") + return + } + s.started = true + s.lastRMS = rmsS16LE(s.turnPCM) + s.turnPCM = nil + s.send("input_audio_buffer.committed", nil, 0, nil) +} + +func (s *s2sStream) control(c *pb.AudioToAudioControl) { + switch c.GetEvent() { + case "input_audio_buffer.commit": + s.commit() + case "response.create": + s.responseCreate(c.GetPayload()) + case "response.cancel": + // The real backend refuses a cancel when nothing is running. + if s.resp == nil || s.responseFinished() { + s.sendError("response_cancel_not_active", "response.cancel: there is no response to cancel") + return + } + s.stopResponse() + case "conversation.item.create": + s.itemCreate(c.GetPayload()) + case "session.update": + s.sessionUpdate(c.GetPayload()) + default: + s.sendError("invalid_request_error", "unknown control event "+c.GetEvent()) + } +} + +func (s *s2sStream) itemCreate(payload []byte) { + var item struct { + Type string `json:"type"` + } + if err := json.Unmarshal(payload, &item); err != nil { + s.sendError("invalid_request_error", "conversation.item.create: "+err.Error()) + return + } + switch item.Type { + case "function_call_output": + s.started = true + s.toolOutputSeen = true + case "message": + s.started = true + default: + s.sendError("invalid_request_error", "unsupported item type "+item.Type) + } +} + +func (s *s2sStream) sessionUpdate(payload []byte) { + var upd struct { + Tools json.RawMessage `json:"tools"` + Instructions string `json:"instructions"` + Voice string `json:"voice"` + OutputModalities []string `json:"output_modalities"` + } + if err := json.Unmarshal(payload, &upd); err != nil { + s.sendError("invalid_request_error", "session.update: "+err.Error()) + return + } + if s.started { + for _, f := range []struct { + name string + set bool + }{ + {"instructions", upd.Instructions != ""}, + {"tools", len(upd.Tools) > 0 && string(upd.Tools) != "null"}, + {"voice", upd.Voice != ""}, + } { + if f.set { + s.sendError("invalid_request_error", "session.update: "+f.name+" cannot change after the first audio or message") + return + } + } + } + if len(upd.Tools) > 0 && string(upd.Tools) != "null" { + tools, err := parseS2STools(string(upd.Tools)) + if err != nil { + s.sendError("invalid_request_error", "tools: "+err.Error()) + return + } + s.tools = tools + } + if len(upd.OutputModalities) > 0 { + s.outputModalities = upd.OutputModalities + } + s.send("session.updated", nil, 0, map[string]any{ + "input_sample_rate": mockS2SInputRate, + "output_sample_rate": mockS2SOutputRate, + "voices": []string{"female", "male"}, + }) +} + +func (s *s2sStream) responseCreate(payload []byte) { + if !s.configured { + s.sendScopedError("invalid_request_error", "response.create before config") + return + } + if s.resp != nil && !s.responseFinished() { + s.sendScopedError("conversation_already_has_active_response", "response.create: a response is already running") + return + } + var req struct { + OutputModalities []string `json:"output_modalities"` + MaxOutputTokens int `json:"max_output_tokens"` + } + if len(payload) > 0 { + if err := json.Unmarshal(payload, &req); err != nil { + s.sendScopedError("invalid_request_error", "response.create: "+err.Error()) + return + } + } + s.stopResponse() + + // The mode is picked when the response starts: a response option, else the + // fixed session mode, else the default for the tools set now. + modalities := s.outputModalities + if len(req.OutputModalities) > 0 { + modalities = req.OutputModalities + } + forced := len(modalities) > 0 + if !forced && len(s.tools) == 0 { + modalities = []string{"text", "audio"} + } + textOnly := len(modalities) == 1 && modalities[0] == "text" + + var plan s2sPlan + if len(s.tools) > 0 && !s.toolOutputSeen { + s.calls++ + plan = s2sPlan{toolName: s.tools[0], callID: fmt.Sprintf("call_mock_%d", s.calls), hasTools: true} + if forced && !textOnly { + // Forced text plus audio: the model reads the call out loud. + plan.spoken = "calling " + s.tools[0] + } + } else { + frames := mockS2SFrames() + if req.MaxOutputTokens > 0 && req.MaxOutputTokens < frames { + frames = req.MaxOutputTokens + } + plan = s2sPlan{ + rms: int(math.Round(s.lastRMS)), + textOnly: textOnly, + frames: frames, + hasTools: len(s.tools) > 0, + } + if textOnly { + plan.frames = 0 + } + } + + r := &s2sResponse{cancel: make(chan struct{}), done: make(chan struct{})} + s.resp = r + go s.runResponse(r, plan) +} + +func (s *s2sStream) responseFinished() bool { + select { + case <-s.resp.done: + return true + default: + return false + } +} + +// stopResponse cancels the reply in flight, if any, and waits until its +// response.done is sent. It is a no-op when the reply already finished. +func (s *s2sStream) stopResponse() { + r := s.resp + if r == nil { + return + } + close(r.cancel) + <-r.done + s.resp = nil +} + +func (s *s2sStream) runResponse(r *s2sResponse, p s2sPlan) { + defer close(r.done) + + if p.toolName != "" { + select { + case <-r.cancel: + s.finish("cancelled", 0, "", "", p.hasTools) + return + default: + } + if p.spoken != "" { + s.send("response.audio_transcript.delta", nil, 0, map[string]any{"delta": p.spoken}) + s.send("response.audio.delta", s2sToneFrame(0), mockS2SOutputRate, nil) + } + s.send("response.function_call_arguments.done", nil, 0, map[string]any{ + "call_id": p.callID, + "name": p.toolName, + "arguments": `{"city":"Paris"}`, + }) + call := "\n\n\nParis\n\n\n" + s.finish("completed", 0, "", call, true) + return + } + + textEvent := "response.audio_transcript.delta" + if p.textOnly { + textEvent = "response.text.delta" + } + // The transcript is sent in three pieces, before the audio. + pieces := []string{"mock ", "reply ", fmt.Sprintf("rms=%d", p.rms)} + var text strings.Builder + for _, piece := range pieces { + select { + case <-r.cancel: + s.finish("cancelled", 0, text.String(), text.String(), p.hasTools) + return + default: + } + s.send(textEvent, nil, 0, map[string]any{"delta": piece}) + text.WriteString(piece) + } + + ticker := time.NewTicker(mockS2SFrameDuration) + defer ticker.Stop() + sent := 0 + for i := range p.frames { + select { + case <-r.cancel: + s.finish("cancelled", sent, text.String(), text.String(), p.hasTools) + return + case <-ticker.C: + } + s.send("response.audio.delta", s2sToneFrame(i), mockS2SOutputRate, nil) + sent++ + } + s.finish("completed", sent, text.String(), text.String(), p.hasTools) +} + +func (s *s2sStream) finish(status string, frames int, text, raw string, withRaw bool) { + meta := map[string]any{ + "status": status, + "usage": map[string]any{"output_tokens": frames}, + "text": text, + } + if withRaw { + meta["raw_text"] = raw + } + s.send("response.done", nil, 0, meta) +} + +func (s *s2sStream) sendError(typ, msg string) { + s.send("error", nil, 0, map[string]any{"type": typ, "message": msg}) +} + +// sendScopedError reports that a response.create did not start a response. +func (s *s2sStream) sendScopedError(typ, msg string) { + s.send("error", nil, 0, map[string]any{"type": typ, "message": msg, "response_scoped": true}) +} + +// send writes one event. meta is JSON-encoded; nil means "{}". Errors are +// dropped on purpose: a dead stream is torn down by the receive loop. +func (s *s2sStream) send(event string, pcm []byte, sampleRate int32, meta any) { + metaBytes := []byte("{}") + if meta != nil { + b, err := json.Marshal(meta) + if err != nil { + return + } + metaBytes = b + } + s.sendMu.Lock() + defer s.sendMu.Unlock() + s.seq++ + _ = s.stream.Send(&pb.AudioToAudioResponse{ + Event: event, + Pcm: pcm, + SampleRate: sampleRate, + Meta: metaBytes, + Sequence: s.seq, + }) +} + +// parseS2STools reads an OpenAI-style tool array and returns the function +// names. Both {"type":"function","function":{"name":...}} and a bare +// {"name":...} are accepted. An empty string means no tools. +func parseS2STools(raw string) ([]string, error) { + if strings.TrimSpace(raw) == "" { + return nil, nil + } + var defs []struct { + Name string `json:"name"` + Function struct { + Name string `json:"name"` + } `json:"function"` + } + if err := json.Unmarshal([]byte(raw), &defs); err != nil { + return nil, err + } + names := make([]string, 0, len(defs)) + for _, d := range defs { + name := d.Function.Name + if name == "" { + name = d.Name + } + if name == "" { + return nil, errors.New("tool without a name") + } + names = append(names, name) + } + return names, nil +} + +// parseS2SModalities accepts a JSON array or a comma-separated list. +func parseS2SModalities(raw string) []string { + raw = strings.TrimSpace(raw) + if raw == "" { + return nil + } + var mods []string + if json.Unmarshal([]byte(raw), &mods) == nil { + return mods + } + for _, m := range strings.Split(raw, ",") { + if m = strings.TrimSpace(m); m != "" { + mods = append(mods, m) + } + } + return mods +} + +// rmsS16LE is the RMS of mono s16le PCM. A trailing odd byte is ignored. +func rmsS16LE(pcm []byte) float64 { + n := len(pcm) / 2 + if n == 0 { + return 0 + } + var sumSq float64 + for i := range n { + v := float64(int16(uint16(pcm[2*i]) | uint16(pcm[2*i+1])<<8)) + sumSq += v * v + } + return math.Sqrt(sumSq / float64(n)) +} + +// s2sToneFrame returns the i-th 20 ms frame of a 440 Hz tone at 24 kHz, s16le. +// The sample index is absolute, so consecutive frames join without a click. +func s2sToneFrame(i int) []byte { + samplesPerFrame := mockS2SOutputRate * int(mockS2SFrameDuration/time.Millisecond) / 1000 + pcm := make([]byte, samplesPerFrame*2) + for j := range samplesPerFrame { + t := float64(i*samplesPerFrame+j) / mockS2SOutputRate + v := int16(mockS2SToneAmplitude * math.Sin(2*math.Pi*mockS2SToneHz*t)) + pcm[2*j] = byte(v) + pcm[2*j+1] = byte(uint16(v) >> 8) + } + return pcm +} diff --git a/tests/e2e/mock-backend/s2s_test.go b/tests/e2e/mock-backend/s2s_test.go new file mode 100644 index 000000000..79c4ad9fb --- /dev/null +++ b/tests/e2e/mock-backend/s2s_test.go @@ -0,0 +1,353 @@ +// SPDX-License-Identifier: MIT +package main + +import ( + "context" + "encoding/binary" + "encoding/json" + "errors" + "io" + "math" + "net" + "testing" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" +) + +func TestMockBackend(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Mock backend") +} + +// startMockS2S serves MockBackend on a loopback port and returns a client. +// The mock is driven over real gRPC so the tests cover the wire contract too. +func startMockS2S() pb.BackendClient { + lis, err := net.Listen("tcp", "127.0.0.1:0") + Expect(err).ToNot(HaveOccurred()) + srv := grpc.NewServer() + pb.RegisterBackendServer(srv, &MockBackend{}) + go func() { _ = srv.Serve(lis) }() + DeferCleanup(srv.Stop) + + conn, err := grpc.NewClient("passthrough:///"+lis.Addr().String(), + grpc.WithTransportCredentials(insecure.NewCredentials())) + Expect(err).ToNot(HaveOccurred()) + DeferCleanup(func() { _ = conn.Close() }) + return pb.NewBackendClient(conn) +} + +func openS2S(client pb.BackendClient) pb.Backend_AudioToAudioStreamClient { + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + DeferCleanup(cancel) + stream, err := client.AudioToAudioStream(ctx) + Expect(err).ToNot(HaveOccurred()) + return stream +} + +func sendS2S(stream pb.Backend_AudioToAudioStreamClient, req *pb.AudioToAudioRequest) { + GinkgoHelper() + Expect(stream.Send(req)).To(Succeed(), "send") +} + +func recvS2S(stream pb.Backend_AudioToAudioStreamClient) *pb.AudioToAudioResponse { + GinkgoHelper() + evt, err := stream.Recv() + Expect(err).ToNot(HaveOccurred(), "recv") + return evt +} + +// untilDone reads events up to and including response.done and returns them. +func untilDone(stream pb.Backend_AudioToAudioStreamClient) []*pb.AudioToAudioResponse { + GinkgoHelper() + var out []*pb.AudioToAudioResponse + for { + evt := recvS2S(stream) + out = append(out, evt) + if evt.GetEvent() == "response.done" || evt.GetEvent() == "error" { + return out + } + } +} + +func metaOf(evt *pb.AudioToAudioResponse) map[string]any { + GinkgoHelper() + var m map[string]any + Expect(json.Unmarshal(evt.GetMeta(), &m)).To(Succeed(), "%s meta is not JSON", evt.GetEvent()) + return m +} + +// withReplyFrames sets the length of the replies for the current spec. +func withReplyFrames(n int) { + s2sReplyFrames.Store(int32(n)) + DeferCleanup(func() { s2sReplyFrames.Store(mockS2SDefaultFrames) }) +} + +func configReq(tools string) *pb.AudioToAudioRequest { + return &pb.AudioToAudioRequest{Payload: &pb.AudioToAudioRequest_Config{ + Config: &pb.AudioToAudioConfig{Tools: tools}, + }} +} + +func controlReq(event, payload string) *pb.AudioToAudioRequest { + return &pb.AudioToAudioRequest{Payload: &pb.AudioToAudioRequest_Control{ + Control: &pb.AudioToAudioControl{Event: event, Payload: []byte(payload)}, + }} +} + +func frameReq(pcm []byte, endOfInput bool) *pb.AudioToAudioRequest { + return &pb.AudioToAudioRequest{Payload: &pb.AudioToAudioRequest_Frame{ + Frame: &pb.AudioToAudioFrame{Pcm: pcm, EndOfInput: endOfInput}, + }} +} + +// tonePCM is n samples of a sine at amplitude a, mono s16le. Its RMS is a/sqrt(2). +func tonePCM(n int, a float64) []byte { + pcm := make([]byte, n*2) + for i := range n { + v := int16(a * math.Sin(2*math.Pi*300*float64(i)/16000)) + binary.LittleEndian.PutUint16(pcm[2*i:], uint16(v)) + } + return pcm +} + +const weatherTool = `[{"type":"function","function":{"name":"get_time","parameters":{"type":"object"}}}]` + +func eventNames(events []*pb.AudioToAudioResponse) []string { + names := make([]string, 0, len(events)) + for _, e := range events { + names = append(names, e.GetEvent()) + } + return names +} + +func configWithModalities(tools, modalities string) *pb.AudioToAudioRequest { + return &pb.AudioToAudioRequest{Payload: &pb.AudioToAudioRequest_Config{ + Config: &pb.AudioToAudioConfig{Tools: tools, Params: map[string]string{"output_modalities": modalities}}, + }} +} + +var _ = Describe("mock speech-to-speech stream", func() { + It("opens with session.created, the fixed rates and the voices", func() { + stream := openS2S(startMockS2S()) + sendS2S(stream, configReq("")) + + evt := recvS2S(stream) + Expect(evt.GetEvent()).To(Equal("session.created")) + m := metaOf(evt) + Expect(m["input_sample_rate"]).To(Equal(float64(16000))) + Expect(m["output_sample_rate"]).To(Equal(float64(24000))) + Expect(m["voices"]).To(Equal([]any{"female", "male"})) + }) + + It("answers an audio turn with the transcript and tone frames", func() { + withReplyFrames(4) + stream := openS2S(startMockS2S()) + sendS2S(stream, configReq("")) + recvS2S(stream) // session.created + + // One second of a 300 Hz tone at amplitude 10000: RMS is about 7071. + sendS2S(stream, frameReq(tonePCM(8000, 10000), false)) + sendS2S(stream, frameReq(nil, true)) + Expect(recvS2S(stream).GetEvent()).To(Equal("input_audio_buffer.committed")) + + sendS2S(stream, controlReq("response.create", `{"max_output_tokens":200}`)) + events := untilDone(stream) + + var text string + var audio int + for _, evt := range events { + switch evt.GetEvent() { + case "response.audio_transcript.delta": + text += metaOf(evt)["delta"].(string) + case "response.audio.delta": + audio++ + Expect(evt.GetSampleRate()).To(Equal(int32(24000))) + Expect(evt.GetPcm()).To(HaveLen(960)) + Expect(rmsS16LE(evt.GetPcm())).To(BeNumerically(">=", 1000), "audio frame is silent") + } + } + Expect(text).To(Equal("mock reply rms=7071")) + Expect(audio).To(Equal(4)) + m := metaOf(events[len(events)-1]) + Expect(m["status"]).To(Equal("completed")) + Expect(m["text"]).To(Equal(text)) + }) + + It("sends no audio for a text-only response", func() { + withReplyFrames(4) + stream := openS2S(startMockS2S()) + sendS2S(stream, configReq("")) + recvS2S(stream) + + sendS2S(stream, controlReq("response.create", `{"output_modalities":["text"]}`)) + for _, evt := range untilDone(stream) { + Expect(evt.GetEvent()).ToNot(Equal("response.audio.delta"), "text-only response sent audio") + } + }) + + It("calls a tool once, then continues after its output", func() { + withReplyFrames(2) + stream := openS2S(startMockS2S()) + sendS2S(stream, configReq(weatherTool)) + recvS2S(stream) + + sendS2S(stream, controlReq("conversation.item.create", + `{"type":"message","role":"user","content":[{"type":"input_text","text":"Call the get_time tool."}]}`)) + sendS2S(stream, controlReq("response.create", `{}`)) + events := untilDone(stream) + Expect(eventNames(events)).To(Equal([]string{"response.function_call_arguments.done", "response.done"})) + call := metaOf(events[0]) + Expect(call["name"]).To(Equal("get_time")) + Expect(call["call_id"]).To(Equal("call_mock_1")) + Expect(call["arguments"]).To(Equal(`{"city":"Paris"}`)) + Expect(metaOf(events[1])["status"]).To(Equal("completed")) + raw, _ := metaOf(events[1])["raw_text"].(string) + Expect(raw).To(ContainSubstring(""), "raw_text keeps the call markup") + + sendS2S(stream, controlReq("conversation.item.create", + `{"type":"function_call_output","call_id":"call_mock_1","output":"{\"time\":\"12:00\"}"}`)) + sendS2S(stream, controlReq("response.create", `{}`)) + audio := 0 + for _, evt := range untilDone(stream) { + switch evt.GetEvent() { + case "response.function_call_arguments.done": + Fail("second tool call after the output was already sent") + case "response.audio.delta": + audio++ + case "response.done": + meta := metaOf(evt) + Expect(meta["raw_text"]).To(Equal(meta["text"]), "a plain reply with tools") + } + } + Expect(audio).To(Equal(2)) + }) + + It("stops the reply at a cancel and closes cleanly", func() { + withReplyFrames(500) + stream := openS2S(startMockS2S()) + sendS2S(stream, configReq("")) + recvS2S(stream) + + sendS2S(stream, controlReq("response.create", `{}`)) + for recvS2S(stream).GetEvent() != "response.audio.delta" { + } + sendS2S(stream, controlReq("response.cancel", `{}`)) + + var last *pb.AudioToAudioResponse + for last == nil { + evt := recvS2S(stream) + if evt.GetEvent() == "response.done" { + last = evt + break + } + Expect(evt.GetEvent()).To(BeElementOf("response.audio.delta", "response.audio_transcript.delta"), "unexpected event while cancelling") + } + Expect(metaOf(last)["status"]).To(Equal("cancelled")) + + // Nothing follows the cancelled response.done except the answer to a + // cancel with no reply running, which the llama.cpp backend refuses with + // response_cancel_not_active. Then the stream closes cleanly. + sendS2S(stream, controlReq("response.cancel", `{}`)) + evt := recvS2S(stream) + Expect(evt.GetEvent()).To(Equal("error")) + Expect(metaOf(evt)["type"]).To(Equal("response_cancel_not_active")) + Expect(stream.CloseSend()).To(Succeed()) + _, err := stream.Recv() + Expect(errors.Is(err, io.EOF)).To(BeTrue(), "after cancel the stream sent more data: err=%v", err) + }) + + Describe("protocol errors", func() { + It("needs a config as the first message", func() { + stream := openS2S(startMockS2S()) + sendS2S(stream, controlReq("response.create", `{}`)) + Expect(recvS2S(stream).GetEvent()).To(Equal("error")) + }) + + It("ends the stream with an error on invalid tools", func() { + stream := openS2S(startMockS2S()) + sendS2S(stream, configReq(`{not json`)) + Expect(recvS2S(stream).GetEvent()).To(Equal("error")) + }) + + It("answers an unknown control event with an error", func() { + stream := openS2S(startMockS2S()) + sendS2S(stream, configReq("")) + recvS2S(stream) + sendS2S(stream, controlReq("response.bogus", `{}`)) + Expect(recvS2S(stream).GetEvent()).To(Equal("error")) + }) + }) + + It("keeps a tool call silent unless the modes are fixed to text plus audio", func() { + spoken := func(stream pb.Backend_AudioToAudioStreamClient) int { + sendS2S(stream, controlReq("response.create", "")) + n := 0 + for _, evt := range untilDone(stream) { + if evt.GetEvent() == "response.audio.delta" { + n++ + } + } + return n + } + + // No modalities in the config: the model chooses, so a tool call is silent. + auto := openS2S(startMockS2S()) + sendS2S(auto, configReq(weatherTool)) + recvS2S(auto) + Expect(spoken(auto)).To(BeZero(), "a tool call in the default mode sent audio frames") + + // Text plus audio fixed at open: the model reads the call out loud. + forced := openS2S(startMockS2S()) + sendS2S(forced, configWithModalities(weatherTool, `["text","audio"]`)) + recvS2S(forced) + Expect(spoken(forced)).ToNot(BeZero(), "a tool call in the fixed text plus audio mode was silent") + }) + + It("refuses session changes after the first turn", func() { + stream := openS2S(startMockS2S()) + sendS2S(stream, configReq("")) + recvS2S(stream) + + sendS2S(stream, controlReq("session.update", `{"voice":"male"}`)) + Expect(recvS2S(stream).GetEvent()).To(Equal("session.updated"), "a voice change before the first turn") + sendS2S(stream, controlReq("conversation.item.create", `{"type":"message"}`)) + sendS2S(stream, controlReq("session.update", `{"voice":"female"}`)) + evt := recvS2S(stream) + Expect(evt.GetEvent()).To(Equal("error"), "a late voice change") + Expect(metaOf(evt)["response_scoped"]).To(BeNil(), "an error outside a response") + }) + + It("keeps the reply running through errors that are not about it", func() { + withReplyFrames(10) + stream := openS2S(startMockS2S()) + sendS2S(stream, configReq("")) + recvS2S(stream) + + sendS2S(stream, controlReq("response.create", "")) + sendS2S(stream, controlReq("input_audio_buffer.commit", "")) + sendS2S(stream, controlReq("response.create", "")) + + var errTypes []string + for done := false; !done; { + evt := recvS2S(stream) + switch evt.GetEvent() { + case "error": + m := metaOf(evt) + typ, _ := m["type"].(string) + errTypes = append(errTypes, typ) + scoped := m["response_scoped"] == true + Expect(scoped).To(Equal(typ == "conversation_already_has_active_response"), "response_scoped of %s", typ) + case "response.done": + done = true + Expect(metaOf(evt)["status"]).To(Equal("completed"), "the reply was cut short") + } + } + Expect(errTypes).To(Equal([]string{"input_audio_buffer_commit_empty", "conversation_already_has_active_response"})) + }) +}) diff --git a/tests/e2e/realtime_s2s_test.go b/tests/e2e/realtime_s2s_test.go new file mode 100644 index 000000000..b2d1d8852 --- /dev/null +++ b/tests/e2e/realtime_s2s_test.go @@ -0,0 +1,661 @@ +package e2e_test + +import ( + "bytes" + "encoding/base64" + "encoding/json" + "fmt" + "io" + "mime/multipart" + "net/http" + "os" + "strings" + "time" + + "github.com/gorilla/websocket" + "github.com/mudler/LocalAI/pkg/sound" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// Realtime speech-to-speech (S2S) specs. The same specs run for the four +// shapes the LFM3-Audio gallery entries come in: +// +// - any-to-any: one realtime_audio model with pipeline.any_to_any takes +// audio in and returns audio, transcripts and tool calls from one stream. +// - pipeline: a chained VAD, transcription, LLM and TTS pipeline whose model +// slots can all name one model (the lfm3-audio-2b-pipeline entries). +// +// each with a VAD model (server-side turn detection) and without one (the +// client commits each turn). +// +// Variants, chosen by environment: +// - mock (label Realtime): all four shapes at once, from the fixtures in +// e2e_suite_test.go (realtime-s2s-mock, realtime-s2s-mock-novad, +// realtime-pipeline, realtime-pipeline-novad), served by the mock backend. +// - real (labels Realtime and real-models, only when REALTIME_S2S_MODEL is +// set): one shape of a real model per run, manual on GPU with +// `make test-realtime-s2s-real` (it runs the four shapes one after the +// other). Assertions marked "real" run only here. +// Environment of the real variant: +// REALTIME_S2S_MODEL model name in the models path (e.g. lfm3-audio-2b) +// REALTIME_S2S_KIND any-to-any (default) or pipeline +// REALTIME_S2S_VAD 1 (default) when the model has a VAD, 0 when it has none +// REALTIME_S2S_TTS TTS model that renders the input utterances; pick one +// that is not the model under test (e.g. vits-ljs-sherpa) +// REALTIME_S2S_ASR ASR model that transcribes the OUTPUT audio, so the +// spoken answer is checked independently of the model's +// own transcript (e.g. omnilingual-0.3b-ctc-q8-sherpa) + +const s2sInputRate = 24000 + +// s2sResponseTimeout bounds one response. A real model on CPU is slow. +func s2sResponseTimeout() time.Duration { + if isRealS2S() { + return 10 * time.Minute + } + return 2 * time.Minute +} + +func isRealS2S() bool { return os.Getenv("REALTIME_S2S_MODEL") != "" } + +// s2sVariant is one shape of realtime model under test. +type s2sVariant struct { + label string // part of the Describe title + model string + pipeline bool // chained pipeline instead of one any-to-any model + vad bool // the model names a VAD, so turns are detected server side + real bool +} + +func (v s2sVariant) session(update map[string]any) *websocket.Conn { + conn := connectWS(v.model) + drainUntil(conn, "session.created", 30*time.Second) + if update != nil { + evt := update + if _, whole := update["type"]; !whole { + evt = map[string]any{"type": "session.update", "session": update} + } + sendClientEvent(conn, evt) + drainUntil(conn, "session.updated", 10*time.Second) + } + return conn +} + +// manualCommitSession opens a session in which the client commits turns: a +// VAD-less model starts that way, one with a VAD needs turn detection turned off. +func (v s2sVariant) manualCommitSession(update map[string]any) *websocket.Conn { + if v.vad { + conn := v.session(disableVADEvent()) + if update != nil { + sendClientEvent(conn, map[string]any{"type": "session.update", "session": update}) + drainUntil(conn, "session.updated", 10*time.Second) + } + return conn + } + return v.session(update) +} + +// speak sends one spoken turn and, without server VAD, commits it and asks for +// the response. +func (v s2sVariant) speak(conn *websocket.Conn, text string) { + pcm := s2sUtterance(text) + if v.vad { + streamUtterance(conn, padPCMBase64(pcm, s2sInputRate, 300, 1200)) + return + } + streamUtterance(conn, pcm) + sendClientEvent(conn, map[string]any{"type": "input_audio_buffer.commit"}) + drainUntil(conn, "input_audio_buffer.committed", 30*time.Second) + sendClientEvent(conn, map[string]any{"type": "response.create"}) +} + +func realS2SVariant() s2sVariant { + kind := os.Getenv("REALTIME_S2S_KIND") + v := s2sVariant{model: os.Getenv("REALTIME_S2S_MODEL"), pipeline: kind == "pipeline", vad: os.Getenv("REALTIME_S2S_VAD") != "0", real: true} + v.label = "real " + map[bool]string{false: "any-to-any", true: "pipeline"}[v.pipeline] + map[bool]string{false: " no-vad", true: " vad"}[v.vad] + return v +} + +// s2sTTSModel names the TTS that renders the test utterances. The mock variant +// uses the mock TTS. The real variant uses a fixed single-speaker voice so the +// input is clear and does not share failure modes with the model under test. +func s2sTTSModel() string { + if m := os.Getenv("REALTIME_S2S_TTS"); m != "" { + return m + } + return "mock-tts" +} + +// s2sUtterance renders text and returns 24 kHz s16le PCM as base64. +func s2sUtterance(text string) string { + return ttsPCMBase64(s2sTTSModel(), text, s2sInputRate) +} + +// streamUtterance sends base64 PCM in 40 ms chunks at 2x real time, as +// input_audio_buffer.append events. +func streamUtterance(conn *websocket.Conn, pcmB64 string) { + raw, err := base64.StdEncoding.DecodeString(pcmB64) + ExpectWithOffset(1, err).ToNot(HaveOccurred()) + chunk := s2sInputRate * 40 / 1000 * 2 + for off := 0; off < len(raw); off += chunk { + end := min(off+chunk, len(raw)) + sendClientEvent(conn, map[string]any{ + "type": "input_audio_buffer.append", + "audio": base64.StdEncoding.EncodeToString(raw[off:end]), + }) + time.Sleep(20 * time.Millisecond) + } +} + +// readResponse collects events until response.done, which it also returns. +func readResponse(conn *websocket.Conn, timeout time.Duration) []map[string]any { + var events []map[string]any + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + evt := readServerEvent(conn, time.Until(deadline)) + events = append(events, evt) + if evt["type"] == "response.done" { + return events + } + } + Fail("timed out waiting for response.done") + return nil +} + +func eventTypes(events []map[string]any) []string { + types := make([]string, 0, len(events)) + for _, e := range events { + types = append(types, fmt.Sprint(e["type"])) + } + return types +} + +func hasEvent(events []map[string]any, typ string) bool { + for _, e := range events { + if e["type"] == typ { + return true + } + } + return false +} + +// responseStatus returns status from a response.done event. +func responseStatus(done map[string]any) string { + resp, _ := done["response"].(map[string]any) + s, _ := resp["status"].(string) + return s +} + +// keywordsIn reports whether any keyword appears in text, case-insensitive. +func keywordsIn(text string, keywords ...string) bool { + lower := strings.ToLower(text) + for _, k := range keywords { + if strings.Contains(lower, k) { + return true + } + } + return false +} + +const capitalQuestion = "What is the capital of France? Answer in one short sentence." + +var weatherTool = map[string]any{ + "type": "function", + "name": "get_weather", + "description": "Current weather for a city.", + "parameters": map[string]any{ + "type": "object", + "properties": map[string]any{"city": map[string]any{"type": "string"}}, + "required": []string{"city"}, + }, +} + +// turnDetectionOf reads audio.input.turn_detection out of a session object. +func turnDetectionOf(session map[string]any) any { + audio, _ := session["audio"].(map[string]any) + input, _ := audio["input"].(map[string]any) + return input["turn_detection"] +} + +// defineS2SSpecs registers the specs for one shape. The bodies are the same in +// every variant; "real" guards the assertions that need real weights. +func defineS2SSpecs(v s2sVariant) { + isReal := v.real + decorators := []any{Label("Realtime")} + if v.real { + decorators = append(decorators, Label("real-models")) + } + Describe("Realtime S2S ("+v.label+")", append(decorators, func() { + + It("reports the audio format and echoes session.update", func() { + conn := connectWS(v.model) + defer func() { _ = conn.Close() }() + + created := readServerEvent(conn, 30*time.Second) + Expect(created["type"]).To(Equal("session.created")) + session, _ := created["session"].(map[string]any) + Expect(session).To(HaveKey("audio")) + + sendClientEvent(conn, map[string]any{ + "type": "session.update", + "session": map[string]any{ + "instructions": "Answer briefly.", + "tools": []map[string]any{weatherTool}, + }, + }) + updated := drainUntil(conn, "session.updated", 10*time.Second) + updSession, _ := updated["session"].(map[string]any) + Expect(updSession["instructions"]).To(Equal("Answer briefly.")) + Expect(updSession["tools"]).ToNot(BeEmpty()) + }) + + It("starts with the turn detection its VAD slot allows", func() { + conn := connectWS(v.model) + defer func() { _ = conn.Close() }() + created := readServerEvent(conn, 30*time.Second) + Expect(created["type"]).To(Equal("session.created")) + session, _ := created["session"].(map[string]any) + if v.vad { + Expect(turnDetectionOf(session)).ToNot(BeNil(), "a model with a VAD starts with server_vad") + return + } + Expect(turnDetectionOf(session)).To(BeNil(), "a model without a VAD starts with turn detection off") + + // Turning server_vad on has nothing to detect with, so it is refused + // and the session goes on. + sendClientEvent(conn, map[string]any{ + "type": "session.update", + "session": map[string]any{"audio": map[string]any{"input": map[string]any{ + "turn_detection": map[string]any{"type": "server_vad"}, + }}}, + }) + errEvt := drainUntil(conn, "error", 10*time.Second) + Expect(fmt.Sprint(errEvt)).To(ContainSubstring("VAD")) + }) + + if v.vad { + It("runs a server VAD turn in order", func() { + conn := v.session(nil) + defer func() { _ = conn.Close() }() + + v.speak(conn, capitalQuestion) + events := readResponse(conn, s2sResponseTimeout()) + types := eventTypes(events) + Expect(types).To(ContainElements("input_audio_buffer.speech_started", "input_audio_buffer.speech_stopped", + "input_audio_buffer.committed", "conversation.item.added", "response.created", "response.output_audio.delta")) + Expect(responseStatus(events[len(events)-1])).To(Equal("completed")) + if isReal { + checkSpokenAnswer(events, "paris") + } + }) + } + + It("runs a manual-commit turn without VAD events", func() { + conn := v.manualCommitSession(nil) + defer func() { _ = conn.Close() }() + + streamUtterance(conn, s2sUtterance(capitalQuestion)) + sendClientEvent(conn, map[string]any{"type": "input_audio_buffer.commit"}) + drainUntil(conn, "input_audio_buffer.committed", 30*time.Second) + sendClientEvent(conn, map[string]any{"type": "response.create"}) + + events := readResponse(conn, s2sResponseTimeout()) + Expect(hasEvent(events, "input_audio_buffer.speech_started")).To(BeFalse()) + Expect(hasEvent(events, "response.output_audio.delta")).To(BeTrue()) + Expect(responseStatus(events[len(events)-1])).To(Equal("completed")) + if isReal { + checkSpokenAnswer(events, "paris") + } + }) + + It("emits input and output transcripts", func() { + conn := v.session(nil) + defer func() { _ = conn.Close() }() + + v.speak(conn, capitalQuestion) + events := readResponse(conn, s2sResponseTimeout()) + + var inputText, outputText string + inputDone := false + for _, e := range events { + switch e["type"] { + case "conversation.item.input_audio_transcription.completed": + inputText, _ = e["transcript"].(string) + inputDone = true + case "response.output_audio_transcript.done": + outputText, _ = e["transcript"].(string) + } + } + // The transcript of the input comes from a model of its own and runs next + // to the reply, so it may arrive after response.done: an any-to-any reply + // can be over before the ASR is. The contract is that it comes, not when. + for deadline := time.Now().Add(30 * time.Second); !inputDone && time.Now().Before(deadline); { + evt := readServerEvent(conn, time.Until(deadline)) + if evt["type"] == "conversation.item.input_audio_transcription.completed" { + inputText, _ = evt["transcript"].(string) + inputDone = true + } + } + Expect(strings.TrimSpace(outputText)).ToNot(BeEmpty()) + if isReal { + Expect(keywordsIn(inputText, "france")).To(BeTrue(), "input transcript: %q", inputText) + } + }) + + if v.vad && (!v.pipeline || isReal) { + // The mock chained pipeline answers at once, so there is nothing to + // talk over; the real one and any-to-any have a reply to interrupt. + It("cancels the first response on barge-in and completes the second", func() { + conn := v.session(nil) + defer func() { _ = conn.Close() }() + + first := padPCMBase64(s2sUtterance("Count slowly from one to forty."), s2sInputRate, 300, 1200) + // Rendered up front: a text-to-speech call between the first reply + // and the interruption would eat the window to cancel it in. + interruption := padPCMBase64(s2sUtterance("Stop. What is two plus two?"), s2sInputRate, 300, 1200) + firstSent := make(chan struct{}) + go func() { + defer GinkgoRecover() + streamUtterance(conn, first) + close(firstSent) + }() + + // Wait for the reply to start, then talk over it. An any-to-any model + // streams its audio while it generates, so the first audio delta is + // early in the reply. A chained pipeline with unary stages (the + // gallery default) synthesizes the whole reply and sends it as one + // audio delta right before response.done: nothing is left to cancel + // by then, and on a GPU the reply is ready within a second. Its + // reply is in flight between response.created and that delta. + var firstID string + for firstID == "" { + evt := readServerEvent(conn, s2sResponseTimeout()) + switch { + case v.pipeline && evt["type"] == "response.created": + resp, _ := evt["response"].(map[string]any) + firstID, _ = resp["id"].(string) + case !v.pipeline && evt["type"] == "response.output_audio.delta": + firstID, _ = evt["response_id"].(string) + } + } + Eventually(firstSent, 60*time.Second).Should(BeClosed()) + + streamUtterance(conn, interruption) + + // The first response ends as cancelled, and no audio for it follows. + var firstDone map[string]any + // The cancel must land within 2 s of the barge-in; the loop bound is + // generous so a slow CI runner reports the missing event, not a timeout. + cancelDeadline := time.Now().Add(60 * time.Second) + for firstDone == nil && time.Now().Before(cancelDeadline) { + evt := readServerEvent(conn, 30*time.Second) + if evt["type"] == "response.done" { + resp, _ := evt["response"].(map[string]any) + if resp["id"] == firstID { + firstDone = evt + } + } + } + Expect(firstDone).ToNot(BeNil(), "no response.done for the first response") + Expect(responseStatus(firstDone)).To(Equal("cancelled")) + + second := readResponse(conn, s2sResponseTimeout()) + Expect(responseStatus(second[len(second)-1])).To(Equal("completed")) + if isReal { + checkSpokenAnswer(second, "four", "4") + } + }) + } + + It("round-trips a tool call and its output", func() { + update := map[string]any{"tools": []map[string]any{weatherTool}} + chainedMock := v.pipeline && !isReal + conn := v.session(update) + defer func() { _ = conn.Close() }() + + v.speak(conn, "What is the weather in Paris right now? Use the get_weather tool.") + events := readResponse(conn, s2sResponseTimeout()) + + var callID string + for _, e := range events { + if e["type"] == "response.function_call_arguments.done" { + Expect(e["name"]).To(Equal("get_weather")) + raw, _ := e["arguments"].(string) + if !chainedMock { + var args struct { + City string `json:"city"` + } + Expect(json.Unmarshal([]byte(raw), &args)).To(Succeed(), "arguments: %q", raw) + Expect(strings.ToLower(args.City)).To(ContainSubstring("paris")) + } + callID, _ = e["call_id"].(string) + } + } + Expect(callID).ToNot(BeEmpty(), "no function call in %v", eventTypes(events)) + + if chainedMock { + // The mock LLM calls the first tool whenever tools are set, so drop + // them to get its plain answer. + sendClientEvent(conn, map[string]any{"type": "session.update", "session": map[string]any{"tools": []map[string]any{}}}) + drainUntil(conn, "session.updated", 10*time.Second) + } + sendClientEvent(conn, map[string]any{ + "type": "conversation.item.create", + "item": map[string]any{ + "type": "function_call_output", + "call_id": callID, + "output": `{"temperature_c":21,"condition":"sunny"}`, + }, + }) + sendClientEvent(conn, map[string]any{"type": "response.create"}) + + second := readResponse(conn, s2sResponseTimeout()) + Expect(hasEvent(second, "response.output_audio.delta")).To(BeTrue(), "events: %v", eventTypes(second)) + Expect(responseStatus(second[len(second)-1])).To(Equal("completed")) + if isReal { + checkSpokenAnswer(second, "21", "twenty", "sunny") + } + }) + })) +} + +// outputPCM joins the audio deltas of one response (s16le at s2sInputRate, +// the session output rate). +func outputPCM(events []map[string]any) []byte { + var pcm []byte + for _, e := range events { + if e["type"] != "response.output_audio.delta" { + continue + } + d, _ := e["delta"].(string) + raw, err := base64.StdEncoding.DecodeString(d) + ExpectWithOffset(1, err).ToNot(HaveOccurred()) + pcm = append(pcm, raw...) + } + return pcm +} + +// assertPlausibleAudio checks sampling-robust properties of a spoken reply: a +// duration between 0.4 s and 30 s, at least 30% of 100 ms windows above an RMS +// floor of 300, and no clipped run longer than 50 ms. +func assertPlausibleAudio(pcm []byte) { + GinkgoHelper() + samples := sound.BytesToInt16sLE(pcm) + seconds := float64(len(samples)) / s2sInputRate + Expect(seconds).To(BeNumerically(">=", 0.4), "reply audio too short") + Expect(seconds).To(BeNumerically("<=", 30), "reply audio too long (missing EOS?)") + + win := s2sInputRate / 10 + var loud, total int + for off := 0; off+win <= len(samples); off += win { + total++ + if sound.CalculateRMS16(samples[off:off+win]) > 300 { + loud++ + } + } + Expect(total).To(BeNumerically(">", 0)) + Expect(float64(loud)/float64(total)).To(BeNumerically(">=", 0.3), + "only %d of %d windows are above the RMS floor", loud, total) + + maxRun, run := 0, 0 + for _, v := range samples { + if v >= 32767 || v <= -32768 { + run++ + maxRun = max(maxRun, run) + } else { + run = 0 + } + } + Expect(maxRun).To(BeNumerically("<=", s2sInputRate*50/1000), "clipped run of %d samples", maxRun) +} + +// asrOfOutput transcribes reply audio with the separate ASR model +// (REALTIME_S2S_ASR). It returns "" when no ASR model is configured. +func asrOfOutput(pcm []byte) string { + model := os.Getenv("REALTIME_S2S_ASR") + if model == "" { + return "" + } + var body bytes.Buffer + mw := multipart.NewWriter(&body) + ExpectWithOffset(1, mw.WriteField("model", model)).To(Succeed()) + fw, err := mw.CreateFormFile("file", "reply.wav") + ExpectWithOffset(1, err).ToNot(HaveOccurred()) + _, err = fw.Write(wavFromPCM(pcm, s2sInputRate)) + ExpectWithOffset(1, err).ToNot(HaveOccurred()) + ExpectWithOffset(1, mw.Close()).To(Succeed()) + + resp, err := http.Post(apiURL+"/audio/transcriptions", mw.FormDataContentType(), &body) + ExpectWithOffset(1, err).ToNot(HaveOccurred()) + defer func() { _ = resp.Body.Close() }() + raw, err := io.ReadAll(resp.Body) + ExpectWithOffset(1, err).ToNot(HaveOccurred()) + ExpectWithOffset(1, resp.StatusCode).To(Equal(http.StatusOK), "asr returned: %s", raw) + var out struct { + Text string `json:"text"` + } + ExpectWithOffset(1, json.Unmarshal(raw, &out)).To(Succeed(), "asr body: %s", raw) + return out.Text +} + +// numberWords are the digits and number words a recogniser writes for a spoken +// number, with the canonical word they stand for. +var numberWords = map[string]string{ + "0": "zero", "1": "one", "2": "two", "3": "three", "4": "four", "5": "five", + "6": "six", "7": "seven", "8": "eight", "9": "nine", "10": "ten", + "zero": "zero", "one": "one", "two": "two", "three": "three", "four": "four", "five": "five", + "six": "six", "seven": "seven", "eight": "eight", "nine": "nine", "ten": "ten", +} + +// numberHomophones are the words an ASR may write for a spoken number. They are +// ordinary words in most sentences, so they only count as a number when the +// other text has the number itself. +var numberHomophones = map[string]string{ + "won": "one", "to": "two", "too": "two", "for": "four", "ate": "eight", +} + +// spokenContent returns the content of a text: strong words (4+ letters, or a +// number in canonical form) and weak words (the number a homophone may stand for). +func spokenContent(text string) (strong, weak map[string]bool) { + strong, weak = map[string]bool{}, map[string]bool{} + for _, w := range strings.FieldsFunc(strings.ToLower(text), func(r rune) bool { + return (r < 'a' || r > 'z') && (r < '0' || r > '9') + }) { + switch n, isNumber := numberWords[w]; { + case isNumber: + strong[n] = true + case numberHomophones[w] != "": + weak[numberHomophones[w]] = true + case len(w) >= 4: + strong[w] = true + } + } + return strong, weak +} + +// sharesContentWord reports whether a and b have a word of content in common. +// The model writes "Four" and the ASR hears "for" or "4", so numbers are +// compared in their canonical spoken form and a homophone matches the number +// it sounds like. Two homophones alone ("for" and "for") share nothing, so the +// check does not pass on a stop word. +func sharesContentWord(a, b string) bool { + sa, wa := spokenContent(a) + sb, wb := spokenContent(b) + for w := range sa { + if sb[w] || wb[w] { + return true + } + } + for w := range wa { + if sb[w] { + return true + } + } + return false +} + +// checkSpokenAnswer is the real-variant check of one response: the audio is +// plausible speech, the keywords appear in the union of the model's transcript +// and the separate ASR of the audio, and (when both exist) the two share a +// content word, which shows the audio says what the transcript says. +func checkSpokenAnswer(events []map[string]any, keywords ...string) { + GinkgoHelper() + pcm := outputPCM(events) + assertPlausibleAudio(pcm) + transcript := transcriptOf(events) + asr := asrOfOutput(pcm) + GinkgoWriter.Printf("reply transcript=%q asr=%q\n", transcript, asr) + if len(keywords) > 0 { + Expect(keywordsIn(transcript+" "+asr, keywords...)).To(BeTrue(), + "none of %v in transcript %q or asr %q", keywords, transcript, asr) + } + if strings.TrimSpace(transcript) != "" && strings.TrimSpace(asr) != "" { + Expect(sharesContentWord(transcript, asr)).To(BeTrue(), + "audio and transcript disagree: transcript %q, asr %q", transcript, asr) + } +} + +// transcriptOf joins the output transcript deltas of one response. +func transcriptOf(events []map[string]any) string { + var b strings.Builder + for _, e := range events { + if e["type"] == "response.output_audio_transcript.delta" { + d, _ := e["delta"].(string) + b.WriteString(d) + } + } + return b.String() +} + +var _ = Describe("sharesContentWord", Label("Realtime"), func() { + DescribeTable("compares the transcript with the ASR of the audio", + func(transcript, asr string, want bool) { + Expect(sharesContentWord(transcript, asr)).To(Equal(want)) + }, + Entry("same content word", "The capital of France is Paris.", "the capital is paris", true), + Entry("number word against its homophone", "Four", "for", true), + Entry("number word against a digit", "Two plus two is four.", "2 plus 2 is 4", true), + Entry("homophone of two", "Two", "too", true), + Entry("different number", "Four", "five", false), + Entry("homophone against a different number", "Five", "for", false), + Entry("a stop word is no content", "for the", "for a", false), + Entry("unrelated sentences", "The weather is sunny.", "I do not know.", false), + ) +}) + +var _ = func() bool { + if isRealS2S() { + defineS2SSpecs(realS2SVariant()) + return true + } + for _, v := range []s2sVariant{ + {label: "mock any-to-any vad", model: "realtime-s2s-mock", vad: true}, + {label: "mock any-to-any no-vad", model: "realtime-s2s-mock-novad"}, + {label: "mock pipeline vad", model: "realtime-pipeline", pipeline: true, vad: true}, + {label: "mock pipeline no-vad", model: "realtime-pipeline-novad", pipeline: true}, + } { + defineS2SSpecs(v) + } + return true +}() diff --git a/tests/e2e/run-realtime-lfm3.sh b/tests/e2e/run-realtime-lfm3.sh new file mode 100755 index 000000000..16b0b9c4a --- /dev/null +++ b/tests/e2e/run-realtime-lfm3.sh @@ -0,0 +1,248 @@ +#!/bin/bash +# Real-weights realtime speech-to-speech e2e (manual run, never part of CI). +# +# Drives tests/e2e/realtime_s2s_test.go (the "Realtime S2S (real ...)" specs) +# against LFM3-Audio served by the llama-cpp backend, for each shape of the +# gallery entries: {any-to-any, chained pipeline} x {VAD, no VAD}. The any-to-any +# model takes the audio and answers with audio; the pipeline runs transcription, +# chat and speech synthesis as separate stages on the same model. +# The test client generates its input speech with a sherpa-onnx TTS and checks +# the reply audio with a separate sherpa-onnx ASR. The VAD is silero. +# +# Pre-requisites: +# - `make build-mock-backend` and `make protogen-go` (the suite needs them) +# - `make docker-build-llama-cpp` -> local-ai-backend:llama-cpp +# - `make docker-build-sherpa-onnx` -> local-ai-backend:sherpa-onnx +# - `ffmpeg` on PATH of the machine that runs this script: LocalAI converts +# the input audio with it, so the run fails without it. +# Build the llama-cpp image for your hardware first (for example +# BUILD_TYPE=cublas CUDA_MAJOR_VERSION=13 make docker-build-llama-cpp); a CPU +# image works too, but a real turn then takes minutes. +# Environment overrides: +# LFM3_VARIANT q4 (default, lfm3-audio-2b) or q8 (lfm3-audio-2b-q8) +# LFM3_GPU_LAYERS Written as gpu_layers in the model config (default: unset) +# LLAMA_IMAGE Default local-ai-backend:llama-cpp +# SHERPA_IMAGE Default local-ai-backend:sherpa-onnx +# WORK_DIR Staging directory (default: a fresh temporary directory). +# KEEP_WORK Non-empty keeps WORK_DIR after the run; point WORK_DIR at it +# on the next run to skip the downloads. +# LFM3_MATRIX Shapes to run, kind:vad separated by spaces (default: all four: +# "any-to-any:vad any-to-any:novad pipeline:vad pipeline:novad"). +# GINKGO_FOCUS Ginkgo focus regexp (default: "Realtime S2S \(real"). +set -euo pipefail + +if ! command -v ffmpeg >/dev/null 2>&1; then + echo "ffmpeg not found on PATH: the real realtime run converts the input audio with it. Install ffmpeg and retry." >&2 + exit 2 +fi + +ROOT=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd) +LLAMA_IMAGE=${LLAMA_IMAGE:-local-ai-backend:llama-cpp} +SHERPA_IMAGE=${SHERPA_IMAGE:-local-ai-backend:sherpa-onnx} +VARIANT=${LFM3_VARIANT:-q4} +VAD_MODEL=silero-vad-sherpa +ASR_MODEL=omnilingual-0.3b-ctc-q8-sherpa +TTS_MODEL=vits-ljs-sherpa + +# Hugging Face revision and hashes match the gallery entries lfm3-audio-2b(-q8). +# The VAD and pipeline shapes mirror the -vad, -pipeline and -pipeline-vad entries. +HF=https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/fcfae9027c00a6f3f872c48a841cfeb77e2b5a1c +MMPROJ=mmproj-LFM3-Audio-2B-Q8_0.gguf +MMPROJ_SHA=26601e904245a85bd8f3cf320a629e897cfc01c1d09369e8b4fd951f13024d3e +case "$VARIANT" in + q4) S2S_MODEL=lfm3-audio-2b + GGUF=LFM3-Audio-2B-Q4_K_M.gguf + GGUF_SHA=9a773289bd321ed879cd3a76545f6aca506e403851a45fb315420aee45625be2 ;; + q8) S2S_MODEL=lfm3-audio-2b-q8 + GGUF=LFM3-Audio-2B-Q8_0.gguf + GGUF_SHA=d4e36902ba8f0ee7b333596030a59d650f88ac76660b4d9ba5ff77c3c2258fc0 ;; + *) echo "LFM3_VARIANT must be q4 or q8" >&2; exit 2 ;; +esac + +WORK_DIR=${WORK_DIR:-$(mktemp -d -t localai-lfm3-realtime.XXXXXX)} +CIDS=() +cleanup() { + for c in "${CIDS[@]}"; do docker rm -f "$c" >/dev/null 2>&1 || true; done + [[ -z "${KEEP_WORK:-}" ]] && rm -rf "$WORK_DIR" +} +trap cleanup EXIT +echo "WORK_DIR=$WORK_DIR" + +BACKENDS_DIR="$WORK_DIR/backends" +MODELS_DIR="$WORK_DIR/models" +mkdir -p "$BACKENDS_DIR" "$MODELS_DIR" + +# 1. Extract each backend image rootfs (same approach as run-realtime-sherpa.sh). +extract_backend() { + local name="$1" image="$2" dir="$BACKENDS_DIR/$1" + if [[ -x "$dir/run.sh" ]]; then + echo "cached backend: $dir" + return + fi + mkdir -p "$dir" + echo "Extracting $image into $dir ..." + local cid + cid=$(docker create --entrypoint=/run.sh "$image") + CIDS+=("$cid") + docker export "$cid" | tar -xC "$dir" --exclude='dev/*' --exclude='proc/*' --exclude='sys/*' + docker rm -f "$cid" >/dev/null + chmod +x "$dir/run.sh" 2>/dev/null || true +} +extract_backend llama-cpp "$LLAMA_IMAGE" +extract_backend sherpa-onnx "$SHERPA_IMAGE" + +# 2. Download model files, verifying sha256. +download() { + local dst="$1" url="$2" sha="$3" + if [[ -f "$dst" && "$(sha256sum "$dst" | awk '{print $1}')" == "$sha" ]]; then + echo "cached: $dst" + return + fi + mkdir -p "$(dirname "$dst")" + echo "downloading: $url -> $dst" + curl -sSfL "$url" -o "$dst" + local actual + actual=$(sha256sum "$dst" | awk '{print $1}') + if [[ "$actual" != "$sha" ]]; then + echo "sha256 mismatch for $dst: got $actual, expected $sha" >&2 + exit 1 + fi +} + +download "$MODELS_DIR/$GGUF" "$HF/$GGUF" "$GGUF_SHA" +download "$MODELS_DIR/$MMPROJ" "$HF/$MMPROJ" "$MMPROJ_SHA" + +download "$MODELS_DIR/silero-vad/silero-vad.onnx" \ + "https://huggingface.co/onnx-community/silero-vad/resolve/main/onnx/model.onnx" \ + "a4a068cd6cf1ea8355b84327595838ca748ec29a25bc91fc82e6c299ccdc5808" +download "$MODELS_DIR/omnilingual-asr/model.int8.onnx" \ + "https://huggingface.co/csukuangfj/sherpa-onnx-omnilingual-asr-1600-languages-300M-ctc-int8-2025-11-12/resolve/main/model.int8.onnx" \ + "e7c4e54ee4c4c47829cc6667d5d00ed8ea7bef1dcfeef0fce766f77752a2726c" +download "$MODELS_DIR/omnilingual-asr/tokens.txt" \ + "https://huggingface.co/csukuangfj/sherpa-onnx-omnilingual-asr-1600-languages-300M-ctc-int8-2025-11-12/resolve/main/tokens.txt" \ + "a7a044c52cb29cbe8b0dc1953e92cefd4ca16b0ed968177b6beab21f9a7d0b31" +download "$MODELS_DIR/vits-ljs/vits-ljs.onnx" \ + "https://huggingface.co/csukuangfj/vits-ljs/resolve/main/vits-ljs.onnx" \ + "5bbd273797a9ecf8d94bd6ec02ad16cb41cbb85f055ad98d528ced3e44c9b31a" +download "$MODELS_DIR/vits-ljs/tokens.txt" \ + "https://huggingface.co/csukuangfj/vits-ljs/resolve/main/tokens.txt" \ + "5fee2c6b238d712287f2ecb08f34a8a8b413bcb7390862ef6fb6fd6f0f8d3a17" +download "$MODELS_DIR/vits-ljs/lexicon.txt" \ + "https://huggingface.co/csukuangfj/vits-ljs/resolve/main/lexicon.txt" \ + "bdccfc6da71c45c48e2e0056fcf0aab760577c5f959f6c1b5eb3e3e916fd5a0e" + +# 3. Model configs. One config per shape, each mirroring the gallery overrides +# of the entry with the same name (the gallery names its VAD silero-vad-ggml; +# this script uses the sherpa silero model its extracted backend already has). +# $S2S_MODEL any-to-any, no VAD (lfm3-audio-2b[-q8]) +# $S2S_MODEL-vad any-to-any, VAD +# $S2S_MODEL-pipeline chained pipeline on the same model, no VAD +# $S2S_MODEL-pipeline-vad chained pipeline, VAD +GPU_LINE="" +[[ -n "${LFM3_GPU_LAYERS:-}" ]] && GPU_LINE="gpu_layers: $LFM3_GPU_LAYERS" + +write_model_config() { + local name="$1" kind="$2" vad="$3" usecases pipeline + if [[ "$kind" == "pipeline" ]]; then + usecases=" - chat + - transcript + - tts" + pipeline="pipeline: + llm: $name + transcription: $name + tts: $name" + else + usecases=" - realtime_audio + - chat + - transcript + - tts" + # the input transcript of an any-to-any model comes from a model of its + # own: on the model itself it would share the llama.cpp context with the reply + pipeline="pipeline: + any_to_any: true + transcription: $ASR_MODEL" + fi + [[ "$vad" == "1" ]] && pipeline="$pipeline + vad: $VAD_MODEL" + cat > "$MODELS_DIR/$name.yaml" < "$MODELS_DIR/$VAD_MODEL.yaml" < "$MODELS_DIR/$ASR_MODEL.yaml" < "$MODELS_DIR/$TTS_MODEL.yaml" <&2; exit 2; } + echo "=== $shape -> model $name ===" + REALTIME_S2S_MODEL="$name" REALTIME_S2S_KIND="$kind" REALTIME_S2S_VAD="$vad" \ + go test -v -count=1 -timeout 120m ./tests/e2e/ \ + -ginkgo.label-filter="Realtime && real-models" \ + -ginkgo.focus="${GINKGO_FOCUS:-Realtime S2S \(real}" \ + -ginkgo.v || status=1 +done +exit $status