mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 02:18:50 -04:00
41b4e72041
TTSStream leads with a streaming WAV header carrying 0xFFFFFFFF sizes, matching the convention backend/go/vibevoice-cpp established, so an HTTP client can start playback before the full PCM exists. Its chunks are read from StreamEvent::named_audio_outputs and not audio_output: supertonic, omnivoice and voxcpm2 all put their streamed audio there and leave audio_output empty until the very end, so reading the obvious field yields a stream with no audio in it. The finish_stream result is the family's own merged whole rather than a tail, so it is emitted only when nothing was streamed. Streaming transcription sends incremental deltas and degrades to a single delta plus the final result on families that offer no streaming ASR, which is the same message sequence with fewer deltas. The four streaming ASR families disagree on what partial_text means: nemotron_asr, vibevoice_asr and higgs_audio_stt report incremental fragments while voxtral_realtime reports the whole hypothesis and reports it twice, so the reconciliation lives in one tested unit rather than in the handler. nemotron_asr reports only through the stream event sink, and only from inside finalize, so the audio driver installs one and clears it again before returning: the session is cached and a sink left holding the caller's frame is a use after free waiting for the next stream. begin_stream is now the only implementation of the streaming state obligation, prepare then start_stream. Streaming sessions are cached, and what clears the previous stream is start_stream's reset; a family override that dropped it would break every call site with no compile error, so there is one call site. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
40 lines
1.9 KiB
C++
40 lines
1.9 KiB
C++
#pragma once
|
|
|
|
// Builds the 44 byte canonical WAV header that precedes a streamed PCM body.
|
|
// Standard library only.
|
|
//
|
|
// TTSStream chunks travel in Reply.audio and the FIRST chunk must be this
|
|
// header, or an HTTP client has no format to decode the PCM with. Because the
|
|
// total length is unknown while the model is still generating, both size fields
|
|
// carry 0xFFFFFFFF; that is the convention backend/go/vibevoice-cpp established
|
|
// (govibevoicecpp.go, TTSStream) and the one core/backend/tts.go writes when it
|
|
// synthesises a header itself.
|
|
//
|
|
// WHY THIS BACKEND SENDS THE HEADER RATHER THAN LETTING GO DO IT.
|
|
// core/backend/tts.go's ModelTTSStream will emit a header of its own, but only
|
|
// when the FIRST Reply carries a non-empty `message` field holding a JSON blob
|
|
// with a sample_rate. This backend sends audio and never sets `message`, so
|
|
// that branch never fires and there is exactly one header on the wire: this
|
|
// one. Do not start setting Reply.message on this RPC without deleting the
|
|
// header below, or every stream gains a second header 44 bytes into the PCM.
|
|
//
|
|
// The header this produces is byte-identical to the one pkg/audio.WAVHeader
|
|
// serialises, which is what core/http/endpoints/openai/realtime_model.go
|
|
// assumes when it reads the sample rate out of byte offset 24 of the first
|
|
// callback.
|
|
|
|
#include <string>
|
|
|
|
namespace audiocpp_backend {
|
|
|
|
// 16-bit PCM, little endian, interleaved.
|
|
//
|
|
// `channels` is clamped to at least 1 and at most 65535, so a garbage channel
|
|
// count can never write a zero block align, which is what a reader divides the
|
|
// data size by. `sample_rate` is clamped to at least 0 rather than wrapped:
|
|
// a zero rate is visibly wrong to whoever reads the header, whereas the
|
|
// 4294967295 an unsigned conversion of -1 would write looks like a real field.
|
|
std::string streaming_wav_header(int sample_rate, int channels);
|
|
|
|
} // namespace audiocpp_backend
|