mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 10:28:43 -04:00
a996d7dd23
TTSStream leads with a streaming WAV header carrying 0xFFFFFFFF sizes, matching the convention backend/go/vibevoice-cpp established, so an HTTP client can start playback before the full PCM exists. Its chunks are read from StreamEvent::named_audio_outputs and not audio_output: supertonic, omnivoice and voxcpm2 all put their streamed audio there and leave audio_output empty until the very end, so reading the obvious field yields a stream with no audio in it. The finish_stream result is the family's own merged whole rather than a tail, so it is emitted only when nothing was streamed. Streaming transcription sends incremental deltas and degrades to a single delta plus the final result on families that offer no streaming ASR, which is the same message sequence with fewer deltas. The four streaming ASR families disagree on what partial_text means: nemotron_asr, vibevoice_asr and higgs_audio_stt report incremental fragments while voxtral_realtime reports the whole hypothesis and reports it twice, so the reconciliation lives in one tested unit rather than in the handler. nemotron_asr reports only through the stream event sink, and only from inside finalize, so the audio driver installs one and clears it again before returning: the session is cached and a sink left holding the caller's frame is a use after free waiting for the next stream. begin_stream is now the only implementation of the streaming state obligation, prepare then start_stream. Streaming sessions are cached, and what clears the previous stream is start_stream's reset; a family override that dropped it would break every call site with no compile error, so there is one call site. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
81 lines
3.9 KiB
C++
81 lines
3.9 KiB
C++
#pragma once
|
|
|
|
// Turns whatever a streaming session calls a "partial transcript" into the
|
|
// incremental deltas AudioTranscriptionStream is contracted to send. Standard
|
|
// library only, so it is tested without an audio.cpp checkout.
|
|
//
|
|
// THIS UNIT EXISTS BECAUSE THE FAMILIES DISAGREE, and the disagreement is
|
|
// invisible at the interface: StreamEvent::partial_text is a Transcript either
|
|
// way. Read out of the pinned upstream, one family at a time:
|
|
//
|
|
// nemotron_asr INCREMENTAL. decoder.cpp emits
|
|
// current_text.substr(emitted_text.size()) per non-blank
|
|
// token, and only through the stream event SINK, during
|
|
// finalize(). process_audio_chunk returns empty events.
|
|
// vibevoice_asr INCREMENTAL. process_audio_chunk returns
|
|
// text.substr(common_prefix_size(...)); the sink is
|
|
// deliberately swapped out around its internal run_single,
|
|
// so the fragment arrives once, on the return value.
|
|
// higgs_audio_stt INCREMENTAL. Same shape as vibevoice_asr.
|
|
// voxtral_realtime CUMULATIVE. partial_text is
|
|
// tokenizer_.decode(streaming_token_ids_), the whole
|
|
// hypothesis so far, and process_available_stream_chunks
|
|
// hands the SAME event to the sink AND returns it, so every
|
|
// partial arrives TWICE.
|
|
//
|
|
// Applying either convention to the other family corrupts the transcript: read
|
|
// a cumulative report as a delta and the client sees the transcript repeated on
|
|
// every event; read an incremental fragment as cumulative and the suffix
|
|
// arithmetic eats the front of it. So the tracker decides per fragment, from
|
|
// what it has already delivered, and the one rule it enforces is that TEXT THE
|
|
// CLIENT HAS ALREADY BEEN SENT IS NEVER SENT AGAIN.
|
|
|
|
#include <string>
|
|
|
|
namespace audiocpp_backend {
|
|
|
|
class TranscriptDeltaTracker {
|
|
public:
|
|
// Takes one StreamEvent::partial_text and returns the fragment to put on
|
|
// the wire, empty when there is nothing new.
|
|
//
|
|
// The rules, in order:
|
|
// 1. An empty partial says nothing.
|
|
// 2. A partial the assembly ALREADY STARTS WITH has been delivered:
|
|
// nothing is emitted. This is what absorbs voxtral's double delivery
|
|
// of every event, and a cumulative hypothesis that shrinks.
|
|
// 3. A partial that EXTENDS the assembly is a cumulative report: only its
|
|
// new suffix is emitted.
|
|
// 4. Anything else is an incremental fragment: it is emitted whole and
|
|
// appended.
|
|
//
|
|
// Rule 3 is the one judgement call, since a fragment that happens to begin
|
|
// with the entire transcript so far is indistinguishable from a cumulative
|
|
// report. It is read as cumulative because every cumulative family produces
|
|
// that shape on EVERY event, while an incremental family produces it only
|
|
// when one fragment repeats everything before it, which no tokenizer output
|
|
// does in practice.
|
|
std::string observe(const std::string &partial_text);
|
|
|
|
// Reconciles against TaskResult::text_output, which is authoritative, and
|
|
// returns the fragment that makes appending every delta equal it.
|
|
//
|
|
// This is what makes the OFFLINE FALLBACK a single line rather than its own
|
|
// branch: with no partials observed, the assembly is empty and the whole
|
|
// final text comes back as one delta.
|
|
//
|
|
// A final text that CONTRADICTS what was already sent returns empty. A
|
|
// fragment on the wire cannot be retracted, so the alternative would be to
|
|
// send the transcript a second time and let the client hold it twice.
|
|
// final_result carries the authoritative text either way.
|
|
std::string reconcile(const std::string &final_text);
|
|
|
|
// Everything the client has been sent, concatenated.
|
|
const std::string &assembled() const noexcept { return assembled_; }
|
|
|
|
private:
|
|
std::string assembled_;
|
|
};
|
|
|
|
} // namespace audiocpp_backend
|