mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 18:09:05 -04:00
8da5cde0c0
The one bidirectional stream this backend serves. The client sends a
TranscriptLiveConfig, then TranscriptLiveAudio frames; the server acknowledges
with ready, emits deltas as the audio arrives, and sends final_result once the
read side closes. There is no offline fallback: live transcription has to
consume audio incrementally, so a family with no streaming ASR is refused
rather than served a batch run, which is what this RPC's Streaming-only
mode_candidates list already says.
The driver is a new sibling of run_streaming_audio, run_streaming_live, because
the audio does not exist yet: instead of slicing a buffer it pulls frames from
the caller until the read side closes. It installs the same ScopedStreamSink in
the same order, which is not optional, since nemotron_asr returns a bare event
from process_audio_chunk and reports every partial through the sink from inside
finalize(). It buffers the wire's frames up to the family's own preferred window
rather than feeding whatever size the client's audio callback produced, and it
does not call finish_stream at all when no audio arrived, because nemotron_asr
throws "finalize requires streamed audio" and an empty transcript is the
truthful answer to transcribing nothing.
Three things the handler had to get right and one it cannot:
- The audio contract. A live request carries no samples, but nemotron_asr's
streaming prepare() throws without an audio contract, and
build_preparation_request derives it from TaskRequest::audio_input, so that
field is an EMPTY buffer holding only the rate and the channel count.
- 16 kHz or a refusal. The families express their spans in their own 16 kHz
feature domain whatever the input was, and live frames cannot be resampled
on the way in the way a file can, so an 8 kHz session would return
timestamps 2x off with a 200. core/backend hardcodes 16000 anyway.
- A mid-stream Config is refused. backend.proto calls it a decoder reset, but
deltas already on the wire cannot be retracted, so a reset would leave the
final text contradicting the transcript the client assembled. Ignoring the
message would hand a client that believes it reset the decoder a transcript
that silently continues the audio it thought it discarded.
- The stale-route identity check cannot run here: TranscriptLiveRequest
carries no ModelIdentity in either arm of its oneof, so snapshot_for does
not instantiate for it. snapshot_unchecked's comment now names that as a
second legitimate class of caller and says the fix is a proto change.
eou and eob stay false. They exist for cache-aware models that emit
end-of-utterance and end-of-backchannel tokens; audio.cpp's StreamEvent has no
equivalent signal, and a client uses eou to decide the speaker yielded the turn,
so a guess inferred from silence cuts people off mid-sentence.
The lane is held for the whole stream, which is as long as the user keeps
talking: the streaming session is stateful and cached, so a concurrent run would
interleave two callers' audio and corrupt both transcripts.
Verified against nemotron_asr over a real connection with a 14 s WAV in
512-sample frames: ready first, 59 incremental deltas with no repeated prefix,
concat(deltas) equal to final_result.text, word timestamps in nanoseconds, eou
and eob false. citrinet_asr answers UNIMPLEMENTED naming the family and listing
asr/offline. A config followed by a close returns an empty final_result rather
than hanging, and a first message that is not a config is INVALID_ARGUMENT. Two
concurrent streams both return the complete transcript.
Two cleanups on lines Task 12 touched, folded in. The DtypeAllowList terminator
is now asserted at compile time: the reported out-of-bounds read did not exist,
the single entry does terminate, but the loops have no other bound and any edit
that widened an entry would walk off the end. And the dtype guard now
short-circuits on "is there a table entry" through a new predicate rather than
on the emptiness of the description string, which would have skipped the check
on an entry with an empty allow list, i.e. on precisely the entry that refuses
every dtype.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
5.6 KiB
5.6 KiB