mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 10:28:43 -04:00
8778600295
The one bidirectional stream this backend serves. The client sends a
TranscriptLiveConfig, then TranscriptLiveAudio frames; the server acknowledges
with ready, emits deltas as the audio arrives, and sends final_result once the
read side closes. There is no offline fallback: live transcription has to
consume audio incrementally, so a family with no streaming ASR is refused
rather than served a batch run, which is what this RPC's Streaming-only
mode_candidates list already says.
The driver is a new sibling of run_streaming_audio, run_streaming_live, because
the audio does not exist yet: instead of slicing a buffer it pulls frames from
the caller until the read side closes. It installs the same ScopedStreamSink in
the same order, which is not optional, since nemotron_asr returns a bare event
from process_audio_chunk and reports every partial through the sink from inside
finalize(). It buffers the wire's frames up to the family's own preferred window
rather than feeding whatever size the client's audio callback produced, and it
does not call finish_stream at all when no audio arrived, because nemotron_asr
throws "finalize requires streamed audio" and an empty transcript is the
truthful answer to transcribing nothing.
Three things the handler had to get right and one it cannot:
- The audio contract. A live request carries no samples, but nemotron_asr's
streaming prepare() throws without an audio contract, and
build_preparation_request derives it from TaskRequest::audio_input, so that
field is an EMPTY buffer holding only the rate and the channel count.
- 16 kHz or a refusal. The families express their spans in their own 16 kHz
feature domain whatever the input was, and live frames cannot be resampled
on the way in the way a file can, so an 8 kHz session would return
timestamps 2x off with a 200. core/backend hardcodes 16000 anyway.
- A mid-stream Config is refused. backend.proto calls it a decoder reset, but
deltas already on the wire cannot be retracted, so a reset would leave the
final text contradicting the transcript the client assembled. Ignoring the
message would hand a client that believes it reset the decoder a transcript
that silently continues the audio it thought it discarded.
- The stale-route identity check cannot run here: TranscriptLiveRequest
carries no ModelIdentity in either arm of its oneof, so snapshot_for does
not instantiate for it. snapshot_unchecked's comment now names that as a
second legitimate class of caller and says the fix is a proto change.
eou and eob stay false. They exist for cache-aware models that emit
end-of-utterance and end-of-backchannel tokens; audio.cpp's StreamEvent has no
equivalent signal, and a client uses eou to decide the speaker yielded the turn,
so a guess inferred from silence cuts people off mid-sentence.
The lane is held for the whole stream, which is as long as the user keeps
talking: the streaming session is stateful and cached, so a concurrent run would
interleave two callers' audio and corrupt both transcripts.
Verified against nemotron_asr over a real connection with a 14 s WAV in
512-sample frames: ready first, 59 incremental deltas with no repeated prefix,
concat(deltas) equal to final_result.text, word timestamps in nanoseconds, eou
and eob false. citrinet_asr answers UNIMPLEMENTED naming the family and listing
asr/offline. A config followed by a close returns an empty final_result rather
than hanging, and a first message that is not a config is INVALID_ARGUMENT. Two
concurrent streams both return the complete transcript.
Two cleanups on lines Task 12 touched, folded in. The DtypeAllowList terminator
is now asserted at compile time: the reported out-of-bounds read did not exist,
the single entry does terminate, but the loops have no other bound and any edit
that widened an entry would walk off the end. And the dtype guard now
short-circuits on "is there a table entry" through a new predicate rather than
on the emptiness of the description string, which would have skipped the check
on an entry with an empty allow list, i.e. on precisely the entry that refuses
every dtype.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
72 lines
3.5 KiB
C++
72 lines
3.5 KiB
C++
#pragma once
|
|
|
|
// Decides which audio.cpp family a model path belongs to, and refuses paths
|
|
// this backend must not claim. Standard library only.
|
|
//
|
|
// This is the guard against issue #9287. A model config with no explicit
|
|
// backend makes LocalAI probe every installed backend and bind to the first
|
|
// Load that succeeds, so accepting an arbitrary GGUF here would capture
|
|
// unrelated LLMs. audio.cpp GGUFs carry an audiocpp.model_spec.family metadata
|
|
// key; llama.cpp GGUFs do not.
|
|
|
|
#include <string>
|
|
|
|
namespace audiocpp_backend {
|
|
|
|
// True when the path ends in ".gguf", case insensitively, and has a stem.
|
|
bool path_looks_like_gguf(const std::string &path);
|
|
|
|
struct FamilyDecision {
|
|
bool ok = false;
|
|
std::string family;
|
|
// Set when ok is false. Suitable verbatim as an INVALID_ARGUMENT message.
|
|
std::string error;
|
|
};
|
|
|
|
// Precedence:
|
|
// 1. an explicit `family:` option, so a user can override wrong metadata;
|
|
// 2. for a GGUF, the family embedded in audiocpp.model_spec.family;
|
|
// 3. otherwise refuse.
|
|
// A directory path never consults embedded metadata: there is no single GGUF
|
|
// to read it from.
|
|
FamilyDecision decide_family(bool path_is_gguf, const std::string &embedded_family,
|
|
const std::string &configured_family);
|
|
|
|
// True when `family` can run weights stored as `dtype`, where dtype is the
|
|
// string a TensorMetadata carries ("f32", "f16", "q8_0", "i64", ...).
|
|
//
|
|
// This is a LIST OF FAMILIES THAT CRASH THE PROCESS, not a list of families that
|
|
// perform badly. It exists because the failure is not an exception: loading a
|
|
// supertonic package whose weights are f16 or q8_0 reaches ggml_concat with one
|
|
// f16 operand and one f32 one, and ggml_abort takes the backend down with
|
|
// SIGABRT on the FIRST request. Nothing upstream of the load can catch that, so
|
|
// an operator sees a model that loaded successfully and a backend that dies on
|
|
// every request with no status and no message.
|
|
//
|
|
// A family with no entry is unrestricted, which is every family but one.
|
|
//
|
|
// Split out of loaded_model.cpp, where the caller lives, so that the policy is
|
|
// stdlib-only and can be held by a test: the caller needs a real GGUF on disk
|
|
// and an engine, and neither is available to a unit test. What the test pins is
|
|
// that the table says what it is meant to say, so widening it is a deliberate
|
|
// act rather than a typo. It CANNOT pin the removal criterion, which is
|
|
// "upstream fixed it": no test can know that without downloading the package and
|
|
// synthesising, so that step stays a documented manual one at the table itself.
|
|
bool weight_dtype_is_supported(const std::string &family, const std::string &dtype);
|
|
|
|
// True when `family` has an entry in the table at all, which is the question a
|
|
// caller deciding whether to OPEN THE FILE has to ask. Distinct from
|
|
// "supported_weight_dtypes(family) is empty": that string is also empty for an
|
|
// entry with an empty allow list, and such an entry means "this family can run
|
|
// nothing", which weight_dtype_is_supported already answers by refusing every
|
|
// dtype. Deciding from the string would skip the check on precisely the entry
|
|
// that most needs it.
|
|
bool family_has_weight_dtype_allow_list(const std::string &family);
|
|
|
|
// The dtypes `family` is restricted to, as "f32, i64", or empty when it is not
|
|
// restricted at all. For the refusal message, so the operator is told what to
|
|
// look for rather than only what is wrong.
|
|
std::string supported_weight_dtypes(const std::string &family);
|
|
|
|
} // namespace audiocpp_backend
|