mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 02:18:50 -04:00
backend(audio-cpp): fix the segment text a transcription response is built from
Segment text is not decoration. core/http/endpoints/openai/transcription.go
routes response_format text, srt, vtt and lrc through
schema.TranscriptionResponse, which builds the entire body out of
Segments[].Text and never reads the top-level text. So for those four
formats the segment text IS the response.
nemotron_asr emits one word_timestamp per SentencePiece token, and the
word boundary is carried as a LEADING SPACE on the piece ("So", "me",
" call"). join_words inserted a space unconditionally, so
response_format=text returned "So me call me na ture ," while the
correct sentence sat unread in the top-level field. The separator is now
chosen from the words themselves: whole words are space-joined, subword
pieces are concatenated, and one leading space anywhere selects the
latter. Concatenating the real nemotron pieces reproduces text_output
exactly, verified end to end.
This does not touch the top-level text, which is still text_output
verbatim. The rule that forbids deriving the transcript from the segments
is about the direction segments -> text; segment text has no source other
than its words.
Two smaller corrections in the same area:
timestamp_granularities ["word"] set only "word_timestamps", a key no
family in the pinned upstream reads. It now sets "return_timestamps",
which qwen3_asr does read and which both runs its forced aligner and
shortens its chunk window, so asking for word granularity no longer
silently returns nothing.
The request-option comment claimed more than it delivered. prompt,
translate and temperature are read by no ASR family, and are forwarded
only so a family adopting them works unchanged; the comment now says so
per key, and gives TranscriptRequest.diarize the same explicit treatment
threads already had.
Also: the shipping target now carries -Wall -Wextra -Wpedantic, which it
never did, so "the build is clean" starts meaning something; and
fill_transcript_result no longer swallows a null response pointer, since
answering OK with an empty transcript is the one failure mode this unit
exists to prevent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
committed by
localai-org-maint-bot
parent
c5dcb00b00
commit
7ce4b36837
@@ -113,11 +113,14 @@ constexpr int kVadSampleRate = 16000;
|
||||
// one, which is the only reason the nanoseconds are right.
|
||||
//
|
||||
// The cost, stated plainly: vibevoice_asr resamples internally to 24 kHz, so a
|
||||
// 48 kHz upload now travels 48 -> 16 -> 24 rather than 48 -> 24, losing the
|
||||
// 8-12 kHz band it could have kept. That is accepted because a silently wrong
|
||||
// timestamp is worse than a band-limited one, and because every other LocalAI
|
||||
// ASR path already feeds 16 kHz. If the framework ever publishes a per-model
|
||||
// preferred input rate, this constant is what should become that lookup.
|
||||
// 48 kHz upload now travels 48 -> 16 -> 24 rather than 48 -> 24. That is not
|
||||
// merely band limiting. resample_mono_linear is linear interpolation with no
|
||||
// anti-alias filter (framework/audio/conversion.cpp), so the content above
|
||||
// 8 kHz is ALIASED down on the way in, and the 16 -> 24 step aliases whatever
|
||||
// the first step left. It is accepted because a silently wrong timestamp is
|
||||
// worse than an aliased band, and because every other LocalAI ASR path already
|
||||
// feeds 16 kHz. If the framework ever publishes a per-model preferred input
|
||||
// rate, this constant is what should become that lookup.
|
||||
constexpr int kSpeechSampleRate = 16000;
|
||||
|
||||
// Parses ModelOptions.MainGPU into a device index.
|
||||
@@ -260,21 +263,47 @@ build_transcription_request(const backend::TranscriptRequest &request,
|
||||
task_request.options["temperature"] = std::to_string(request.temperature());
|
||||
}
|
||||
for (const auto &granularity : request.timestamp_granularities()) {
|
||||
// Upstream families read this as a request for word-level timing.
|
||||
if (granularity == "word") {
|
||||
// return_timestamps is the key upstream actually reads, and it is
|
||||
// the one that does something: qwen3_asr defaults it to false
|
||||
// (include/engine/models/qwen3_asr/types.h) and, when set, both
|
||||
// runs its forced aligner and shortens its chunk window from 30 s
|
||||
// to 15 s (src/models/qwen3_asr/session.cpp). Without it, asking
|
||||
// for word granularity silently came back with no word timing.
|
||||
//
|
||||
// word_timestamps rides along as a forward-tolerant alias. NO
|
||||
// family in the pinned upstream reads that key, a grep over src/
|
||||
// and include/ returns nothing, so it is sent only so that a family
|
||||
// adopting the name later works with no change here.
|
||||
task_request.options["return_timestamps"] = "true";
|
||||
task_request.options["word_timestamps"] = "true";
|
||||
}
|
||||
}
|
||||
// Every option above is advisory. audio.cpp families look their request
|
||||
// options up by name (runtime::find_option) and ignore the rest; the
|
||||
// What lands and what does not. Families look their request options up by
|
||||
// name (runtime::find_option) and ignore every key they do not know; the
|
||||
// unknown-key refusals upstream does have are on SESSION options, which
|
||||
// arrive at load time, not here. So an option a family does not read costs
|
||||
// nothing, and none of these can turn a valid request into an error.
|
||||
// arrive at load time rather than here. So an unread key cannot turn a
|
||||
// valid request into an error, but it is also not a feature, and the
|
||||
// honest accounting against the pinned upstream is:
|
||||
//
|
||||
// TranscriptRequest.threads is deliberately NOT forwarded: thread count is
|
||||
// a SessionOptions field fixed when the session was created, so a
|
||||
// per-request value has nowhere to go and pretending otherwise would be a
|
||||
// knob that silently does nothing.
|
||||
// language read, by nemotron_asr and vibevoice_asr among others.
|
||||
// return_timestamps read, by qwen3_asr.
|
||||
// prompt read by NO ASR family. Forwarded because it is the
|
||||
// whisper meaning of the field and a family adopting
|
||||
// it then works unchanged, not because it does
|
||||
// anything today.
|
||||
// translate read by nobody anywhere in upstream.
|
||||
// temperature read only by TTS and voice conversion families, none
|
||||
// of which this RPC can route to.
|
||||
//
|
||||
// Two request fields are deliberately not forwarded at all:
|
||||
//
|
||||
// threads thread count is a SessionOptions field fixed when the session
|
||||
// was built, so a per-request value has nowhere to go.
|
||||
// diarize this RPC routes to Asr or Alignment. A family that diarizes
|
||||
// is reached through Diarize, which has its own handler and its
|
||||
// own response shape, so forwarding this would imply that
|
||||
// setting it turns speaker labels on here, and nothing would.
|
||||
return task_request;
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user