mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 10:28:43 -04:00
7ce4b36837
Segment text is not decoration. core/http/endpoints/openai/transcription.go
routes response_format text, srt, vtt and lrc through
schema.TranscriptionResponse, which builds the entire body out of
Segments[].Text and never reads the top-level text. So for those four
formats the segment text IS the response.
nemotron_asr emits one word_timestamp per SentencePiece token, and the
word boundary is carried as a LEADING SPACE on the piece ("So", "me",
" call"). join_words inserted a space unconditionally, so
response_format=text returned "So me call me na ture ," while the
correct sentence sat unread in the top-level field. The separator is now
chosen from the words themselves: whole words are space-joined, subword
pieces are concatenated, and one leading space anywhere selects the
latter. Concatenating the real nemotron pieces reproduces text_output
exactly, verified end to end.
This does not touch the top-level text, which is still text_output
verbatim. The rule that forbids deriving the transcript from the segments
is about the direction segments -> text; segment text has no source other
than its words.
Two smaller corrections in the same area:
timestamp_granularities ["word"] set only "word_timestamps", a key no
family in the pinned upstream reads. It now sets "return_timestamps",
which qwen3_asr does read and which both runs its forced aligner and
shortens its chunk window, so asking for word granularity no longer
silently returns nothing.
The request-option comment claimed more than it delivered. prompt,
translate and temperature are read by no ASR family, and are forwarded
only so a family adopting them works unchanged; the comment now says so
per key, and gives TranscriptRequest.diarize the same explicit treatment
threads already had.
Also: the shipping target now carries -Wall -Wextra -Wpedantic, which it
never did, so "the build is clean" starts meaning something; and
fill_transcript_result no longer swallows a null response pointer, since
answering OK with an empty transcript is the one failure mode this unit
exists to prevent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
88 lines
3.1 KiB
C++
88 lines
3.1 KiB
C++
#pragma once
|
|
|
|
// Builds LocalAI's TranscriptResult shape from audio.cpp's TaskResult spans.
|
|
// Standard library only; result_map.cpp converts the engine types into these
|
|
// PODs at the boundary.
|
|
//
|
|
// THE RULE: the top-level transcript text is text_output verbatim, always.
|
|
// audio.cpp carries transcript text in exactly one place, TaskResult.text_output.
|
|
// speech_segments, speaker_turns and word_timestamps carry spans and labels but
|
|
// no text. Deriving the top-level text by concatenating per-segment text
|
|
// therefore yields an empty transcript for every producer that reports segments
|
|
// without word timestamps, which includes VibeVoice diarized ASR.
|
|
|
|
#include <cstdint>
|
|
#include <string>
|
|
#include <vector>
|
|
|
|
namespace audiocpp_backend {
|
|
|
|
// Sample-index span, mirroring engine::runtime::TimeSpan.
|
|
struct Span {
|
|
std::int64_t start_sample = 0;
|
|
std::int64_t end_sample = 0;
|
|
};
|
|
|
|
struct WordSpan {
|
|
Span span;
|
|
std::string word;
|
|
};
|
|
|
|
struct SpeakerSpan {
|
|
Span span;
|
|
std::string speaker;
|
|
};
|
|
|
|
struct OutWord {
|
|
std::int64_t start_ns = 0;
|
|
std::int64_t end_ns = 0;
|
|
std::string text;
|
|
};
|
|
|
|
struct OutSegment {
|
|
int id = 0;
|
|
std::int64_t start_ns = 0;
|
|
std::int64_t end_ns = 0;
|
|
std::string text;
|
|
std::string speaker;
|
|
std::vector<OutWord> words;
|
|
};
|
|
|
|
struct AssembledTranscript {
|
|
std::string text;
|
|
std::vector<OutSegment> segments;
|
|
};
|
|
|
|
// Segment source, first non-empty wins:
|
|
// 1. speech_segments
|
|
// 2. speaker_turns
|
|
// 3. one segment spanning all words, when words are present
|
|
// 4. one zero-span segment carrying the full text, when text is present
|
|
// 5. no segments
|
|
//
|
|
// Words attach to the segment whose range contains their midpoint; a word
|
|
// outside every segment attaches to the nearest one by midpoint distance so it
|
|
// is never silently dropped. A lone segment with no words carries the full text.
|
|
//
|
|
// A segment's text is its words joined, and the separator depends on the
|
|
// producer's convention: whole words ("Some", "call") are joined with a space,
|
|
// while SentencePiece-style subword pieces, which carry the word boundary as a
|
|
// LEADING SPACE (" call"), are concatenated. One leading space anywhere in the
|
|
// segment selects concatenation. This matters beyond tidiness: response_format
|
|
// text, srt, vtt and lrc build their entire body out of the segment text and
|
|
// never read the top-level text.
|
|
//
|
|
// A segment's speaker is the speaker turn with the greatest overlap, except
|
|
// when the segments came from the speaker turns themselves (source 2), where
|
|
// each segment keeps its own turn's label. Re-deriving it there loses a turn
|
|
// nested inside another speaker's turn: the nested turn overlaps its own span
|
|
// completely, so it can only tie with the containing turn, which is listed
|
|
// first and wins the tie.
|
|
AssembledTranscript assemble_transcript(const std::string &text_output,
|
|
const std::vector<Span> &speech_segments,
|
|
const std::vector<SpeakerSpan> &speaker_turns,
|
|
const std::vector<WordSpan> &words,
|
|
int sample_rate);
|
|
|
|
} // namespace audiocpp_backend
|