mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 10:28:43 -04:00
The model's `task:` option is copied into the request shape by all nine
handlers, which is correct, but resolve_route then replaced the RPC's candidate
list with the pin WHOLESALE and never asked whether the pin was something that
RPC routes to. One pin therefore bled across all nine surfaces, and because the
family still supported the pinned task the result was a wrong 200 rather than an
error. Reproduced live: nemotron with task:asr made Vad return 200 with zero
segments after a full ASR decode, so 14 seconds of speech was reported as
silence, and Diarize did the same; silero_vad with task:vad made
AudioTranscription return 200 with empty text and four segments whose spans were
VAD segments, which combined with response_format in {text,srt,vtt,lrc} building
the body solely from Segments[].Text yields a well formed SRT of four timed
EMPTY cues. It also contradicted the documented contract, that a family which
cannot serve a request is refused rather than rerouted.
A pin is now checked against the RPC's admissible task set before it is adopted,
and the refusal names both the pin and the RPC. The set is derived from
task_candidates with every shape flag set rather than restated, so a task added
to an RPC's candidates cannot become inadmissible by omission. Every legitimate
pin survives, and the test asserts all fifteen of them alongside the eight
crossings that must not.
The live watchdog was defeated by empty frames. idle.touch() ran on ANY message,
before the has_audio and pcm.empty() filters, so a peer writing unset-oneof or
zero-length frames faster than the window held the lane indefinitely while
feeding the decoder nothing. There is one lane per model and one model per
process, so that is a single client denying the whole backend, which is what the
watchdog exists to prevent, and the thrown text already said "no audio frame
arrived". The touch moved below the filters, which are now a named predicate so
the distinction is testable rather than a call order nobody can see.
Three comments corrected against measurement rather than reasoning:
- CMakeLists claimed zero google::protobuf:: definitions remain in the
executable. nm -C --defined-only reports 2515, and that is expected: they are
generated code, sentencepiece::ModelProto's own _InternalParse among them. The
claim that holds, and the one the ABI fix is actually about, is that no
vendored protobuf RUNTIME is linked and ParseContext::ParseMessage is
UNDEFINED in the executable, resolving to libprotobuf.so.
- refuse_cloning_without_a_clip's "cannot misfire" paragraph had its reasoning
backwards. Routing picks VoiceCloning as the FALLBACK when there is no clip,
which is the case being caught; chatterbox, which ships in the gallery,
advertises clon and no tts at all, so every voice-less request lands there.
- audio_units read "2.1 min at 96 kHz" for index 11289602, which is 1.96 min.
2.1 min is 96 kHz's OWN first failure at 12288002. Both were remeasured and
the note is now a per-rate table.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
140 lines
6.6 KiB
C++
140 lines
6.6 KiB
C++
#include "audio_units.h"
|
|
|
|
#include <algorithm>
|
|
#include <cmath>
|
|
#include <limits>
|
|
|
|
namespace audiocpp_backend {
|
|
|
|
std::int64_t interleaved_frame_count(std::size_t sample_count, int channels) {
|
|
const std::size_t lanes = channels > 0 ? static_cast<std::size_t>(channels)
|
|
: static_cast<std::size_t>(1);
|
|
// Truncating division is deliberate: a trailing partial frame is not a
|
|
// position every channel reached, so counting it would overstate the length.
|
|
return static_cast<std::int64_t>(sample_count / lanes);
|
|
}
|
|
|
|
std::int64_t samples_to_nanoseconds(std::int64_t samples, int sample_rate) {
|
|
if (sample_rate <= 0) {
|
|
return 0;
|
|
}
|
|
// Split into whole seconds plus a remainder so the intermediate product
|
|
// cannot overflow on long recordings, and so rates like 44100 stay exact.
|
|
// The remainder division truncates deliberately: that matches Go's
|
|
// time.Duration conventions and keeps successive sample indices monotonic.
|
|
const std::int64_t rate = static_cast<std::int64_t>(sample_rate);
|
|
const std::int64_t whole_seconds = samples / rate;
|
|
const std::int64_t remainder = samples % rate;
|
|
return whole_seconds * 1000000000LL + (remainder * 1000000000LL) / rate;
|
|
}
|
|
|
|
float samples_to_seconds(std::int64_t samples, int sample_rate) {
|
|
if (sample_rate <= 0) {
|
|
return 0.0f;
|
|
}
|
|
return static_cast<float>(static_cast<double>(samples) /
|
|
static_cast<double>(sample_rate));
|
|
}
|
|
|
|
std::int64_t seconds_to_samples(double seconds, int sample_rate) {
|
|
// !(seconds > 0.0) rather than seconds <= 0.0: every comparison against NaN
|
|
// is false, so the <= form lets NaN reach the cast below, which is undefined
|
|
// behaviour and lands on INT64_MIN in practice. This is the one entry point
|
|
// fed by untrusted-shaped input (a float-seconds timestamp off the wire, or
|
|
// a boundary from a model that diverged), and a hugely negative sample index
|
|
// used later as an offset or a length is a wild pointer rather than merely a
|
|
// wrong timestamp.
|
|
if (sample_rate <= 0 || !(seconds > 0.0)) {
|
|
return 0;
|
|
}
|
|
const double scaled = seconds * static_cast<double>(sample_rate);
|
|
// Bound before the cast for the same reason: converting a double at or above
|
|
// 2^63 (infinity included) is undefined behaviour, so saturate instead.
|
|
const double limit =
|
|
static_cast<double>(std::numeric_limits<std::int64_t>::max());
|
|
if (scaled >= limit) {
|
|
return std::numeric_limits<std::int64_t>::max();
|
|
}
|
|
// Round rather than truncate: these functions exist to cross the float
|
|
// seconds boundary the VAD and diarize messages use, so a value that came
|
|
// from samples_to_seconds converts back to the sample it started as.
|
|
// Truncation lost one sample about half the time, starting at n=1.
|
|
//
|
|
// That round trip is exact only below roughly 2^23 samples. Past that the
|
|
// float samples_to_seconds returns can no longer resolve adjacent indices
|
|
// and the trip fails whatever the rounding. Both the first failing INDEX
|
|
// and the duration it stands for depend on the rate, so they are listed per
|
|
// rate rather than folded into one range; measured:
|
|
//
|
|
// 16 kHz 16384001 samples 17.1 min
|
|
// 44.1 kHz 11289602 samples 4.3 min
|
|
// 48 kHz 12288002 samples 4.3 min
|
|
// 96 kHz 12288002 samples 2.1 min
|
|
//
|
|
// The shortest recording this bites is therefore a couple of minutes of
|
|
// 96 kHz audio. It is a property of the float seconds API itself, not of
|
|
// the rounding here, and it is why nothing should use these to carry a
|
|
// sample-accurate position in a long recording.
|
|
return static_cast<std::int64_t>(std::llround(scaled));
|
|
}
|
|
|
|
std::vector<float> s16le_to_f32(const std::string &bytes) {
|
|
std::vector<float> samples;
|
|
const size_t count = bytes.size() / 2;
|
|
samples.reserve(count);
|
|
for (size_t i = 0; i < count; ++i) {
|
|
const auto low = static_cast<unsigned char>(bytes[i * 2]);
|
|
const auto high = static_cast<unsigned char>(bytes[i * 2 + 1]);
|
|
const auto raw = static_cast<std::int16_t>(
|
|
static_cast<std::uint16_t>(low) |
|
|
(static_cast<std::uint16_t>(high) << 8));
|
|
// 32768 on decode against 32767 on encode is deliberate, not a typo.
|
|
// 32768 is what keeps INT16_MIN at exactly -1.0 and every other code
|
|
// inside the [-1, 1] range this header promises; dividing by 32767
|
|
// would decode INT16_MIN to -1.00003. See f32_to_s16le for the other
|
|
// half of the pair. The cost is that a round trip shrinks a sample by
|
|
// 32767/32768, well under one LSB.
|
|
samples.push_back(static_cast<float>(raw) / 32768.0f);
|
|
}
|
|
return samples;
|
|
}
|
|
|
|
std::string f32_to_s16le(const std::vector<float> &samples) {
|
|
std::string bytes;
|
|
bytes.reserve(samples.size() * 2);
|
|
for (const float sample : samples) {
|
|
// NaN maps to silence. A NaN sample rendered as a full-scale click is
|
|
// worse audio than a dropped one, and this unit converts audio that may
|
|
// have originated off the wire.
|
|
//
|
|
// This guard also removes what used to be a spelling hazard in the
|
|
// clamp below. std::min and std::max return their first argument when
|
|
// the comparison is false, and every comparison against NaN is false,
|
|
// so before this branch existed the choice of spelling silently decided
|
|
// whether a NaN reached std::lround, whose result is unspecified for
|
|
// NaN. These three leaked it, the last being the idiomatic C++17 way to
|
|
// write a clamp and so the likeliest future edit:
|
|
// std::min(std::max(sample, -1.0f), 1.0f)
|
|
// std::max(std::min(sample, 1.0f), -1.0f)
|
|
// std::clamp(sample, -1.0f, 1.0f)
|
|
// The order is no longer load-bearing now that the guard runs first,
|
|
// but the history is why the guard is here, so do not drop it.
|
|
if (std::isnan(sample)) {
|
|
bytes.push_back(0);
|
|
bytes.push_back(0);
|
|
continue;
|
|
}
|
|
const float clamped = std::max(-1.0f, std::min(1.0f, sample));
|
|
// 32767 rather than 32768 so +1.0 saturates at INT16_MAX instead of
|
|
// overflowing to INT16_MIN. See s16le_to_f32 for why decode differs.
|
|
const auto value =
|
|
static_cast<std::int16_t>(std::lround(clamped * 32767.0f));
|
|
const auto raw = static_cast<std::uint16_t>(value);
|
|
bytes.push_back(static_cast<char>(raw & 0xFF));
|
|
bytes.push_back(static_cast<char>((raw >> 8) & 0xFF));
|
|
}
|
|
return bytes;
|
|
}
|
|
|
|
} // namespace audiocpp_backend
|