Ettore Di Giacinto 6170e03f1b backend(audio-cpp): serve the AudioTransform RPC ...

Covers voice conversion, singing voice conversion, speech to speech and source
separation, the four tasks LocalAI's AudioTransform can represent.

AudioTransformResult carries one dst while htdemucs and mel_band_roformer
produce several named stems from a single run, so inference runs ONCE, every
stem is written as a sibling file <dst-stem>.<name>.<ext>, and params[stem]
selects which one dst receives, defaulting to vocals and falling back to the
first output. An unknown stem name is INVALID_ARGUMENT listing the real stem
names rather than a silent substitution, and the selection happens before the
first write so a refused request leaves no files behind. params[stem] is
consumed here and is not forwarded into the engine's request options.

The stem decision lives in stem_selection, which is stdlib only and therefore
tested by backend/cpp/run-unit-tests.sh. It also validates the names, because
they come from the model (htdemucs reads them from the GGUF's config.sources)
and each becomes a component of a path this backend writes: a name carrying a
path separator would escape the caller's output directory, and two stems
sharing a name would silently overwrite one another.

Both files are read at their native rate and channel count. Separation forces
it, since demucs and roformer refuse any rate but 44.1 kHz and lose the stereo
image that separates a centred vocal from a wide mix. The conversion families
all resample internally (seed_vc, vevo2, miocodec, chatterbox were each
checked), so passing the file through unchanged is also strictly better than
band limiting it to 16 kHz first.

Verified end to end against htdemucs f16 on a 44.1 kHz stereo mix: four stems
plus dst, dst byte identical to the selected stem, params[stem] selecting a
different one, an unknown stem refused with no files written, and mono input
preserved as mono output. Also against miocodec for the single output path,
where params[stem] is refused rather than ignored.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

2026-07-29 19:03:33 +00:00

3.1 KiB

Raw Blame History

View Raw