mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
docs(audio-cpp): document params[text] and the pinned transform tasks
The text channel and the two task pins are both invisible from the endpoint contract alone: nothing in the AudioTransform form tells a reader that a speech-to-speech model needs the line it is resynthesising, and nothing says that asking for singing voice conversion without task:svc silently gets plain voice conversion instead. Both are the kind of thing a user only discovers from a refusal or, worse, from output that looks right and is not. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code]
This commit is contained in:
committed by
localai-org-maint-bot
parent
ed14796a15
commit
f72e409457
@@ -180,6 +180,36 @@ the body, is listed in the `X-Audio-Stems` response header. See
|
||||
`params[stem]` against a model that produces a single unnamed output is refused rather than
|
||||
ignored.
|
||||
|
||||
## Text-conditioned transforms
|
||||
|
||||
`s2s` is a text-and-prosody route, not an audio-only one: `vevo2` resynthesises a line in
|
||||
another voice and reads the line itself, refusing the run without it. `AudioTransform` has
|
||||
no text field, so the text arrives as `params[text]` (or `params[target_text]`, the name
|
||||
vevo2's own option table uses) and the backend lifts it into the engine's text input.
|
||||
`params[language]` rides along when the line is not English.
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/audio/transformations \
|
||||
-F model=audio-cpp-vevo2-speech-to-speech \
|
||||
-F audio=@source.wav -F reference=@target-speaker.wav \
|
||||
-F 'params[text]=The quick brown fox jumps over the lazy dog.' -o converted.wav
|
||||
```
|
||||
|
||||
A model that takes no text ignores the keys, so separation and plain voice conversion are
|
||||
unaffected.
|
||||
|
||||
## Pinned tasks
|
||||
|
||||
Two routes are unreachable by auto-routing and need `task:` in the model config, because
|
||||
nothing in a request distinguishes them from a task the same family also advertises:
|
||||
|
||||
- `task:svc` for singing voice conversion. `seed_vc` and `vevo2` advertise both `svc` and
|
||||
ordinary `vc`, and no request signal means "this input is singing", so `vc` always wins.
|
||||
- `task:s2s` for speech to speech, for the same reason against the same `vc`.
|
||||
|
||||
The gallery entries that ship these routes carry the pin already. Removing it gives plain
|
||||
voice conversion from the same weights.
|
||||
|
||||
## Family notes
|
||||
|
||||
- **Supertonic**: use the `orig` GGUF package, whose weights are f32. The f16 package was
|
||||
|
||||
@@ -56,6 +56,7 @@ form-data, returns audio bytes.
|
||||
| `sample_rate` | int | no | Desired output sample rate in Hz. Omit it for the backend's own rate; otherwise it must be between 8000 and 192000, and a value outside that range is refused with a 400 |
|
||||
| `params[<key>]` | string | no | Repeated; forwarded to backend |
|
||||
| `params[stem]` | string | no | Multi-output transforms only; picks which named output the body carries (see [stems](#multi-output-transforms-source-separation-stems)) |
|
||||
| `params[text]` | string | no | Text-conditioned transforms only; the line being resynthesised (see [text-conditioned transforms](#text-conditioned-transforms-speech-to-speech)) |
|
||||
|
||||
First install an audio-transform model from the gallery (the examples below use `localvqe-v1.3-4.8m`):
|
||||
|
||||
@@ -143,6 +144,28 @@ The stems live in the generated-content directory beside the main output and are
|
||||
served from `/generated-audio/`. Like every other generated artifact, they are
|
||||
not swept automatically.
|
||||
|
||||
### Text-conditioned transforms (speech to speech)
|
||||
|
||||
Most transforms are audio in, audio out. A few are not: a speech-to-speech
|
||||
model resynthesises a line in another voice and needs to know what the line
|
||||
says. `/audio/transform` has no text field, so the text travels as
|
||||
`params[text]`.
|
||||
|
||||
```bash
|
||||
curl -sS -X POST http://localhost:8080/audio/transformations \
|
||||
-F model=audio-cpp-vevo2-speech-to-speech \
|
||||
-F audio=@source.wav \
|
||||
-F reference=@target-speaker.wav \
|
||||
-F 'params[text]=The quick brown fox jumps over the lazy dog.' \
|
||||
-o converted.wav
|
||||
```
|
||||
|
||||
For these models the text is required, not a hint: the run is refused without
|
||||
it. `params[target_text]` is accepted as the same field, and
|
||||
`params[language]` sets the language when the line is not English. A model that
|
||||
does not take text ignores all three, so an ordinary separation or
|
||||
noise-suppression request is unaffected.
|
||||
|
||||
## Streaming endpoint
|
||||
|
||||
`GET /audio/transformations/stream` - bidirectional WebSocket. The first
|
||||
|
||||
Reference in New Issue
Block a user