mirror of
https://github.com/mudler/LocalAI.git
synced 2026-06-14 11:49:33 -04:00
feat(omnivoice-cpp): add OmniVoice TTS backend (file + streaming, voice cloning + voice design) (#10310)
* feat(omnivoice-cpp): add C wrapper + CMake/Makefile build over OmniVoice ov_* ABI Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(omnivoice-cpp): add option/language parsing + WAV framing helpers with tests Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(omnivoice-cpp): wire purego binding with TTS + streaming TTSStream Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(omnivoice-cpp): wire backend into root Makefile Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(omnivoice-cpp): add build matrix entries + dep-bump registration Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(omnivoice-cpp): register backend meta + image entries Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(omnivoice-cpp): expose as preference-only importable backend Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add omnivoice-cpp TTS models (Q8_0 default + BF16 HQ) Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(omnivoice-cpp): document the OmniVoice TTS backend Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(omnivoice-cpp): add env-gated e2e for TTS + streaming Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(omnivoice-cpp): honor tts.audio_path/tts.voice config as default cloning reference The model config tts.audio_path (ModelOptions.AudioPath) and tts.voice now provide a default voice-cloning reference used when a request omits Voice, so a cloned voice can be pinned in the model YAML instead of passed per request. A per-request voice still overrides. Paths resolve relative to the model dir. Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(omnivoice-cpp): add missing omnivoice-cpp-development backend meta Mirrors the whisper/vibevoice convention: a -development meta aggregating the master-tagged image variants (the production meta and per-variant prod+dev image entries already existed; only the development meta aggregator was missing). Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
LocalAI [bot]
@@ -226,6 +226,82 @@ curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
|
||||
}' | aplay
|
||||
```
|
||||
|
||||
### OmniVoice
|
||||
|
||||
[OmniVoice](https://github.com/ServeurpersoCom/omnivoice.cpp) (`omnivoice-cpp` backend) is a native C++ / GGML text-to-speech engine. It supports voice cloning (from reference audio plus its transcript), voice design (steering the voice with attribute keywords such as gender, age, pitch, style, volume, and emotion), and streaming synthesis. Output is 24kHz mono audio and it covers 646 languages.
|
||||
|
||||
#### Setup
|
||||
|
||||
Install the `omnivoice-cpp` model in the Model gallery or run `local-ai run models install omnivoice-cpp`. A higher-quality BF16 variant is available as `omnivoice-cpp-hq` (the default `omnivoice-cpp` ships Q8_0 GGUFs).
|
||||
|
||||
#### Usage
|
||||
|
||||
Use the speech endpoint by specifying the omnivoice-cpp backend:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -d '{
|
||||
"model": "omnivoice-cpp",
|
||||
"input": "Hello world, this is a test."
|
||||
}' | aplay
|
||||
```
|
||||
|
||||
#### Voice cloning
|
||||
|
||||
Pass a reference audio file via the `voice` parameter and its transcript via the `ref_text` generation parameter:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -d '{
|
||||
"model": "omnivoice-cpp",
|
||||
"input": "Hello world, this is a test.",
|
||||
"voice": "path/to/reference_audio.wav",
|
||||
"params": { "ref_text": "This is the transcript of the reference audio." }
|
||||
}' | aplay
|
||||
```
|
||||
|
||||
You can also pin a default cloned voice in the model config so callers do not have to pass it on every request. Both `tts.voice` and `tts.audio_path` are honored as the reference audio (a per-request `voice` overrides them); paths are resolved relative to the model directory:
|
||||
|
||||
```yaml
|
||||
name: omnivoice-cpp
|
||||
backend: omnivoice-cpp
|
||||
parameters:
|
||||
model: omnivoice-cpp/omnivoice-base-Q8_0.gguf
|
||||
tts:
|
||||
audio_path: "voices/my_reference.wav" # default cloning reference (or use tts.voice)
|
||||
options:
|
||||
- "tokenizer:omnivoice-cpp/omnivoice-tokenizer-Q8_0.gguf"
|
||||
```
|
||||
|
||||
#### Voice design
|
||||
|
||||
Steer the synthesized voice with attribute keywords (gender, age, pitch, style, volume, emotion) by passing an `instructions` string per request:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -d '{
|
||||
"model": "omnivoice-cpp",
|
||||
"input": "Hello world, this is a test.",
|
||||
"instructions": "female young high soft emotion:happy"
|
||||
}' | aplay
|
||||
```
|
||||
|
||||
#### Configuration
|
||||
|
||||
The backend loads the base GGUF from `parameters.model` and its tokenizer from the `tokenizer:` option. A few optional generation knobs are available as `options`:
|
||||
|
||||
```yaml
|
||||
name: omnivoice-cpp
|
||||
backend: omnivoice-cpp
|
||||
parameters:
|
||||
model: omnivoice-cpp/omnivoice-base-Q8_0.gguf
|
||||
options:
|
||||
- "tokenizer:omnivoice-cpp/omnivoice-tokenizer-Q8_0.gguf"
|
||||
- "use_fa:true" # enable flash attention
|
||||
- "clamp_fp16:true" # clamp activations for fp16 stability
|
||||
- "seed:42" # deterministic generation
|
||||
- "denoise:true" # denoise the generated audio
|
||||
```
|
||||
|
||||
A per-request `seed` can also be supplied through the `params` map alongside `ref_text`.
|
||||
|
||||
### Pocket TTS
|
||||
|
||||
[Pocket TTS](https://github.com/kyutai-labs/pocket-tts) is a lightweight text-to-speech model designed to run efficiently on CPUs. It supports voice cloning through HuggingFace voice URLs or local audio files.
|
||||
|
||||
@@ -57,6 +57,7 @@ LocalAI will attempt to automatically load models which are not explicitly confi
|
||||
| [VoxCPM](https://github.com/ModelBest/VoxCPM) | Expressive end-to-end TTS | CPU, CUDA 12/13, ROCm, Intel, Metal |
|
||||
| [Kitten TTS](https://github.com/KittenML/KittenTTS) | Kitten TTS model | CPU, Metal |
|
||||
| [MLX-Audio](https://github.com/Blaizzy/mlx-audio) | Audio models on Apple Silicon | Metal, CPU, CUDA 12/13, Jetson L4T |
|
||||
| [OmniVoice](https://github.com/ServeurpersoCom/omnivoice.cpp) | Native C++/GGML TTS with voice cloning, voice design, and streaming | CPU, CUDA 12/13, ROCm, Intel, Metal, Vulkan, Jetson L4T |
|
||||
|
||||
## Music Generation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user