57e2e0bef0
Registers audio-cpp as preference-only in /backends/known: the family lives in GGUF metadata that an importer cannot read from a remote repo, and one repo hosts thirty families, so there is no honest auto-detect signal. Modality is a single string and the import form chips on a fixed key set, so it registers as tts with the other modalities named in the description rather than under an invented key the UI would bucket as "other". Adds a features page covering the option namespacing, the routing table per endpoint, the RPCs this backend declines and why, the bundled VAD path, the separation stem behaviour, and the family gotchas (supertonic needs the orig package; chatterbox advertises cloning and no plain tts; nemotron_asr defers its whole decode to finalize so live transcription emits nothing until the client half-closes, unlike higgs_audio_stt and voxtral_realtime). Every option name and family capability in it was read off the pinned upstream checkout. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
3.0 KiB
+++ disableToc = false title = "Voice Activity Detection (VAD)" weight = 35 url = "/features/voice-activity-detection/" +++
Voice Activity Detection (VAD) identifies segments of speech in audio data. LocalAI provides a /v1/vad endpoint powered by the Silero VAD backend.
The [audio.cpp backend]({{%relref "features/audio-cpp" %}}) also serves this endpoint, and ships the silero_vad and marblenet_vad assets inside its own package, so VAD works there with nothing to download (model: bundled:silero_vad plus the family:silero_vad option).
API
- Method:
POST - Endpoints:
/v1/vad,/vad
Request
The request body is JSON with the following fields:
| Parameter | Type | Required | Description |
|---|---|---|---|
model |
string |
Yes | Model name (e.g. silero-vad) |
audio |
float32[] |
Yes | Array of audio samples (16kHz PCM float) |
Response
Returns a JSON object with detected speech segments:
| Field | Type | Description |
|---|---|---|
segments |
array |
List of detected speech segments |
segments[].start |
float |
Start time in seconds |
segments[].end |
float |
End time in seconds |
Usage
Example request
The /v1/vad endpoint expects the audio field to be an array of raw
16kHz mono PCM samples as float32 values, so the request body is usually
built from a real audio file rather than typed by hand.
First convert any audio file to 16kHz mono with ffmpeg:
ffmpeg -i input.mp3 -ar 16000 -ac 1 -f wav speech.wav
Then load the samples and POST them (this snippet needs
pip install soundfile numpy requests):
import soundfile as sf
import numpy as np
import requests
audio, sample_rate = sf.read("speech.wav")
if audio.ndim > 1:
audio = audio.mean(axis=1) # downmix to mono
samples = audio.astype(np.float32).tolist()
response = requests.post(
"http://localhost:8080/v1/vad",
json={"model": "silero-vad", "audio": samples},
)
print(response.json())
Example response
{
"segments": [
{
"start": 0.5,
"end": 2.3
},
{
"start": 3.1,
"end": 5.8
}
]
}
Model Configuration
Create a YAML configuration file for the VAD model:
name: silero-vad
backend: silero-vad
Detection Parameters
The Silero VAD backend uses the following internal defaults:
- Sample rate: 16kHz
- Threshold: 0.5
- Min silence duration: 100ms
- Speech pad duration: 30ms
Error Responses
| Status Code | Description |
|---|---|
| 400 | Missing or invalid model or audio field |
| 500 | Backend error during VAD processing |