localai-org-maint-botandEttore Di Giacinto
6b794651a4
* feat(diarization): return sound events with include_sounds
A client that wants text, speakers, voice prints and sound events had to
make a diarization call and a separate sound call. Add an include_sounds
request field to /v1/audio/diarization that adds a sounds array of closed
events {start, end, label, confidence}, in seconds.
The parakeet-cpp backend runs a tagger-only scene stream over the clip,
the same stream and thresholds the live path uses, so a clip gives the
same events offline and live. A model with no sound_model companion, or a
backend that does not report sound events, fails with 501 and the stable
code include_sounds_unsupported instead of an empty list. The proto
carries sounds_included so an empty list still means "nothing heard".
The localai-proxy backend forwards the field. Swagger, docs and the
e2e mock backend are updated.
Assisted-by: Claude:claude-sonnet-5-5 [protoc swag go]
* feat(gallery): add parakeet-cpp-multilingual-diarization-speakers-sounds
Same as parakeet-cpp-multilingual-diarization-speakers (TDT 0.6B v3,
Nemotron-3-Diarization, WeSpeaker) plus a CED-Tiny sound_model, so one
model name serves /v1/audio/diarization with include_text,
include_speaker_profiles and include_sounds. It declares the
sound_classification usecase like the realtime scene entries.
Assisted-by: Claude:claude-sonnet-5-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
4.1 KiB
+++ disableToc = false title = "Sound Classification" weight = 32 url = "/features/audio-classification/" +++
Sound-event classification (audio tagging) answers the question "what am I hearing?" - given an audio clip, it returns a list of scored AudioSet labels (e.g. Baby cry, infant cry, Glass breaking, Dog bark, Alarm).
LocalAI exposes this through the /v1/audio/classification endpoint, modelled after /v1/audio/transcriptions. The reference backend is ced.cpp (CED, a 527-class AudioSet tagger), a small ViT over a log-mel spectrogram ported to ggml with full PyTorch parity. Apache-2.0 weights are redistributable as GGUF.
parakeet.cpp can also load a CED model (through third_party/ced.cpp) and serve /v1/audio/classification from the same backend used for ASR and diarization. It scores the clip in 10 s windows and averages each class's score across the windows before sorting and applying top_k/threshold - CED's own method for clips longer than one window. Install parakeet-cpp-ced-tiny or parakeet-cpp-ced-base from the gallery, or point parameters.model at a CED GGUF under backend: parakeet-cpp. A parakeet-cpp ASR model can also point sound_model at a CED GGUF to add live sound events during realtime transcription - see [Realtime API]({{% relref "openai-realtime" %}}) - and to add timed sound events to an offline diarization call with include_sounds=true - see [Speaker Diarization]({{% relref "audio-diarization" %}}#sound-events). Such a model also answers /v1/audio/classification itself when it declares the sound_classification usecase, as the gallery entry parakeet-cpp-multilingual-diarization-speakers-sounds does.
Because classification is exposed as a regular OpenAI-style endpoint, any HTTP client works - there is no Python dependency on the consumer side.
In distributed mode, LocalAI stages uploaded audio and realtime sound-detection windows on the selected worker before classification. The API server and worker do not need a shared temporary directory.
Endpoint
POST /v1/audio/classification
Content-Type: multipart/form-data
| Field | Type | Description |
|---|---|---|
file |
file (required) | audio file in any format ffmpeg accepts |
model |
string (required) | name of the sound-classification-capable model (e.g. ced-base-f16) |
top_k |
int | number of top tags to return (0 = backend default) |
threshold |
float | drop tags scoring below this value |
Response
{
"model": "ced-base-f16",
"detections": [
{"index": 23, "label": "Baby cry, infant cry", "score": 0.87},
{"index": 22, "label": "Crying, sobbing", "score": 0.41}
]
}
Detections are returned in score-descending order. Scores are per-class probabilities (multi-label, independent), so they do not sum to 1.
Example
First install a classification model from the gallery (the example below uses ced-base-f16):
local-ai run ced-base-f16
curl http://localhost:8080/v1/audio/classification \
-H "Content-Type: multipart/form-data" \
-F file="@/path/to/clip.wav" \
-F model="ced-base-f16" \
-F top_k=10
The same request works unchanged against a parakeet-cpp CED model:
name: parakeet-ced-tiny
backend: parakeet-cpp
parameters:
model: ced-tiny-q8_0.gguf
known_usecases:
- sound_classification
curl http://localhost:8080/v1/audio/classification \
-H "Content-Type: multipart/form-data" \
-F file="@/path/to/clip.wav" \
-F model="parakeet-ced-tiny" \
-F top_k=10
The bundle entries parakeet-cpp-bundle-small and parakeet-cpp-bundle-standard also serve this endpoint: they hold CED-Small next to the transcription, VAD and diarization models (sound_component:ced). See [Bundle GGUF files]({{% relref "audio-to-text" %}}#bundle-gguf-files-several-models-in-one-file).
See also
- [Audio to Text]({{% relref "audio-to-text" %}}) - speech transcription
- [Speaker Diarization]({{% relref "audio-diarization" %}}) - who spoke when