mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-10 15:52:29 -04:00
mudler-agentandEttore Di Giacinto
3185b6fcf8
* feat(parakeet-cpp): load bundle GGUF files and use their components by role A bundle GGUF holds several models (ASR, VAD, diarization, sound events, speaker encoder) in one file, each with its own licence. Detect a bundle at load through parakeet_capi_bundle_components_json and open components with parakeet_capi_load_component. The three symbols are probed together, so an older libparakeet.so still loads plain files as before. The only ASR component is the primary model; bundle_asr:<name> picks one when there are several. A Silero VAD component of the primary bundle is loaded without an option and serves /v1/vad and vad:true. The diar, ced and voice components load on request: diar_component, sound_component and speaker_component, or a companion option (diarization_model, sound_model, speaker_model, vad_model) that names a bundle, even the model file itself. vad_component picks a VAD component and implies vad:true. A role the bundle cannot fill fails the load with the component list, and a diarization or sound request on a model without that role names the bundle components. Every existing option and single-file model behaves as before. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * chore(parakeet-cpp): bump parakeet.cpp to 781a973 Brings in the bundle GGUF format and its C-API (parakeet_capi_load_component, parakeet_capi_bundle_components_json, parakeet_capi_load_error). Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] * feat(parakeet-cpp): gallery entries for the bundle GGUF files, docs Add parakeet-cpp-bundle-small (338 MB: Parakeet TDT+CTC 110M, Nemotron-3- Diarization, CED-Small, WeSpeaker ResNet34-LM, Silero VAD), -standard (1.1 GB, Parakeet TDT 0.6B v3 instead of the 110M model) and -moondream-redux (215 MB: packed Redux and Silero VAD, CPU only). One install serves transcription, VAD, diarization, sound events and speaker naming through the component options. The existing single-purpose entries stay. A bundle has no single licence, so the entries use license: other and state the licence and credit of each component in the description, with the upstream inconsistency of the CED licence. The docs get a section on bundles in audio-to-text with the entries, the roles, the options and the licence notice, and pointers from the VAD, diarization and sound classification pages. A gallery test checks the file names, checksums, usecases and options. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
89 lines
3.7 KiB
Markdown
89 lines
3.7 KiB
Markdown
+++
|
|
disableToc = false
|
|
title = "Sound Classification"
|
|
weight = 32
|
|
url = "/features/audio-classification/"
|
|
+++
|
|
|
|
Sound-event classification (audio tagging) answers the question **"what am I hearing?"** - given an audio clip, it returns a list of scored [AudioSet](https://research.google.com/audioset/) labels (e.g. *Baby cry, infant cry*, *Glass breaking*, *Dog bark*, *Alarm*).
|
|
|
|
LocalAI exposes this through the `/v1/audio/classification` endpoint, modelled after `/v1/audio/transcriptions`. The reference backend is **[ced.cpp](https://github.com/localai-org/ced.cpp)** (CED, a 527-class AudioSet tagger), a small ViT over a log-mel spectrogram ported to ggml with full PyTorch parity. Apache-2.0 weights are redistributable as GGUF.
|
|
|
|
**[parakeet.cpp](https://github.com/mudler/parakeet.cpp)** can also load a CED model (through `third_party/ced.cpp`) and serve `/v1/audio/classification` from the same backend used for ASR and diarization. It scores the clip in 10 s windows and averages each class's score across the windows before sorting and applying `top_k`/`threshold` - CED's own method for clips longer than one window. Install `parakeet-cpp-ced-tiny` or `parakeet-cpp-ced-base` from the gallery, or point `parameters.model` at a CED GGUF under `backend: parakeet-cpp`. A parakeet-cpp ASR model can also point `sound_model` at a CED GGUF to add live sound events during realtime transcription - see [Realtime API]({{% relref "openai-realtime" %}}).
|
|
|
|
Because classification is exposed as a regular OpenAI-style endpoint, any HTTP client works - there is no Python dependency on the consumer side.
|
|
|
|
In distributed mode, LocalAI stages uploaded audio and realtime sound-detection
|
|
windows on the selected worker before classification. The API server and worker
|
|
do not need a shared temporary directory.
|
|
|
|
## Endpoint
|
|
|
|
```
|
|
POST /v1/audio/classification
|
|
Content-Type: multipart/form-data
|
|
```
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `file` | file (required) | audio file in any format `ffmpeg` accepts |
|
|
| `model` | string (required) | name of the sound-classification-capable model (e.g. `ced-base-f16`) |
|
|
| `top_k` | int | number of top tags to return (0 = backend default) |
|
|
| `threshold` | float | drop tags scoring below this value |
|
|
|
|
### Response
|
|
|
|
```json
|
|
{
|
|
"model": "ced-base-f16",
|
|
"detections": [
|
|
{"index": 23, "label": "Baby cry, infant cry", "score": 0.87},
|
|
{"index": 22, "label": "Crying, sobbing", "score": 0.41}
|
|
]
|
|
}
|
|
```
|
|
|
|
Detections are returned in score-descending order. Scores are per-class probabilities (multi-label, independent), so they do not sum to 1.
|
|
|
|
## Example
|
|
|
|
First install a classification model from the gallery (the example below uses `ced-base-f16`):
|
|
|
|
```bash
|
|
local-ai run ced-base-f16
|
|
```
|
|
|
|
```bash
|
|
curl http://localhost:8080/v1/audio/classification \
|
|
-H "Content-Type: multipart/form-data" \
|
|
-F file="@/path/to/clip.wav" \
|
|
-F model="ced-base-f16" \
|
|
-F top_k=10
|
|
```
|
|
|
|
The same request works unchanged against a parakeet-cpp CED model:
|
|
|
|
```yaml
|
|
name: parakeet-ced-tiny
|
|
backend: parakeet-cpp
|
|
parameters:
|
|
model: ced-tiny-q8_0.gguf
|
|
known_usecases:
|
|
- sound_classification
|
|
```
|
|
|
|
```bash
|
|
curl http://localhost:8080/v1/audio/classification \
|
|
-H "Content-Type: multipart/form-data" \
|
|
-F file="@/path/to/clip.wav" \
|
|
-F model="parakeet-ced-tiny" \
|
|
-F top_k=10
|
|
```
|
|
|
|
The bundle entries `parakeet-cpp-bundle-small` and `parakeet-cpp-bundle-standard` also serve this endpoint: they hold CED-Small next to the transcription, VAD and diarization models (`sound_component:ced`). See [Bundle GGUF files]({{% relref "audio-to-text" %}}#bundle-gguf-files-several-models-in-one-file).
|
|
|
|
## See also
|
|
|
|
- [Audio to Text]({{% relref "audio-to-text" %}}) - speech transcription
|
|
- [Speaker Diarization]({{% relref "audio-diarization" %}}) - who spoke when
|