mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-17 08:50:57 -04:00
localai-org-maint-botandEttore Di Giacinto
fb8b7a359a
* fix(distributed): stage sound detection audio Sound detection passes frontend temporary paths directly to remote workers, unlike transcription. Stage the WAV before classification so CED can read it without a shared temporary directory. Preserve the original request for retries and propagate staging errors without calling the backend. Cover staging, request preservation, and error handling with regression tests. Assisted-by: Codex:GPT-6 golangci-lint * test(distributed): verify routed sound staging Call sound detection through the client returned by SmartRouter.Route. This checks interface dispatch through both routing wrappers, rather than constructing FileStagingClient directly. The test fails without the sound-staging override and passes with it. Assisted-by: Codex:GPT-6 golangci-lint --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
66 lines
2.3 KiB
Markdown
66 lines
2.3 KiB
Markdown
+++
|
|
disableToc = false
|
|
title = "Sound Classification"
|
|
weight = 32
|
|
url = "/features/audio-classification/"
|
|
+++
|
|
|
|
Sound-event classification (audio tagging) answers the question **"what am I hearing?"** - given an audio clip, it returns a list of scored [AudioSet](https://research.google.com/audioset/) labels (e.g. *Baby cry, infant cry*, *Glass breaking*, *Dog bark*, *Alarm*).
|
|
|
|
LocalAI exposes this through the `/v1/audio/classification` endpoint, modelled after `/v1/audio/transcriptions`. The reference backend is **[ced.cpp](https://github.com/localai-org/ced.cpp)** (CED, a 527-class AudioSet tagger), a small ViT over a log-mel spectrogram ported to ggml with full PyTorch parity. Apache-2.0 weights are redistributable as GGUF.
|
|
|
|
Because classification is exposed as a regular OpenAI-style endpoint, any HTTP client works - there is no Python dependency on the consumer side.
|
|
|
|
In distributed mode, LocalAI stages uploaded audio and realtime sound-detection
|
|
windows on the selected worker before classification. The API server and worker
|
|
do not need a shared temporary directory.
|
|
|
|
## Endpoint
|
|
|
|
```
|
|
POST /v1/audio/classification
|
|
Content-Type: multipart/form-data
|
|
```
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `file` | file (required) | audio file in any format `ffmpeg` accepts |
|
|
| `model` | string (required) | name of the sound-classification-capable model (e.g. `ced-base-f16`) |
|
|
| `top_k` | int | number of top tags to return (0 = backend default) |
|
|
| `threshold` | float | drop tags scoring below this value |
|
|
|
|
### Response
|
|
|
|
```json
|
|
{
|
|
"model": "ced-base-f16",
|
|
"detections": [
|
|
{"index": 23, "label": "Baby cry, infant cry", "score": 0.87},
|
|
{"index": 22, "label": "Crying, sobbing", "score": 0.41}
|
|
]
|
|
}
|
|
```
|
|
|
|
Detections are returned in score-descending order. Scores are per-class probabilities (multi-label, independent), so they do not sum to 1.
|
|
|
|
## Example
|
|
|
|
First install a classification model from the gallery (the example below uses `ced-base-f16`):
|
|
|
|
```bash
|
|
local-ai run ced-base-f16
|
|
```
|
|
|
|
```bash
|
|
curl http://localhost:8080/v1/audio/classification \
|
|
-H "Content-Type: multipart/form-data" \
|
|
-F file="@/path/to/clip.wav" \
|
|
-F model="ced-base-f16" \
|
|
-F top_k=10
|
|
```
|
|
|
|
## See also
|
|
|
|
- [Audio to Text]({{% relref "audio-to-text" %}}) - speech transcription
|
|
- [Speaker Diarization]({{% relref "audio-diarization" %}}) - who spoke when
|