1e0baec2a7
The "Bump Backend dependencies" workflow has failed every night for over ten days. Four upstreams — ced.cpp, moss-transcribe.cpp, voice-detect.cpp and rf-detr.cpp — moved from the mudler org to localai-org, so the GitHub API answers 301 for the old slugs. ced.cpp additionally renamed its default branch to main. bump_deps.sh fetched without -L or -f and never checked the response, so the redirect's JSON body was passed straight to sed, which died with "unterminated `s' command". The loud failure was luck: an error body without slashes would have been substituted into the Makefile as the new pin, silently corrupting the version and shipping it in a bump PR. Point the matrix at the new slugs and branch, and harden the script so a bad response can never reach sed: follow redirects, fail on HTTP errors, and require a bare 40-hex SHA before rewriting anything. Also refresh the now-stale repository URLs in the backend Makefiles, test scripts, backend/index.yaml and the docs. Verified all 25 matrix entries resolve to a commit SHA and that the four previously-failing jobs run end to end against the real API. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
2.1 KiB
+++ disableToc = false title = "Sound Classification" weight = 32 url = "/features/audio-classification/" +++
Sound-event classification (audio tagging) answers the question "what am I hearing?" - given an audio clip, it returns a list of scored AudioSet labels (e.g. Baby cry, infant cry, Glass breaking, Dog bark, Alarm).
LocalAI exposes this through the /v1/audio/classification endpoint, modelled after /v1/audio/transcriptions. The reference backend is ced.cpp (CED, a 527-class AudioSet tagger), a small ViT over a log-mel spectrogram ported to ggml with full PyTorch parity. Apache-2.0 weights are redistributable as GGUF.
Because classification is exposed as a regular OpenAI-style endpoint, any HTTP client works - there is no Python dependency on the consumer side.
Endpoint
POST /v1/audio/classification
Content-Type: multipart/form-data
| Field | Type | Description |
|---|---|---|
file |
file (required) | audio file in any format ffmpeg accepts |
model |
string (required) | name of the sound-classification-capable model (e.g. ced-base-f16) |
top_k |
int | number of top tags to return (0 = backend default) |
threshold |
float | drop tags scoring below this value |
Response
{
"model": "ced-base-f16",
"detections": [
{"index": 23, "label": "Baby cry, infant cry", "score": 0.87},
{"index": 22, "label": "Crying, sobbing", "score": 0.41}
]
}
Detections are returned in score-descending order. Scores are per-class probabilities (multi-label, independent), so they do not sum to 1.
Example
First install a classification model from the gallery (the example below uses ced-base-f16):
local-ai run ced-base-f16
curl http://localhost:8080/v1/audio/classification \
-H "Content-Type: multipart/form-data" \
-F file="@/path/to/clip.wav" \
-F model="ced-base-f16" \
-F top_k=10
See also
- [Audio to Text]({{% relref "audio-to-text" %}}) - speech transcription
- [Speaker Diarization]({{% relref "audio-diarization" %}}) - who spoke when