LocalAI/docs/content/features/audio-cpp.md

+++ disableToc = false title = "audio.cpp backend" weight = 38 url = "/features/audio-cpp/" +++

audio.cpp is a GGML audio inference framework covering roughly thirty model families in one engine. LocalAI exposes it through the native C++ audio-cpp backend, so a single installed backend can serve text to speech, voice cloning, transcription, forced alignment, voice activity detection, speaker diarization, source separation, voice conversion and music generation, depending only on which model you point it at.

audio.cpp is developed by ShugoAI LLC, not by the LocalAI project. Model packages live in huggingface.co/audio-cpp/audio.cpp-gguf.

Installing

local-ai backends install audio-cpp

Or install it from the Backends page in the web UI. audio-cpp is a preference-only backend: LocalAI never picks it automatically during model import, because the only reliable signal for an audio.cpp model is the audiocpp.model_spec.family key stored inside the GGUF, which cannot be read from a remote repository, and the upstream GGUF repository hosts every family in one place. Set backend: audio-cpp in the model YAML, or select it explicitly in the import form.

What it serves

One model serves one family, and a family advertises the tasks it can perform. The backend routes each LocalAI endpoint onto an audio.cpp task and refuses, with the family's real capability list in the message, when the loaded model cannot serve it.

LocalAI endpoint	gRPC RPC	audio.cpp tasks tried
/v1/audio/speech, /tts	TTS, TTSStream	tts, clon (voice cloning), vdes (voice design)
/v1/audio/transcriptions	AudioTranscription, AudioTranscriptionStream	asr, align (forced alignment, when a prompt is supplied)
/v1/realtime live transcription	AudioTranscriptionLive	asr in streaming mode
/v1/vad	VAD	vad
/v1/audio/diarization	Diarize	diar
/v1/sound-generation	SoundGeneration	gen
/audio/transform, /audio/transformations	AudioTransform	sep (separation), vc (voice conversion), svc, s2s

A supplied speaker clip routes TTS to clon; free-form instructions route it to vdes. Singing voice conversion (svc) is never chosen automatically, because no request signal means "this input is singing"; pin it with task:svc.

Not served, and why

These RPCs return UNIMPLEMENTED with an explanation rather than a degraded answer:

• AudioEncode / AudioDecode: audio.cpp has no codec task kind, so no family can be asked to turn PCM into codec frames or back.
• AudioTransformStream: no family advertises streaming for any task AudioTransform routes to (sep, vc, svc, s2s). Upstream advertises streaming for tts, asr and vad only, so a streaming transform here would be a buffered offline call in disguise.
• AudioToAudioStream: LocalAI's contract is an OpenAI-Realtime shaped audio conversation with a system prompt and a tool loop. audio.cpp's s2s is offline clip-to-clip processing against a target voice.
• VoiceEmbed: no family advertises the spk (speaker recognition) task, so nothing in the engine can produce a speaker embedding. Use the [voice recognition]({{%relref "features/voice-recognition" %}}) backends instead.

Model YAML

None of the option names are guessable, so here is the full shape:

name: supertonic
backend: audio-cpp
parameters:
  model: supertonic-3-orig.gguf
known_usecases:
  - FLAG_TTS
options:
  # Optional for an audio.cpp GGUF: the family is read from the file's
  # audiocpp.model_spec.family metadata. REQUIRED for a safetensors file or a
  # package directory, which carry no such key.
  - family:supertonic
  # Optional: pins the audio.cpp task instead of routing it from the RPC.
  - task:tts
  # ggml compute backend inside audio.cpp (not the LocalAI backend above).
  - backend:cuda
  - device:0
  - threads:8
  # Give up on a queued request instead of piling up behind a wedged run.
  - busy_timeout_ms:120000
  # Per-family knobs, passed through to the upstream loader and session.
  - load.vibevoice.lora:/models/my-adapter
  - session.seed_vc.weight_type:q8_0

Every entry is key:value, split on the first colon so values may contain more (paths, for instance). An unknown key fails the load with INVALID_ARGUMENT and a message listing the keys that exist, rather than being silently ignored.

Option reference

Option	Default	Meaning
family:<name>	read from the GGUF	The audio.cpp family. Optional for a standalone audio.cpp GGUF, which embeds audiocpp.model_spec.family. Required for a safetensors file or a package directory. Setting it explicitly also overrides the embedded value.
task:<name>	routed from the RPC	Pins the task. One of gen, tts, clon, vc, svc, s2s, asr, align, vad, diar, sep, vdes, spk. A pinned task is honoured exactly: if the family cannot serve it, the request is refused rather than rerouted.
backend:<name>	cpu	ggml compute backend: cpu, cuda, vulkan, metal, best. Must match the backend image you installed (a cuda value needs the CUDA image).
device:<n>	0	GPU index for the selected compute backend. Non-negative integer.
threads:<n>	runtime default	CPU threads. 0 leaves the choice to the runtime.
busy_timeout_ms:<n>	0 (unbounded)	Bounds the wait for the model's single inference lane. A request that arrives while a run has already been in flight longer than this fails immediately with UNAVAILABLE instead of queueing, and a request that waits this long without the lane freeing up fails the same way. 0 waits indefinitely.
live_idle_timeout_ms:<n>	30000	How long AudioTranscriptionLive waits for the next audio frame before cancelling the stream and releasing the lane. That RPC holds the lane for the whole stream, so a peer that stops sending without closing would otherwise block every other request against this model. 0 disables the limit.
model_spec_override:<path>	unset	A model spec JSON file, or a directory containing <family>.json, used instead of the spec embedded in the GGUF. The path must exist at load time.
load.<key>:<value>	none	Forwarded to the upstream loader with the load. prefix stripped.
session.<key>:<value>	none	Forwarded to the upstream session with the session. prefix stripped.

Numeric options are validated at load time: a non-integer or negative value is refused before the model is read, not after.

The load. and session. namespaces

Upstream families advertise their own options in two buckets, and the prefix decides which bucket an entry lands in. The key that follows the prefix is passed through verbatim, and upstream names those keys after the family, so both segments are needed:

options:
  # Loader options (upstream "load options"):
  - load.ace_step.dit_model_path:acestep-v15-turbo
  - load.vevo2.whisper_model_path:/models/whisper-dir
  - load.vibevoice.lora:/models/my-adapter
  # Session options (upstream "session options"):
  - session.seed_vc.weight_type:q8_0

The names each family accepts are declared upstream in its loader, and audio.cpp's own CLI prints them under "Model session options" and "Model load options" when asked for a model's help. Options that upstream lists as request options are not configurable here: they come from the API request, not from the model YAML.

Voice activity detection with no download

The backend package ships upstream's silero_vad and marblenet_vad assets, so VAD works with nothing to fetch. bundled:<name> resolves to the asset directory inside the installed backend. Those assets are directories rather than GGUF files, so the family cannot be inferred and family: is required:

name: silero-vad
backend: audio-cpp
parameters:
  model: bundled:silero_vad
known_usecases:
  - FLAG_VAD
options:
  - family:silero_vad

Then call it like any other VAD model, as described in [Voice Activity Detection]({{%relref "features/voice-activity-detection" %}}).

Source separation stems

Separation families such as htdemucs and mel_band_roformer produce several named outputs from one run. Running inference once per stem would cost a full separation each time, so the backend runs once, writes every stem to a sibling file next to the main output, and puts the selected one in the response body. params[stem] picks which one:

curl http://localhost:8080/audio/transformations \
  -F model=htdemucs -F audio=@song.wav -F 'params[stem]=drums' -o drums.wav

Without params[stem] the body carries vocals when the model produces it, and the first output otherwise. An unknown stem name is refused with a message listing the stems the model really has, rather than quietly returning a different one. Every stem, including the one in the body, is listed in the X-Audio-Stems response header. See [Audio Transform]({{%relref "features/audio-transform" %}}) for the full endpoint contract.

params[stem] against a model that produces a single unnamed output is refused rather than ignored.

Text-conditioned transforms

s2s is a text-and-prosody route, not an audio-only one: vevo2 resynthesises a line in another voice and reads the line itself, refusing the run without it. AudioTransform has no text field, so the text arrives as params[text] (or params[target_text], the name vevo2's own option table uses) and the backend lifts it into the engine's text input. params[language] rides along when the line is not English.

curl http://localhost:8080/audio/transformations \
  -F model=audio-cpp-vevo2-speech-to-speech \
  -F audio=@source.wav -F reference=@target-speaker.wav \
  -F 'params[text]=The quick brown fox jumps over the lazy dog.' -o converted.wav

A model that takes no text ignores the keys, so separation and plain voice conversion are unaffected.

Pinned tasks

Two routes are unreachable by auto-routing and need task: in the model config, because nothing in a request distinguishes them from a task the same family also advertises:

• task:svc for singing voice conversion. seed_vc and vevo2 advertise both svc and ordinary vc, and no request signal means "this input is singing", so vc always wins.
• task:s2s for speech to speech, for the same reason against the same vc.

The gallery entries that ship these routes carry the pin already. Removing it gives plain voice conversion from the same weights.

Family notes

• Supertonic: use the orig GGUF package, whose weights are f32. The f16 package was observed to reach ggml_concat with mismatched operand types and take the backend process down with SIGABRT on the first request, rather than returning an error. Upstream's docs/gguf.md leaves supertonic's 16-bit column untested and records its q8_0 as "No (unsupported weight dtype)", which says the format is unusable rather than that it crashes. The backend refuses both at load time with a message naming the offending tensor, so neither reaches that abort.
• Live transcription latency: AudioTranscriptionLive needs a family advertising asr in streaming mode. At the pinned upstream revision that is nemotron_asr, higgs_audio_stt and voxtral_realtime. higgs_audio_stt and voxtral_realtime decode incrementally and emit partial text as chunks arrive. nemotron_asr only buffers in its chunk handler and decodes everything in finalize(), so with that family every delta arrives after the client closes its send side. Pick accordingly if you need partials during speech.
• Voice conversion vs singing: seed_vc and vevo2 advertise both vc and svc, and automatic routing always picks vc. Pin task:svc for singing voice conversion.
• Cloning-only families: chatterbox advertises clon and vc, and no plain tts at all, so every /v1/audio/speech request against it must carry a reference clip in voice. Without one the request is refused with INVALID_ARGUMENT naming that field, rather than failing deep inside the model.

Acceleration

Variant	Platform
CPU	linux/amd64, linux/arm64
CUDA 12	linux/amd64
CUDA 13	linux/amd64
Metal	darwin/arm64

There is no ROCm image, because upstream has no HIP build configuration, and no Vulkan image: it would ship a Vulkan loader with no ICD inside the container. BUILD_TYPE=vulkan still works when building the backend yourself.