LocalAI/docs/content/features/openai-realtime.md

---

title: "Realtime API" weight: 37

LocalAI supports the OpenAI Realtime API which enables low-latency, multi-modal conversations (voice and text) over WebSocket.

To use the Realtime API, you need to configure a pipeline model that defines the components for Voice Activity Detection (VAD), Transcription (STT), Language Model (LLM), and Text-to-Speech (TTS).

Configuration

Create a model configuration file (e.g., gpt-realtime.yaml) in your models directory. For a complete reference of configuration options, see [Model Configuration]({{%relref "advanced/model-configuration" %}}).

name: gpt-realtime
pipeline:
  vad: silero-vad-ggml
  transcription: whisper-large-turbo
  llm: qwen3-4b
  tts: tts-1

This configuration links the following components:

• vad: The Voice Activity Detection model (e.g., silero-vad-ggml) to detect when the user is speaking.
• transcription: The Speech-to-Text model (e.g., whisper-large-turbo) to transcribe user audio.
• llm: The Large Language Model (e.g., qwen3-4b) to generate responses.
• tts: The Text-to-Speech model (e.g., tts-1) to synthesize the audio response.

Make sure all referenced models (silero-vad-ggml, whisper-large-turbo, qwen3-4b, tts-1) are also installed or defined in your LocalAI instance.

A pipeline stage can name a [failover chain]({{%relref "features/model-failover" %}}); the stage then switches targets without closing the session.

Install the Qwen streaming assistant preset

The gallery entry qwen3-streaming-assistant configures an English voice assistant with Silero VAD, Parakeet TDT 0.6B v3, Qwen3 0.6B, and Qwen3-TTS 0.6B CustomVoice using Ryan's voice. It streams LLM output text into one persistent TTS session while audio is emitted.

The preset installs only its pipeline configuration. Install all four component models first, keeping their default names so the pipeline can resolve them:

local-ai models install silero-vad-ggml
local-ai models install parakeet-cpp-tdt-0.6b-v3
local-ai models install qwen3-0.6b
local-ai models install qwen3-tts-cpp-customvoice --variant qwen3-tts-cpp-customvoice
local-ai models install qwen3-streaming-assistant

The TTS command pins the Q8 CustomVoice entry. These are separate installs; installing the preset does not install missing components. Use a LocalAI build with incremental Qwen3-TTS input support, then select qwen3-streaming-assistant on the Talk page or as the Realtime API model. WebRTC also requires the opus backend, available from the Backends page.

This preset targets English speech. It uses a named voice and does not require reference audio or ref_text. CPU synthesis can be slower than playback; this preset does not establish GPU or browser latency or throughput.

Install a clause-buffered assistant preset

Two English presets use the same Silero VAD, Parakeet TDT 0.6B v3, and Qwen3 0.6B components. vits-streaming-assistant uses VITS-LJS through Sherpa; piper-streaming-assistant uses Piper's Amy voice.

Install the shared components once, keeping their default names:

local-ai models install silero-vad-ggml
local-ai models install parakeet-cpp-tdt-0.6b-v3
local-ai models install qwen3-0.6b

Then install either TTS component and its matching preset, or install both:

local-ai models install vits-ljs-sherpa
local-ai models install vits-streaming-assistant
local-ai models install voice-en-us-amy-low
local-ai models install piper-streaming-assistant

These presets install pipeline configurations only. They do not install missing components. Both enable LLM streaming, TTS streaming, and clause_chunking: true. Sherpa streams audio within each clause. Piper emits one audio result per clause. Each clause starts a separate synthesis, which can introduce pauses between clauses. The single-speaker models need no voice override or reference audio.

Select the installed preset on the Talk page or use its name as the Realtime API model. WebSocket and WebRTC share the pipeline; WebRTC also needs the opus backend from the Backends page. Config installation and discovery checks do not establish speech quality, browser latency, or hardware throughput.

Streaming the pipeline

By default each stage runs to completion before the next begins: the whole utterance is transcribed, the full LLM reply is generated, then it is synthesized. Each stage can instead be streamed incrementally, which lowers the time-to-first-audio of a turn:

name: gpt-realtime
pipeline:
  vad: silero-vad-ggml
  transcription: whisper-large-turbo
  llm: qwen3-4b
  tts: tts-1
  streaming:
    llm: true             # stream LLM tokens as transcript deltas
    tts: true             # emit audio deltas per synthesized chunk
    transcription: true   # stream transcript text deltas of the user's speech
    clause_chunking: true # synthesize each clause as soon as it completes

• streaming.tts: emit a response.output_audio.delta per audio chunk the TTS backend produces (requires a backend that supports streaming synthesis), instead of one delta for the whole utterance. Falls back to a single unary delta otherwise.
• streaming.transcription: stream conversation.item.input_audio_transcription.delta events as the transcript is produced (requires a transcription backend that supports streaming).
• streaming.llm: stream the LLM reply token-by-token as response.output_audio_transcript.delta events. When TTS streaming is disabled, the full reply is synthesized once it is complete unless explicit clause chunking is enabled. Native incremental input or explicitly enabled clause buffering can produce audio before the reply finishes, as described below. Reasoning/thinking is always stripped from the spoken transcript. Tool calls are supported while streaming when the LLM uses its tokenizer template (use_tokenizer_template: true): the backend's autoparser then delivers content and tool calls separately, so the spoken transcript never leaks tool-call tokens. Grammar-based function calling keeps the buffered path.
• Incremental Qwen3-TTS input: with a qwen3-tts-cpp TTS model, enabling both streaming.llm and streaming.tts forwards cleaned LLM text fragments into one persistent synthesis session. Audio can arrive before the LLM finishes. WebSocket and WebRTC use this path. It takes precedence over clause_chunking; other backends can opt into clause buffering as described below. Pending text is bounded and slow audio playback applies backpressure. Cancellation stops the session. A failed live utterance is not replayed through a fallback model. The initial native implementation supports a single batch slot and reference audio without ref_text; transcript-based ICL cloning is unavailable for live input. Incomplete words are buffered until a stable boundary; CJK, Kana and Hangul characters can commit without whitespace.
• streaming.clause_chunking: instead of buffering the whole reply before TTS, split it into speakable clauses and synthesize each as soon as it completes, lowering the time-to-first-audio. The splitter is script-aware: it uses Unicode sentence segmentation (so it handles CJK 。！？ with no whitespace), CJK clause punctuation (，、；：), and Thai/Lao spaces - it does not rely on whitespace sentence boundaries, so it works for languages such as Chinese, Japanese and Thai where the old per-sentence approach degraded to whole-message buffering. Requires streaming.llm; scripts that genuinely need a dictionary (e.g. Khmer, Burmese) simply stay buffered until a space or end-of-message. Set clause_chunking: true to enable a bounded clause queue. Unset and false retain whole-reply synthesis for backends without native incremental text input. With streaming.tts enabled, Sherpa/VITS can stream audio within each clause; unary engines such as Piper produce one audio result per clause. With streaming.tts disabled, each clause uses unary synthesis. These engines start a separate synthesis for each clause, so they may pause at clause boundaries. Cancellation stops queued work; a streaming failure after audio has been emitted never replays that clause through unary synthesis.

All streaming flags are off by default. Pipelines with streaming disabled remain unaffected.

Model warm-up (cold start)

Without warm-up the pipeline's models are loaded into memory only on first use within a session: the VAD on the first audio chunk, transcription at the first end-of-speech, the LLM on the first reply, and TTS on the first spoken output. On a cold session this staggers a load delay across those first few interactions - and a model that fails to load (missing weights, wrong backend, out of memory) only fails part-way through the first turn.

To avoid that, LocalAI warms the pipeline by default: it loads the VAD, transcription, LLM and TTS backends into memory before the session is announced, and the session start blocks until they are all ready. The loads run concurrently, so the wait is the slowest single model, not the sum. This means:

• The first turn pays no cold-start cost - every backend is already resident.
• Model-load errors surface at session start. If any stage fails to load, the session is not started and the client receives a model_load_error instead of session.created, so a broken pipeline fails fast and visibly rather than mid-call.

Set disable_warmup: true to restore the lazy "load on first use" behavior - session start no longer waits on loading and load errors surface on the first turn instead. Useful if you want idle sessions to avoid holding model memory they may never use:

name: gpt-realtime
pipeline:
  vad: silero-vad-ggml
  transcription: whisper-large-turbo
  llm: qwen3-4b
  tts: tts-1
  disable_warmup: true   # lazily load each model on first use instead of at session start

Pre-loading a pipeline on demand

Warm-up only fires when a realtime session opens. To load a pipeline into memory ahead of time - e.g. to warm it right after boot, or when running with disable_warmup: true - POST the model name to the admin-only /backend/load endpoint. For a pipeline model it loads every configured sub-model (VAD, transcription, LLM, TTS, sound_detection, voice_recognition) concurrently:

curl -X POST http://localhost:8080/backend/load \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-realtime"}'

The endpoint is not realtime-specific - it pre-loads any model. See [Backend Monitor]({{%relref "operations/backend-monitor" %}}) for the full request/response reference (it is the inverse of /backend/shutdown).

Controlling response interruption

By default, speech detected by either VAD mode interrupts the active response. Set interrupt_response: false to let the response finish while the user speaks:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "audio": {
      "input": {
        "turn_detection": {
          "type": "server_vad",
          "silence_duration_ms": 500,
          "interrupt_response": false
        }
      }
    }
  }
}

This option also applies to semantic_vad. Speech events and transcription continue. If a response is active when another turn commits, LocalAI retains that turn in conversation history but skips its automatic response. It does not queue a response for that turn. A later turn can generate a response once the active one finishes. To request a response manually, wait for both response.done and the overlapping turn's conversation.item.added event, then send response.create. Transcription can finish after the active response. Explicit response.cancel and client response creation keep their existing behavior. When replacing the turn-detection settings, omit interrupt_response or set it to true to enable interruption again.

Turn detection

For client-controlled turns, set turn_detection to null in session.update. After sending audio, send input_audio_buffer.commit, followed by response.create. If transcription is still running, the response waits for the committed user item before starting the LLM. A standalone response.create with no pending client commit starts without this wait. If the commit produces no user item (transcription error, empty transcript or a voice gate rejection), the server does not run the LLM. It sends response.done with status failed and an input_not_committed error, after the error event that describes the cause. If you cancel the response while it waits, you get response.done with status cancelled.

Turn detection decides when the user has finished speaking and the pipeline should respond. Two modes are supported, matching the OpenAI session schema:

• server_vad (default): silence-based. The VAD model watches the audio and the turn commits after silence_duration_ms (default 500 ms) of silence. Simple and model-agnostic, but a fixed silence window must trade interrupting mid-sentence pauses against sluggish responses.
• semantic_vad: model-driven. The transcription model itself signals end-of-utterance and the silence window becomes dynamic: short right after the model emits its end-of-utterance token, much longer when it does not - so pausing to think no longer gets cut off, while finished sentences get a fast response.

semantic_vad requires a transcription model that emits an end-of-utterance token over a cache-aware streaming decode - currently parakeet-cpp-realtime_eou_120m-v1 (the model is trained to distinguish "paused, expecting a reply" from "paused mid-thought"). The realtime pipeline feeds it the microphone audio live while the user speaks. With any other transcription backend the session degrades gracefully to silence-only detection using the eagerness timeout below (a warning is logged once). The model also emits a distinct end-of-backchannel token (<EOB>) for short acknowledgments like "uh-huh": those are transcribed but never treated as the user yielding the turn.

Sessions can opt in via session.update (turn_detection: {"type": "semantic_vad", "eagerness": "medium"}), or the pipeline can set a server-side default so clients need no changes:

name: gpt-realtime
pipeline:
  vad: silero-vad-ggml
  transcription: parakeet-cpp-realtime_eou_120m-v1
  llm: qwen3-4b
  tts: tts-1
  turn_detection:
    type: semantic_vad   # default for sessions on this model (server_vad if unset)
    eagerness: medium    # low | medium | high | auto (auto == medium)
    retranscribe: false  # see below
    # vad_window_sec: 6  # widen the per-tick VAD scan window (see below)

A client session.update still overrides type and eagerness per session.

VAD scan window: each turn-detection tick the VAD rescans only the most recent slice of buffered audio — sized automatically from the commit silence threshold (the server_vad silence window, or the semantic eagerness fallback) plus a warm-up margin, so long turns cost the same per tick as short ones. vad_window_sec widens the window if needed; values below the automatic floor are ignored. Buffered turn audio is retained for at most 90 s — a turn that genuinely never pauses (continuous speech or a noise source the VAD keeps classifying as speech) keeps only its most recent 90 s for the commit-time batch transcription (the semantic live stream is unaffected — it already consumed the audio incrementally).

Eagerness sets the fallback silence window used when no end-of-utterance token was seen (the model missed it, or the user genuinely trails off): low waits 8 s, medium/auto 4 s, high 2 s - the same max-timeout semantics OpenAI documents. After the token is seen, the turn commits on the next VAD tick (~300 ms).

Live captions: while the user speaks, semantic_vad streams conversation.item.input_audio_transcription.delta events under the item id the commit will later reuse, so clients can render the words as they are recognized. The completed event at commit carries the authoritative transcript and replaces the partial text (with retranscribe: true it may differ from the captions); a turn discarded before commit emits conversation.item.input_audio_transcription.failed so clients can retract its captions.

retranscribe (server-side only, semantic_vad only) cross-checks the streaming decode against a batch decode at commit time:

• false (default): the transcript accumulated from the live stream is used as-is - the model runs once per utterance and the LLM starts immediately at commit.
• true: the committed audio is re-transcribed offline. If the batch decode also ends with the end-of-utterance token the turn proceeds (using the batch transcript); if it does not, the commit is cancelled and the session keeps listening - treating the streaming token as a false positive. Both transcripts are compared and logged, which makes this mode a useful diagnostic for how well the streaming and batch decodes align, at the cost of one extra decode per turn.

Live speaker and sound events (parakeet-cpp)

When the semantic_vad transcription model is a parakeet-cpp model loaded with a diarization_model and/or sound_model companion (see [Audio to Text]({{% relref "audio-to-text" %}})), the realtime session also streams speaker and sound events while a turn is live, alongside the transcript deltas. Nothing needs to change on the client: unrecognized event types are ignored by standard OpenAI Realtime clients.

The transcription model, with its companions:

name: parakeet-realtime-scene
backend: parakeet-cpp
parameters:
  model: realtime_eou_120m-v1-f16.gguf
options:
  - diarization_model:nemotron-3-diarization-q8_0.gguf
  - sound_model:ced-tiny-q8_0.gguf

The realtime pipeline that uses it:

name: gpt-realtime
pipeline:
  vad: silero-vad-ggml
  transcription: parakeet-realtime-scene
  llm: qwen3-4b
  tts: tts-1
  turn_detection:
    type: semantic_vad

Each closed speaker segment emits a conversation.item.input_audio_transcription.segment event under the turn's item id, with an empty text (the event exists to carry the speaker boundary, not a transcript - the transcript still comes from the ordinary delta/completed events):

{
  "type": "conversation.item.input_audio_transcription.segment",
  "item_id": "item_abc",
  "content_index": 0,
  "speaker": "0",
  "speaker_name": "Alice",
  "start": 1.92,
  "end": 4.10,
  "text": ""
}

speaker_name is the name of a voice registered through /v1/voice/register, and is present only when the model has a speaker_model: (or, for a parakeet-cpp bundle file, a speaker_component:) and the speaker was identified. A segment that closes before its speaker is identified has none, and later segments of the same speaker do. See [Voice Recognition]({{% relref "voice-recognition" %}}#naming-speakers-in-diarization-and-live-transcription). The segments of the offline path below carry no speaker_name.

Each sound event emits a conversation.item.sound_detection event with one tag and the detection window's start/end:

{
  "type": "conversation.item.sound_detection",
  "item_id": "item_abc",
  "content_index": 0,
  "detections": [{"label": "Chicken, rooster", "score": 0.91, "index": 99}],
  "start": 24.0,
  "end": 30.0
}

The start/end on both event types are seconds measured from the start of the current turn's own audio, not the session or the WebSocket connection - the same base the streamed transcript words use.

The companion stream is opened fresh for each speech turn, alongside that turn's ASR live session, and closed when the turn commits: whatever it had not yet emitted is drained and sent at that point. Because the diarization model runs a brand new session every turn, its speaker indices are scoped to the turn too - "speaker": "0" in one turn and "speaker": "0" in the next are not guaranteed to be the same person, even within the same conversation.

score is the peak score seen for that tag while the sound was live, not an average.

Limitation: under semantic_vad, live transcription (and so this companion stream) only runs during speech turns - it does not see audio between turns. A sound that happens while nobody is speaking is not detected this way. If you need sound events independent of speech turns, use the pipeline's sound_detection model instead (see [Sound Classification]({{% relref "audio-classification" %}})), which classifies each VAD-committed utterance on its own. Use one or the other, not both, on the same session - they overlap in purpose and would emit sound detections twice.

Speaker and sound events with an offline model (Parakeet TDT v3)

The live events above need a cache-aware streaming transcription model. An offline model such as Parakeet TDT 0.6B v3 (25 languages) runs under server_vad instead: each VAD-committed turn is transcribed as a whole. The gallery model parakeet-cpp-realtime-scene-tdt bundles it with Nemotron-3-Diarization and CED-Tiny, so one parakeet-cpp backend handles transcription, speakers and sounds. Point both transcription and sound_detection at it and turn on diarization:

name: gpt-realtime-scene
pipeline:
  vad: silero-vad-ggml
  transcription: parakeet-cpp-realtime-scene-tdt
  sound_detection: parakeet-cpp-realtime-scene-tdt
  diarization: true
  llm: qwen3-4b
  tts: tts-1

pipeline.diarization asks the transcription model for speaker labels on each committed turn and emits every labelled segment as a conversation.item.input_audio_transcription.segment event before the turn's completed event. Unlike the live path, these segments carry their text:

{
  "type": "conversation.item.input_audio_transcription.segment",
  "item_id": "item_abc",
  "content_index": 0,
  "id": "seg_1",
  "speaker": "1",
  "start": 6.85,
  "end": 10.82,
  "text": "Well, I don't wish to see it any more, observed Phoebe, turning away her eyes."
}

sound_detection classifies the same committed audio and emits one conversation.item.sound_detection event per turn (see [Sound Classification]({{% relref "audio-classification" %}})). As on the live path, times are relative to the turn's audio and speaker labels are only consistent within a turn. pipeline.diarization is off by default: it needs a transcription model that diarizes (parakeet-cpp with a diarization_model companion), and some other backends fail a diarization request they cannot serve.

Choosing the sound model

Both scene models ship with CED-Tiny, the cheapest to run all the time. parakeet-cpp-realtime-scene-base and parakeet-cpp-realtime-scene-tdt-base are the same pipelines with CED-Base (86M, the largest CED), which tags sounds more confidently. parakeet-cpp-multilingual-diarization-speakers-sounds loads the same TDT, Nemotron-3-Diarization and CED-Tiny trio plus the WeSpeaker encoder, and is meant for offline /v1/audio/diarization calls with include_sounds=true (see [Speaker Diarization]({{% relref "audio-diarization" %}}#sound-events)). Any CED GGUF from mudler/ced-gguf (tiny, mini, small, base) works as sound_model. Measured on CPU (Ryzen 9 9950X3D) over a 37 s clip with two speakers and a rooster, as a fraction of real time:

	CED-Tiny	CED-Base
Live scene stream (diarization low + sound), EOU path	0.103	0.125
Sound detection per committed turn, TDT path	0.005	0.031

The EOU model's own ASR stream adds 0.016. Diarization dominates the live cost, so CED-Base keeps the live path about 7x faster than real time.

Disabling thinking

For reasoning models, you can force the pipeline LLM's thinking off without editing the LLM model config:

pipeline:
  llm: qwen3-4b
  disable_thinking: true   # maps to enable_thinking=false for the realtime LLM

This is applied only to the realtime session's copy of the LLM config, so it does not affect other users of the same model. Leave it unset to use the LLM model config's own reasoning settings.

Conversation compaction (long sessions on CPU)

By default a realtime session feeds only the last max_history_items turns to the LLM; older turns are dropped and forgotten. On CPU, long calls also grow expensive as the prompt fills with verbatim history. Enable compaction to instead fold older turns into a rolling summary, so long calls stay cheap without losing earlier context.

Compaction works with two numbers:

• max_history_items is the live window - the recent turns kept verbatim in the prompt.
• compaction.trigger_items is the high-water mark - let the buffer grow to here, then summarize the overflow (everything above max_history_items) into a rolling memory and evict it. It must be greater than max_history_items; if it is not, it is clamped up.

The gap between the two controls how often summarization runs: a summary call fires roughly every (trigger_items - max_history_items) turns (here, about every 6 turns).

pipeline:
  max_history_items: 6        # live window - recent turns kept verbatim
  compaction:
    enabled: true
    trigger_items: 12         # summarize overflow back down to max_history_items
    summary_model: ""         # optional: a small model for the summary (CPU); default = pipeline LLM
    max_summary_tokens: 512

{{% notice tip %}} On CPU, set summary_model to a small, fast model so compaction never competes with the conversation LLM for compute. Left empty, the pipeline's own LLM produces the summary. {{% /notice %}}

Clients can also manage history directly via the now-supported conversation.item.delete, conversation.item.truncate, and input_audio_buffer.clear realtime events.

Classifier mode (LocalAI extension)

On hardware that can afford prompt processing but not token generation — a Raspberry Pi running a small LLM, for example — a realtime session can replace autoregressive generation with prefill-only classification: you register a fixed list of options, each user turn is scored against them with the Score primitive (a single forward pass, no decode), and the winning option's canned reply is spoken and/or its canned tool call is emitted. On llama-cpp, scoring runs through the same server slot the LLM uses, so the conversation prefix stays KV-cached across turns, and all options are scored together in one batched decode: the shared prefix (prompt plus the options' common token prefix) is processed once, then each option's unique tail rides a forked sequence in a single forward pass. A warm turn costs roughly one pass over the new words plus one small batch over the option tails, independent of the option count.

Enable it in the pipeline config:

name: drone-pi
pipeline:
  vad: silero-vad-sherpa
  transcription: parakeet-cpp-realtime_eou_120m-v1
  llm: lfm2.5-1.2b-instruct       # scores AND (if asked) generates
  tts: vits-piper-en_US-amy-sherpa
  classifier:
    enabled: true
    threshold: 0.85               # softmax floor the winner must clear (see note below)
    fallback:
      mode: reply                 # none | reply | generate
      reply: "Say again?"
    options:
      - id: up
        description: the user asks the drone to move or fly up/higher
        reply: Going up.
        tool:
          name: move
          arguments: {direction: up}
      - id: greeting
        description: the user greets the assistant
        reply: Hello, ready to fly.

Or per session / per response from the client (the field is additive — OpenAI clients simply never send it):

{"type": "session.update", "session": {"type": "realtime", "localai_classifier": {
  "enabled": true, "threshold": 0.85,
  "options": [{"id": "up", "description": "...", "reply": "Going up.",
               "tool": {"name": "move", "arguments": {"direction": "up"}}}],
  "fallback": {"mode": "reply", "reply": "Say again?"}
}}}

Like tools, the option list is replaced wholesale on each update. A response.create may carry its own localai_classifier to override the session for one response — {"enabled": false} runs normal generation once.

How a classified response behaves:

• The winner's reply is emitted through the ordinary response events (spoken via TTS, or response.output_text.* in text-only mode), and its tool (if any) is emitted as a standard function_call item with exactly the configured arguments — the client executes it and reports back with conversation.item.create as usual. In classifier mode you typically should not send a follow-up response.create after the tool output: the canned reply already acknowledged the command, and the follow-up would classify a tool-output turn.
• Every classified response also emits a localai.classifier.result event carrying the full softmax distribution, the chosen option id (empty when the fallback applied), the threshold, and the scoring latency — useful for visualizing confidence in a client UI.
• A committed turn whose transcript is empty (the VAD fired on noise and the ASR heard no words) is never scored — an empty prompt produces a confidently arbitrary winner. The fallback applies directly: the result event carries an empty scores list, and generate mode falls through to generation.
• Wake-word gating: set address: {names: ["drone"], mode: ignore} and the assistant only acts on turns that mention one of the names ("Drone go up", not just "go up"). The check is a deterministic case-insensitive whole-word match on the latest transcript — deliberately not model-based: scoring cannot detect a missing name (a 1.2B scorer rates "go up" as addressed with p≈1.0 even with a dedicated addressing stage), while a literal match is exact and free. Unaddressed turns skip scoring entirely (ambient conversation costs nothing) and emit a result event with empty scores and fallback: "not_addressed"; mode: ignore completes the response silently, mode: reply speaks reply.
• When no option clears threshold, fallback.mode decides: none completes the response with no output, reply speaks the canned fallback reply, and generate falls through to normal autoregressive generation (slow on weak hardware, but always available since the same model config serves both paths). Set the threshold high: with the default raw normalization a confident in-list pick lands near 1.0, while an out-of-list request spreads its probability across the options — measured on a 1.2B scorer, in-list utterances scored ≥0.97 and out-of-list ones peaked around 0.8, so a floor of ~0.85 separates them. A low threshold (say 0.35) practically never falls back. Also keep each option's description narrowly scoped: a catch-all clause like "…or asks for help" turns that option into a magnet for every request the model cannot map, defeating the fallback.
• Agentic follow-up turns (after a server-side assistant tool executes) always use generation — the option list describes user intents, not tool outputs.

Knobs that matter for latency and accuracy: keep option descriptions short (they all go into the scoring system prompt) and the option count small. By default only the latest user message is scored — earlier turns echo option names (the canned replies especially) and empirically make small scoring models re-choose the previous option regardless of the new command. history_items: N opts the trailing N conversation messages back in (role-labeled); only do that with a scorer large enough to weigh the context. normalization: mean divides each option's joint log-prob by its token count — useful when option ids have very different lengths. The scoring model needs a Go-side chat template (template.chat / template.chat_message); without one the scoring prompt falls back to a generic ChatML envelope, which may be off-distribution for the model. Use classifier.model to score on a different config than the pipeline LLM (rarely needed).

Argument slots (hybrid classify-then-complete)

A canned tool call can leave holes for the model to fill: declare slots on the option's tool and reference them as "{{name}}" in the arguments template. When the option wins, LocalAI runs a short grammar-constrained completion that continues the exact scoring prompt (so the llama.cpp prompt cache is already warm) with the chosen route JSON re-opened at the first slot — only the value tokens are free; everything else is pinned by the grammar. Number slots substitute unquoted, enum/string slots inside their quotes.

tool:
  name: move_drone
  arguments:
    direction: forward
    distance: "{{distance}}"
    units: "{{units}}"
  slots:
    - name: distance
      type: number            # number | enum | string
    - name: units
      type: enum
      values: [m, meters, ft, feet]
      default: m              # used if inference fails outright
      hint: assume m when the user gives no units

The option's spoken reply can reference the same placeholders — reply: "Going forward {{distance}} {{units}}." — and the filled values are spliced in as plain text before the reply is emitted, so what the assistant says confirms what it inferred. Reply placeholders are optional (one that names no slot stays literal).

Slot declarations (and hints) are appended to the option's description in the shared system prompt, so they also inform scoring and cost no extra per-turn tokens. The localai.classifier.result event carries the final arguments and a fill_latency_ms. On an inference failure the slots' defaults apply (and template the reply); if any slot lacks a default the response fails (or falls through to generation with fallback.mode: generate). Slot filling requires completion in the scoring model's known_usecases alongside score.

Registering an option list (pipeline seed or session.update) prewarms the scoring prompt in the background: one throwaway score prefills the option-list prompt and plants a state checkpoint exactly at the per-turn probe boundary. Every scoring call also declares that boundary (the probe-invariant prompt prefix) to the backend, which checkpoints there on each prefill — on hybrid/recurrent models (which cannot rewind their state arbitrarily, only restore checkpoints) this is what keeps every turn at probe-size cost instead of a full option-list re-prefill. A client that swaps option lists at runtime (e.g. voice-switched command modes) pays nothing on the first turn after a swap: the rewarm hides behind the acknowledgement reply, and it runs on every registration deliberately — if the list's slot was evicted, the rewarm is exactly the re-prefill the next turn would otherwise pay in the foreground. Swapping between several lists? Give the scoring model a slot per list and make the prefix routing selective: options: [parallel:3, sps:0.5] (the default slot-similarity threshold of 0.1 funnels different lists onto one slot — they share enough prompt structure to clear it).

The concrete scoring model must declare score in known_usecases. A single llama.cpp model can serve ordinary inference and classification concurrently by declaring multiple use cases, for example known_usecases: [chat, completion, score]; LocalAI reserves the scoring slots only when score is present. Scoring also requires the unified KV cache, which is enabled by default, so a score-enabled model cannot set kv_unified:false.

Speech-to-speech models (experimental)

A speech-to-speech (S2S) model takes audio in and answers with audio, text or both, in one model. Instead of chaining VAD, transcription, an LLM and TTS, the session sends audio to the model and gets the reply back from it. The llama-cpp backend serves these models.

{{% notice warning %}} Speech-to-speech over the Realtime API is experimental. It needs a llama.cpp build with LFM3-Audio and realtime support, which the LocalAI llama.cpp pin must include. It has not been measured on real hardware, and its behaviour may change. {{% /notice %}}

Configuration

The gallery has LiquidAI LFM3-Audio-2B in two quantizations (Q4_K_M or Q8_0 weights, both with the Q8_0 audio projector) and four shapes each. Every entry is independent: installing lfm3-audio-2b always gets the Q4_K_M build, never Q8_0.

Entry (Q4_K_M)	Entry (Q8_0)	Shape	VAD
lfm3-audio-2b	lfm3-audio-2b-q8	One speech-to-speech model	none, the client commits turns
lfm3-audio-2b-vad	lfm3-audio-2b-q8-vad	One speech-to-speech model	silero-vad-ggml
lfm3-audio-2b-pipeline	lfm3-audio-2b-q8-pipeline	Chained pipeline on the same model	none, the client commits turns
lfm3-audio-2b-pipeline-vad	lfm3-audio-2b-q8-pipeline-vad	Chained pipeline on the same model	silero-vad-ggml

The gallery cannot install one model as a dependency of another. The -vad entries name silero-vad-ggml in pipeline.vad, so install it first with local-ai models install silero-vad-ggml. Without it, the session fails to start with a model-not-found error.

Install one with local-ai models install lfm3-audio-2b. Its model configuration looks like this:

name: lfm3-audio-2b
backend: llama-cpp
context_size: 8192
mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf
parameters:
  model: LFM3-Audio-2B-Q4_K_M.gguf
options:
  - parallel:2
known_usecases: [realtime_audio, chat, transcript, tts]
pipeline:
  any_to_any: true
template:
  use_tokenizer_template: true

• pipeline.any_to_any: true runs the model as one speech-to-speech model. Without it, a realtime_audio model keeps the chained path (VAD, transcription, LLM, TTS).
• parallel:2 gives the model two slots. One is held by the session. The other serves transcription, TTS or chat calls at the same time.

Chained pipeline on LFM3

The -pipeline entries do not use the speech-to-speech stream. They run the classic realtime pipeline (VAD, transcription, LLM, TTS), with the transcription, chat and speech stages all served by the one llama.cpp model: speech recognition through the transcription path, the reply through the chat path (with its templates, tools and streaming), and speech synthesis through the TTS path with the named voice female (or male). Choose it when you want the regular chat path or the text of each stage; choose the speech-to-speech entries for the lowest latency. The model configuration names itself in each slot and is not a realtime_audio model:

name: lfm3-audio-2b-pipeline-vad
backend: llama-cpp
mmproj: mmproj-LFM3-Audio-2B-Q8_0.gguf
parameters:
  model: LFM3-Audio-2B-Q4_K_M.gguf
known_usecases: [chat, transcript, tts]
tts:
  voice: female
pipeline:
  vad: silero-vad-ggml
  transcription: lfm3-audio-2b-pipeline-vad
  llm: lfm3-audio-2b-pipeline-vad
  tts: lfm3-audio-2b-pipeline-vad

Keep the installed model name, because the slots refer to it. The -pipeline entries have no vad line. A chained pipeline without a VAD starts with turn detection off, refuses turn_detection of type server_vad in session.update, and expects input_audio_buffer.commit followed by response.create.

Migrating from the liquid-audio backend

The Python liquid-audio backend is removed, together with its importer and its gallery entries. It served LFM2-Audio models. Use the llama-cpp backend with lfm3-audio-2b or lfm3-audio-2b-q8 instead, and set pipeline.any_to_any: true as shown above. A model with backend: llama-cpp and realtime_audio but without pipeline.any_to_any is refused with invalid_pipeline, because llama.cpp has no VAD, transcription or TTS of its own.

LFM2.5-Audio is not covered: llama.cpp does not run it, so there is no replacement for it. Remove the old model configurations that name backend: liquid-audio.

Voice activity detection

Speech-to-speech models do not have a VAD of their own. Turn detection depends on the pipeline:

• With a VAD model (pipeline.vad, for example silero-vad-ggml, installed separately), the server detects the start and end of each turn. Use turn_detection with server_vad as in the rest of this page. Audio is forwarded from 300 ms before speech starts, so silence before the user speaks is not added to the model context.
• Without a VAD model (the plain lfm3-audio-2b entries), server VAD is off. The client ends each turn with input_audio_buffer.commit and then sends response.create. Set turn_detection to null in session.update to make this explicit.

Tools

Define tools in session.update as usual. When the model calls one, the session sends response.output_item.added with a function_call item, then response.function_call_arguments.delta and response.function_call_arguments.done, and then response.done. To return the result, send a conversation.item.create with type: function_call_output, the same call_id and the output string, and then response.create. The model answers with the result in the next reply. A reply that is only a tool call has no audio.

How a call is read depends on the model:

• The backend reads the text of the reply with the parser and grammar of the model's chat template, as the llama.cpp server does for /v1/realtime. The call is cut out of the transcript. The audio is sent as the model produces it, also after a call started. tool_choice (auto, none or required) works in session.update and in response.create, and the response value wins. A named function cannot be forced: session.update and response.create refuse it with an error. The backend knows only the tools of the session, so a tools list on response.create does not change what it reads. With tool_choice set to none, a call the model writes anyway stays text.
• If the backend found no call and the request had tools, LocalAI parses the raw text of the reply with the function settings of the model, the way the chained pipeline does. Set function.xml_format_preset (the LFM3-Audio gallery entries set it to lfm3) or the other function options for a format the backend parser does not know. In this case the call text can already be in the streamed transcript, and the transcript is replaced only if function.capture_llm_results captures the content.

Transcripts

• Reply transcript. response.output_audio_transcript.delta and response.output_audio_transcript.done carry the text of the spoken reply. In a text-only response, response.output_text.delta is used instead.
• User transcript. The stream answers from the audio, so an any-to-any model does not transcribe the input unless pipeline.transcription is set. When it names another model (a sherpa-onnx ASR, for example), conversation.item.input_audio_transcription.completed is sent for the user item, in parallel with the reply. When it names the model itself, the transcript is made on a second slot before the reply starts, so it arrives before the first output of that reply (for a manual commit and for a VAD turn), and it delays the reply by its own run time. Both share one llama.cpp context, and while they run together llama.cpp rebuilds its buffers on every step, which slows the reply to a few tokens per second; this is why the default is no transcript. The transcript of the reply always comes from the stream. If transcription fails, conversation.item.input_audio_transcription.failed is sent and the reply continues.

Barge-in

Interruption works as described in Controlling response interruption. When the user starts speaking during a reply, the server stops the reply, sends response.done with status: cancelled, and handles the new speech as the next turn. The part of the reply that was already decoded stays in the model context, as in llama.cpp.

Limits

• Audio format. The model listens to 16 kHz mono audio. LocalAI resamples the session input to 16 kHz. Output audio is 24 kHz mono PCM.
• Voice. The voice setting is not applied to speech-to-speech output. llama.cpp's realtime slot has no voice selection. The voice setting still applies to the [text-to-speech endpoint]({{%relref "features/text-to-audio" %}}).
• Context. A session holds one slot for its whole life, and all its audio stays in the context. A long session reaches context_size. The session then sends an error and closes. There is no compaction for speech-to-speech sessions.
• Session updates after the first turn. Once the first audio or message has been added, instructions, tools and voice cannot change. A session.update that tries to change them gets an error. output_modalities and max_output_tokens can still change.
• Not supported. conversation.item.truncate and conversation.item.delete return an error, because the context cannot be cut by audio time. input_audio_buffer.clear drops audio that has not been committed. Without a VAD model, the server keeps appended audio in its buffer and sends it to the model only at input_audio_buffer.commit (or at a response.create that finds uncommitted audio), so a clear discards it for good. With server VAD, audio is streamed to the model during a turn, and what was already sent stays in the context.
• Pipeline features. The pipeline features that need a separate LLM call are rejected with invalid_pipeline: classifier mode, conversation compaction with a summary, streaming.llm, and voice recognition gating.
• Distributed mode. The conversation state lives in one backend slot, so a session stays on the instance that opened it until it ends. See [distributed mode]({{%relref "features/distributed-mode" %}}) for how instances are chosen.

Transports

The Realtime API supports two transports: WebSocket and WebRTC.

WebSocket

Connect to the WebSocket endpoint:

ws://localhost:8080/v1/realtime?model=gpt-realtime

Audio is sent and received as raw PCM in the WebSocket messages, following the OpenAI Realtime API protocol.

WebRTC

The WebRTC transport enables browser-based voice conversations with lower latency. OpenAI-compatible clients can send a raw SDP offer and select the model with the query parameter:

POST http://localhost:8080/v1/realtime/calls?model=gpt-realtime
Content-Type: application/sdp

<SDP offer body>

The response has the application/sdp content type and contains the bare SDP answer.

The unified OpenAI interface is also supported. Send multipart/form-data with an sdp field that contains the offer and a JSON session field. LocalAI reads the model from the session object:

curl http://localhost:8080/v1/realtime/calls \
  -F "sdp=<offer.sdp;type=application/sdp" \
  -F 'session={"type":"realtime","model":"gpt-realtime"};type=application/json'

LocalAI also accepts its original JSON request format for compatibility. A JSON request contains top-level sdp and model fields and receives a JSON response with sdp and session_id fields.

Opus backend requirement

WebRTC uses the Opus audio codec for encoding and decoding audio on RTP tracks. The opus backend must be installed for WebRTC to work. Install it from the Backends page in the web UI, or from the backend gallery with the API:

curl -X POST http://localhost:8080/backends/apply \
  -H "Content-Type: application/json" \
  -d '{"id": "opus"}'

For a local binary installation, you can instead use the CLI:

local-ai backends install opus

Or set the EXTERNAL_GRPC_BACKENDS environment variable if running a local build:

EXTERNAL_GRPC_BACKENDS=opus:/path/to/backend/go/opus/opus

The opus backend is loaded automatically when a WebRTC session starts. It does not require any model configuration file - just the backend binary.

WebRTC behind Docker host networking or NAT

By default pion gathers a host ICE candidate for every local interface. Under Docker host networking that includes bridge addresses (docker0/veth, 172.x) that a remote browser cannot route to: the call typically connects on a good candidate and then drops a few seconds later when ICE consent checks fail on the unreachable ones. Two settings let you advertise only the reachable address:

# Advertise these IPs as the host ICE candidates (e.g. the host's LAN IP)
LOCALAI_WEBRTC_NAT_1TO1_IPS=192.168.1.10

# ...or restrict ICE gathering to specific interfaces
LOCALAI_WEBRTC_ICE_INTERFACES=eth0

{{% notice tip %}} For a browser on another LAN machine talking to LocalAI in a host-networked container, set LOCALAI_WEBRTC_NAT_1TO1_IPS to the host's LAN IP. This is the most reliable fix for WebRTC connections that establish and then drop. {{% /notice %}}

Fixed WebRTC UDP port

By default, each WebRTC peer connection uses an ephemeral UDP port. To route all realtime WebRTC ICE traffic through one shared port, start LocalAI with --web-rtc-udp-port 3478 or set LOCALAI_WEBRTC_UDP_PORT=3478.

When running in a container, publish the same port with the UDP protocol:

docker run -p 8080:8080 -p 3478:3478/udp \
  -e LOCALAI_WEBRTC_UDP_PORT=3478 localai/localai:latest

Allow the selected UDP port through the host and network firewalls. If LocalAI cannot bind it, WebRTC signaling requests return an HTTP 500 error describing the bind failure.

Protocol

The API follows the OpenAI Realtime API protocol for handling sessions, audio buffers, and conversation items.

Response modalities (text-only sessions)

By default a realtime session responds with audio plus a transcript. To make the model respond with text only, set the output modalities on either the session or an individual response:

{"type": "session.update", "session": {"output_modalities": ["text"]}}

{"type": "response.create", "response": {"output_modalities": ["text"]}}

output_modalities is the GA field name. For compatibility, LocalAI also accepts the legacy Realtime beta field name modalities as an alias (a lot of community sample code still sends modalities: ["text"]):

{"type": "session.update", "session": {"modalities": ["text"]}}

The GA output_modalities wins when both are present. A response-level value overrides the session-level one, and when neither is set the session falls back to ["audio"].

Out-of-band responses and metadata

A response can be created outside the default conversation by setting conversation to none. The reply is not added to the conversation history, which is what makes it usable for a side channel — answering a chat message or a webhook while a spoken conversation is in progress.

Because such a response arrives on the same socket as everything else, attach metadata to correlate it. LocalAI echoes the map back verbatim on both response.created and response.done:

{"type": "response.create", "response": {
  "conversation": "none",
  "output_modalities": ["text"],
  "metadata": {"client_run": "abc123"},
  "input": [{"type": "message", "role": "user",
             "content": [{"type": "input_text", "text": "Is the oven still on?"}]}]
}}

{"type": "response.done", "response": {
  "id": "resp_...", "status": "completed",
  "metadata": {"client_run": "abc123"}
}}

Keys are strings up to 64 characters, values up to 512. A response created without metadata omits the field rather than sending an empty object.

Gating a realtime pipeline with voice recognition

A pipeline realtime model can require speaker verification before it responds. Add a voice_recognition block under pipeline. When present, each committed utterance is verified against authorized speakers; unauthorized utterances are dropped before the LLM runs (no LLM call, no tool execution, no TTS). The session stays open.

The same block also drives two optional, independent behaviors: an authorization gate (enforce) and speaker surfacing/personalization (identity). Set enforce: false to keep recognizing the speaker without ever rejecting a turn.

name: my-realtime
pipeline:
  vad: silero-vad
  transcription: whisper
  llm: qwen
  tts: kokoro
  voice_recognition:
    model: speaker-recognition   # the speaker-recognition backend model
    mode: identify               # "identify" (registry) or "verify" (references)
    threshold: 0.25              # cosine distance; <= passes
    enforce: true                # authorization gate (default true)
    when: every                  # "every" (default) or "first"
    on_reject: drop_event        # "drop_event" (default) or "drop_silent"
    anti_spoofing: false         # optional liveness check (verify mode)

    # identify mode: authorized registry identities (multiple persons)
    allow:
      names: ["alice", "bob"]    # match registered speaker names
      labels: ["family"]         # OR any identity carrying this label
      # empty allow = any registered speaker within threshold passes

    # verify mode: reference speakers (multiple persons)
    references:
      - name: alice
        audio: /models/voices/alice.wav
      - name: bob
        audio: /models/voices/bob.wav

One bundle for every stage

A parakeet-cpp bundle can fill the speaker stage as well as the others, so one model serves vad, transcription, sound_detection and voice_recognition. The bundle config must list speaker_recognition in known_usecases (the gallery entries do), and the libparakeet.so must export parakeet_capi_speaker_embed_pcm:

name: my-realtime
pipeline:
  vad: parakeet-cpp-bundle-small
  transcription: parakeet-cpp-bundle-small
  sound_detection: parakeet-cpp-bundle-small
  llm: qwen
  tts: kokoro
  voice_recognition:
    model: parakeet-cpp-bundle-small
    mode: identify
    threshold: 0.5               # WeSpeaker distance; see Voice Recognition

The bundle's encoder has the same weights as voice-detect-wespeaker-resnet34, so voices registered with that model are matched. Embedding an utterance uses the same engine lock as transcription, so it adds to the turn latency. Voices from a 192-dimension model (ECAPA-TDNN, for example) cannot be matched by a bundle; see [Voice Recognition]({{% relref "voice-recognition#a-parakeet-cpp-bundle-as-the-embedding-model" %}}).

Identifying speakers without gating

To recognize who is speaking and surface it to the client and the LLM without ever rejecting a turn, set enforce: false and add an identity block. The identity block works with or without the gate; when it is set, the speaker is resolved on every turn even if when: first.

name: my-realtime
pipeline:
  vad: silero-vad
  transcription: whisper
  llm: qwen
  tts: kokoro
  voice_recognition:
    model: speaker-recognition
    mode: identify
    threshold: 0.25
    # Authorization gate. Defaults to enforcing (rejects unauthorized speakers).
    # Set enforce:false to identify the speaker WITHOUT rejecting anyone.
    enforce: false
    when: every
    # Surface the recognized speaker to the client and the LLM. Works with or
    # without enforce; when set, identity is resolved on every turn even if
    # when:first.
    identity:
      announce: true            # emit the conversation.item.speaker event
      announce_unknown: false   # also emit it when there is no confident match
      personalize: true         # tell the LLM who is speaking
      inject_name: true         # set the per-message OpenAI name field
      inject_system_note: true  # append a "current speaker" line to the system message
      note_unknown: false       # append a "speaker is unknown" note when unidentified

Field	Meaning
model	Speaker-recognition backend model name.
mode	identify matches against speakers registered via /v1/voice/register; verify matches against the references audios.
threshold	Maximum cosine distance that still counts as a match (default ~0.25).
enforce	Authorization gate. true (or omitted) rejects unauthorized speakers (the gating behavior above). false resolves and surfaces the speaker without ever dropping a turn.
when	every verifies each utterance; first verifies once then trusts the session. When an identity block is set, the speaker is still resolved on every turn even with first.
on_reject	drop_event drops and emits a speaker_not_authorized error event; drop_silent drops quietly.
anti_spoofing	Verify mode only: runs the backend liveness check (slower).
allow.names / allow.labels	identify mode: which registry identities are authorized. Empty = any registered speaker.
references	verify mode: authorized reference speakers; the utterance passes if it matches any.
identity.announce	Emit the conversation.item.speaker event to the client (see below).
identity.announce_unknown	Also emit that event when there is no confident match. By default the event is emitted only on a match.
identity.personalize	Inform the LLM who is speaking.
identity.inject_name	Set the per-message OpenAI name field on each user turn.
identity.inject_system_note	Append a The current speaker is <Name>. line to the system message.
identity.note_unknown	When unidentified, append The current speaker is unknown. (lets the model ask who it is talking to).

identify mode requires the voice registry (speakers registered through /v1/voice/register). verify mode needs no registry: reference audios are embedded once at model load.

The conversation.item.speaker event

When identity.announce is enabled, the server emits a conversation.item.speaker event after the user conversation item, naming the recognized speaker. In a transcription-only session there is no conversation item, so the server emits the event before the conversation.item.input_audio_transcription.completed event of the turn, with the same item_id:

{
  "type": "conversation.item.speaker",
  "item_id": "item_abc",
  "speaker": { "name": "Jeremy", "id": "spk_1", "labels": { "role": "owner" }, "confidence": 92.0, "distance": 0.1, "matched": true }
}

confidence is a 0-100 score, distance is the cosine distance, and matched is true when a confident match was found. labels carries any labels attached to the registered speaker (identify mode); it is omitted when the speaker has none. The name and id fields are omitted when empty. By default the event is emitted only on a match; set identity.announce_unknown: true to also emit it (with matched: false) when no speaker is identified.

This event is a LocalAI extension to the OpenAI Realtime API and is server-emitted only. Standard OpenAI Realtime clients ignore event types they do not recognize, so enabling it is non-breaking.

Examples

• Realtime voice assistant demo (Go): a minimal Go client for the Realtime (WebSocket) API with a full talk-back voice loop and an example tool call. Ships a docker compose setup that brings up a realtime-capable LocalAI for you.
• Realtime voice assistant example (Python): thin-client architecture (Silero VAD on the client, heavy lifting on LocalAI), suited to running the client on a Raspberry Pi.

Talk in the web UI

The Talk page of the web UI is a client for this API over WebRTC. Pick a pipeline model with the chip at the top, then press Start session. The microphone streams while the session runs. The server detects when you stop talking and answers on its own, and you can speak over a reply to interrupt it. The page has no push-to-talk or hands-free switch. A speech-to-speech model with no VAD model set has no turn detection: the page shows Done speaking, which ends the turn and asks for the reply.

The heading under the orb says what is happening: connecting, listening, thinking (also while a tool runs), speaking, or interrupted after you cut a reply off. The orb follows the real microphone level while it listens and the reply's audio while it speaks. These states have their own screen:

• Microphone blocked. The browser did not give the page the microphone. Allow it in the address bar and press Try again.
• Connection lost. The WebRTC link failed during a session. The transcript stays and Reconnect starts a new session.
• Something went wrong. The server reported an error or the call could not be set up. The reason is shown, with a link to the traces.
• Talk needs a pipeline model. No pipeline model exists yet. The page links to the model editor with the pipeline template and to the gallery.

The transcript shows what you said, the replies and, in Manage mode, the tool calls and results. Copy puts it on the clipboard. It is not saved.

The sliders button opens the session settings: the instructions, the voice, the transcription language, client-side MCP tools, Manage mode (admin, fixed once a session is open) and the parts of the selected pipeline. The gauge button, shown during a session, adds audio diagnostics (waveform, spectrum and WebRTC statistics).