mirror/LocalAI
1
0 Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-11-06 04:23:15 -05:00
Code Issues Packages Projects Releases Wiki Activity

fix(realtime): leave speech-to-speech client commits at the session rate

Browse Source 
The speech-to-speech path labels its transcription WAV with the session
rate for client commits, so resampling first handed it 16 kHz audio under
a 24 kHz header. Resample only in the chained pipeline, and say so in the
docs.

Assisted-by: Claude:claude-sonnet-5-5 go-test
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
(cherry picked from commit 82cc411af54cf884e4a44a3e96292a323ba5611c)
This commit is contained in:
Ettore Di Giacinto committed 2026-11-03 19:23:38 +00:00
1 parent fbfb2ccadf
commit 9e6a7a077c
2 files changed
+4 -2
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

core/http/endpoints/openai/realtime.go
+3 -1
View File
@@ -2122,7 +2122,9 @@ func vadScanWindowSec(sv *types.RealtimeSessionSemanticVad, silenceThreshold flo
// decoded at 16 kHz (1.5x speed/pitch, garbage or empty transcript, then
// input_not_committed on the follow-up response).
func commitUtterance(ctx context.Context, utt []byte, session *Session, conv *Conversation, t Transport, slot *commitSlot) {
if rate := session.InputSampleRate; rate > 0 && rate != localSampleRate && len(utt) > 0 {
// Speech-to-speech sessions keep the audio at the session rate and label
// the transcription WAV with that rate (see s2sCommitTurn).
if rate := session.InputSampleRate; session.s2s == nil && rate > 0 && rate != localSampleRate && len(utt) > 0 {
utt = sound.Int16toBytesLE(sound.ResampleInt16(sound.BytesToInt16sLE(utt), rate, localSampleRate))
}
commitUtteranceWithTranscript(ctx, utt, nil, nil, "", session, conv, t, slot)
docs/content/features/openai-realtime.md
+1 -1
View File
@@ -589,7 +589,7 @@ ws://localhost:8080/v1/realtime?model=gpt-realtime
Audio is sent and received as raw PCM in the WebSocket messages, following the OpenAI Realtime API protocol.
Committed user audio reaches the transcription backend as 16 kHz mono WAV, whichever way the turn was committed: server-side VAD and client `input_audio_buffer.commit` both resample from the session input sample rate (for example 24 kHz) first.
In the chained pipeline, committed user audio reaches the transcription backend as 16 kHz mono WAV, whichever way the turn was committed: server-side VAD and client `input_audio_buffer.commit` both resample from the session input sample rate (for example 24 kHz) first.
### WebRTC