fix(realtime): keep VAD-commit transcription alive across barge-in, cancel it at teardown, order commits

Barge-in (new speech onset) cancels the turn's SourceVAD response context
(realtime_turncoord.go: respSink.cancel(SourceVAD)). The VAD commit body
runs under that same context, so an in-flight Whisper STT call was aborted
with 'context canceled' whenever the caller kept talking while the first
chunk was transcribing. The user's turn was lost: no transcript, no
LLM/TTS response.

v1 of this fix ran the transcription with context.WithoutCancel(ctx). The
review correctly pointed out two correctness gaps:

1. Teardown lost its cancellation. WithoutCancel detaches from every
   cancellation, so a transcription in flight at session close outlived
   the session and blocked respSink.shutdown (which joins the response
   goroutines) until the backend finished the job.
2. Out-of-order commits. Consecutive commits run in parallel goroutines,
   so a fast second transcription could append its user item before a
   slow first one: the conversation became [second, first] and the second
   response saw only [second].

Changes (core/http/endpoints/openai/):
- Session gains a session-lifetime context (sessionCtx), cancelled by
  conncoord's Teardown BEFORE respSink.shutdown joins the response
  goroutines. The transcription (and the voice-gate resolution) run under
  it: they survive barge-in (which cancels only the per-response context)
  but are cancelled with the session.
- Commit slots order the user-item appends in speech order:
  Session.nextCommitSlot() is claimed at commit issue time (VAD CommitTurn
  / client commit), a commit's item append waits on the previous slot's
  done (aborts on the session context), and every exit closes the slot so
  a failed or torn-down commit never blocks the next. Transcriptions stay
  parallel; only the appends are ordered.
- If the turn's response context was cancelled while the (detached)
  transcription ran — barge-in, superseded by a newer commit — the user
  item still commits (appendUserItem, split out of generateResponse) so
  the LLM context keeps the full user input, but no response is generated
  for the superseded turn; the newer speech triggers its own response on
  the complete history.
- Regression tests (realtime_commit_order_test.go) cover both review
  schedules — teardown during an in-flight transcription, and
  held-first/finished-second out-of-order completion — plus the
  barge-in-during-transcription item survival, driving the real commit
  path with a transcription double that honours context cancellation.
- docs/design/realtime-state-machines.md: implementation-status entry for
  the committed-turn pipeline (transcription lifetime + commit order).

Fixes #12445

Validated: builds, go vet clean, all openai specs + respcoord/turncoord/
conncoord suites pass under -race (incl. the 3 new regression specs).
Production A/B (call-center voice agent, SIP, silero-vad +
whisper-large-turbo + LLM + TTS, server_vad ~600 ms) on LocalAI v4.11.0:
unpatched — 'transcription_failed: context canceled', first part of the
utterance lost, agent answers only the remainder; patched — full
transcript committed, agent answers the complete utterance, barge-in
still cancels the in-flight assistant TTS response as intended, and
teardown cancels the in-flight transcription instead of waiting for the
backend.

Signed-off-by: nexxtmobile.de <kai@nexxtmobile.de>
This commit is contained in:
nexxtmobile.de committed 2026-10-03 16:41:11 +00:00
1 parent 372b1f8983
commit c8f991be14
8 files changed
+421 -38

No files matched your search

+37
View File
@@ -458,6 +458,43 @@ property-test oracles, and FizzBee invariants:
M5's by its existing `Closed`; the persistent coordinators (M3/M4) carry the
explicit `Terminated` state.
- **Committed-turn pipeline: transcription lifetime + commit order (issue #12445,
done).** Two cross-cutting defects in the VAD commit path, neither of which a
single machine owned:
- *Transcription lifetime.* `commitUtteranceWithTranscript` ran the
utterance transcription under the turn's **response** context (M3). A
barge-in (new speech onset) or a superseding commit cancels that context
while Whisper is still in flight, so the in-flight transcription died
("transcription_failed: context canceled") and the user's input was lost
from the conversation — the next response answered the second half of a
two-part utterance. Detaching with `context.WithoutCancel` fixed the
barge-in but broke teardown: the transcription then outlived the session,
and `respSink.shutdown` (which joins the response goroutines) blocked until
the backend finished the job. Fix: the transcription (and the voice-gate
resolution) now run under a **session-lifetime context**
(`Session.sessionCtx`) — cancelled at teardown by `conncoord`'s `Teardown`
*before* `respSink.shutdown()` joins, untouched by barge-in. The
turn's response context still cancels the *response*: when it was
cancelled during the (now detached) transcription, the user item is
committed (`appendUserItem`, split out of `generateResponse`) but no
response is generated for the superseded turn — the newer speech triggers
its own response on the complete history.
- *Commit order.* Consecutive commits run in parallel goroutines (M3 spawns
one per `issue`), so a fast second transcription could append its user
item before a slow first one — the conversation became
`[second, first]` and the second response saw only `[second]`. Fix: a
per-session **commit slot chain** (`Session.nextCommitSlot`, claimed at
commit *issue* time so slot order == speech order). A commit's user-item
append waits on the previous slot's `done` (aborts on the session
context); every exit closes its own slot, so a failed or torn-down commit
never blocks the next. Transcriptions stay parallel; only the item appends
are ordered.
Regression tests: `realtime_commit_order_test.go` (both review schedules —
teardown during an in-flight transcription; held-first/finished-second
out-of-order completion — plus the barge-in-during-transcription item
survival), driving the real commit path with a transcription double that
honours context cancellation. Verified: builds, openai specs under `-race`.
## Part 5 — Library vs hand-rolled (Go ecosystem, verified 2026-06)
Researched against live GitHub/pkg.go.dev data. **Verdict: hand-roll a typed transition