mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-10 07:47:29 -04:00
nexxtmobile.deandnexxtmobile.de
4d681b7f6d
* fix(realtime): keep VAD-commit transcription alive across barge-in, cancel it at teardown, order commits
Barge-in (new speech onset) cancels the turn's SourceVAD response context
(realtime_turncoord.go: respSink.cancel(SourceVAD)). The VAD commit body
runs under that same context, so an in-flight Whisper STT call was aborted
with 'context canceled' whenever the caller kept talking while the first
chunk was transcribing. The user's turn was lost: no transcript, no
LLM/TTS response.
v1 of this fix ran the transcription with context.WithoutCancel(ctx). The
review correctly pointed out two correctness gaps:
1. Teardown lost its cancellation. WithoutCancel detaches from every
cancellation, so a transcription in flight at session close outlived
the session and blocked respSink.shutdown (which joins the response
goroutines) until the backend finished the job.
2. Out-of-order commits. Consecutive commits run in parallel goroutines,
so a fast second transcription could append its user item before a
slow first one: the conversation became [second, first] and the second
response saw only [second].
Changes (core/http/endpoints/openai/):
- Session gains a session-lifetime context (sessionCtx), cancelled by
conncoord's Teardown BEFORE respSink.shutdown joins the response
goroutines. The transcription (and the voice-gate resolution) run under
it: they survive barge-in (which cancels only the per-response context)
but are cancelled with the session.
- Commit slots order the user-item appends in speech order:
Session.nextCommitSlot() is claimed at commit issue time (VAD CommitTurn
/ client commit), a commit's item append waits on the previous slot's
done (aborts on the session context), and every exit closes the slot so
a failed or torn-down commit never blocks the next. Transcriptions stay
parallel; only the appends are ordered.
- If the turn's response context was cancelled while the (detached)
transcription ran — barge-in, superseded by a newer commit — the user
item still commits (appendUserItem, split out of generateResponse) so
the LLM context keeps the full user input, but no response is generated
for the superseded turn; the newer speech triggers its own response on
the complete history.
- Regression tests (realtime_commit_order_test.go) cover both review
schedules — teardown during an in-flight transcription, and
held-first/finished-second out-of-order completion — plus the
barge-in-during-transcription item survival, driving the real commit
path with a transcription double that honours context cancellation.
- docs/design/realtime-state-machines.md: implementation-status entry for
the committed-turn pipeline (transcription lifetime + commit order).
Fixes #12445
Validated: builds, go vet clean, all openai specs + respcoord/turncoord/
conncoord suites pass under -race (incl. the 3 new regression specs).
Production A/B (call-center voice agent, SIP, silero-vad +
whisper-large-turbo + LLM + TTS, server_vad ~600 ms) on LocalAI v4.11.0:
unpatched — 'transcription_failed: context canceled', first part of the
utterance lost, agent answers only the remainder; patched — full
transcript committed, agent answers the complete utterance, barge-in
still cancels the in-flight assistant TTS response as intended, and
teardown cancels the in-flight transcription instead of waiting for the
backend.
Signed-off-by: nexxtmobile.de <kai@nexxtmobile.de>
* fix(realtime): release commit slots in order on every exit; share slot+issue boundary
Follow-up to the review of c8f991be (issue #12445): two schedules still
broke commit ordering.
1. A failed middle commit released later turns before earlier turns
finished. slot.done closed on every return, but the wait on
slot.prevDone happened only on the successful nonempty-transcript
path. Hold transcription A, let B fail (or return an empty
transcript / be rejected by the voice gate), then complete C: B
closed its channel without waiting for A, so C appended and started
its response without A (response history [third], final history
[third first]). Fix: the slot now releases (done closes) only AFTER
the predecessor has finished — on EVERY exit path, including errors,
empty transcripts, gate rejections and teardown (the session context
can still stop the wait, so teardown never blocks on a
never-finishing predecessor). The success path keeps its append gate
(wait before appending the user item); the deferred release gate
enforces the same order on every other exit.
2. Slot order and response issue order could disagree between the two
producers. The VAD CommitTurn and the client
input_audio_buffer.commit reserved the slot and called
respSink.issue separately; a pause between the two let the other
producer reserve AND issue first, so the later issue superseded the
EARLIER turn's response (response history [first], final history
[first second], second turn un-answered). Fix: both producers now go
through Session.issueCommit, which claims the slot and issues the
body under one lock (commitOrderMu) — slot order == issue order.
respSink.issue is non-blocking, so the lock never stalls
VAD/barge-in handling.
Regression tests (realtime_commit_order_test.go) now drive the REAL
issue path — Session.issueCommit into the real responseSink/respcoord,
so coordinator supersession and the spawned response goroutines are
exercised — and cover: teardown during an in-flight transcription;
held-first/finished-second out-of-order completion; barge-in
(respSink.cancel) item survival; a FAILED middle commit; an EMPTY
middle commit; interleaved VAD/client producers in both directions.
The failed/empty middle specs fail deterministically without the
release gate (verified against the pre-fix code).
docs/design/realtime-state-machines.md: implementation-status entry
updated (append gate + release gate + shared issue boundary).
Validated: builds, go vet clean, all openai specs + respcoord/turncoord/
conncoord suites pass under -race (476 specs, incl. the 4 new ones).
Fixes #12445
Signed-off-by: nexxtmobile.de <kai@nexxtmobile.de>
---------
Signed-off-by: nexxtmobile.de <kai@nexxtmobile.de>
Co-authored-by: nexxtmobile.de <kai@nexxtmobile.de>
LocalAI website
LocalAI documentation website
Requirement
In this project, the Docsy theme component is pulled in as a Hugo module, together with other module dependencies:
$ hugo mod graph
hugo: collected modules in 566 ms
hugo: collected modules in 578 ms
github.com/google/docsy-example github.com/google/docsy@v0.5.1-0.20221017155306-99eacb09ffb0
github.com/google/docsy-example github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1
github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1 github.com/twbs/bootstrap@v4.6.2+incompatible
github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1 github.com/FortAwesome/Font-Awesome@v0.0.0-20220831210243-d3a7818c253f
If you want to do SCSS edits and want to publish these, you need to install PostCSS
npm install
Running the website locally
Building and running the site locally requires a recent extended version of Hugo.
You can find out more about how to install Hugo for your environment in our
Getting started guide.
From the LocalAI repository root, run:
make docs
The Hugo configuration lives in the docs directory. To invoke Hugo
directly instead, run:
cd docs
hugo server
Running a container locally
You can run docsy-example inside a Docker
container, the container runs with a volume bound to the docsy-example
folder. This approach doesn't require you to install any dependencies other
than Docker Desktop on
Windows and Mac, and Docker Compose
on Linux.
-
Build the docker image
docker-compose build -
Run the built image
docker-compose upNOTE: You can run both commands at once with
docker-compose up --build. -
Verify that the service is working.
Open your web browser and type
http://localhost:1313in your navigation bar, This opens a local instance of the docsy-example homepage. You can now make changes to the docsy example and those changes will immediately show up in your browser after you save.
Cleanup
To stop Docker Compose, on your terminal window, press Ctrl + C.
To remove the produced images run:
docker-compose rm
For more information see the Docker Compose documentation.
Troubleshooting
As you run the website locally, you may run into the following error:
➜ hugo server
INFO 2021/01/21 21:07:55 Using config file:
Building sites … INFO 2021/01/21 21:07:55 syncing static files to /
Built in 288 ms
Error: Error building site: TOCSS: failed to transform "scss/main.scss" (text/x-scss): resource "scss/scss/main.scss_9fadf33d895a46083cdd64396b57ef68" not found in file cache
This error occurs if you have not installed the extended version of Hugo. See this section of the user guide for instructions on how to install Hugo.
Or you may encounter the following error:
➜ hugo server
Error: failed to download modules: binary with name "go" not found
This error occurs if you have not installed the go programming language on your system.
See this section of the user guide for instructions on how to install go.