Files
LocalAI/scripts/install-fizzbee.sh
Richard Palethorpe 5d0c43ec6e feat(realtime): Semantic VAD EOU token (#10444)
* feat(realtime): EOU-driven semantic_vad turn detection

Add a `semantic_vad` turn-detection mode to the realtime API that feeds
the transcription model live and decides "the user finished speaking"
from the `<EOU>` end-of-utterance token rather than from silence alone.
When EOU fires the turn commits immediately (~0.3s); otherwise it falls
back to an eagerness-scaled silence threshold (low/med/high = 8/4/2s).

Plumbing, bottom to top:

- proto: `AudioTranscriptionLive` bidirectional RPC (config-first oneof,
  mono float PCM @16k, ready-ack / Unimplemented degrade signal) plus
  `TranscriptResult.eou` for the unary retranscribe gate.
- pkg/grpc: client/server/base/embed scaffolding for the bidi stream,
  modeled on AudioTransformStream; release stream conns on terminal Recv.
- parakeet-cpp: live transcription RPC with per-C-call engine locking
  (one live stream per turn, finalize+free at commit); bump parakeet.cpp
  to ABI v5 — incremental StreamingMel (no more quadratic per-feed mel
  recompute that delayed EOU on long turns) and the <EOU>/<EOB> split;
  strip the literal <EOU>/<EOB> from offline text and set Eou.
- core/backend: LiveTranscriptionSession wrapper + pipeline
  `turn_detection:` config block (type/eagerness/retranscribe).
- realtime: semantic_vad integration — live input captions streamed as
  transcription deltas while the user speaks, EOU-immediate commit with
  eagerness fallback, optional retranscribe gate (batch re-decode must
  also end in <EOU> to confirm), clause synthesis off the LLM token
  callback, and per-turn live-transcription / model_load telemetry.
- UI: show the realtime pipeline components as a vertical list.

Docs and tests included; opt-in via the pipeline YAML or per-session
`session.update`. Non-streaming STT backends degrade to silence-only.

Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Write] [Bash]
Assisted-by: Claude Code:claude-fable-5 [Read] [Edit] [Bash]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): explicit formally-verified state machines + parakeet streaming driver

The realtime API had several implicit state machines whose state was inferred
from scattered booleans, channels, and five separate mutexes, leaving
illegal/inconsistent states reachable. Make them explicit and keep the
implementation in step with a formal design; rework the parakeet streaming
backend along the same lines.

Realtime state machines (M1-M5). Each is a sealed sum-type State/Event/Effect
with a total, pure Next(state,event)->(state,[]effect) behind a single-writer
Coordinator:

  M1 conncoord    connection lifecycle: VAD toggle + once-only teardown
                  (replaces vadServerStarted + a `done` channel closed from
                  two sites).
  M2 turncoord    turn detection: collapses speechStarted and the live-stream
                  "turn open" flag into one state, so discardTurn can no longer
                  desync them and suppress the next onset.
  M3 respcoord    response coordination: serializes the dual-writer
                  start/cancel so at most one response is live; one
                  response.done per response.create.
  M4 compactcoord conversation compaction: single-flight (replaces the
                  `compacting atomic.Bool` CAS).
  M5 ttscoord     TTS pipeline: open->closing->closed, idempotent wait(),
                  rejects enqueue-after-close (was a silent drop).

The Coordinator/Sink/Next plumbing — only the sealed types and Next differed
per machine — is extracted once into core/http/endpoints/openai/coordinator as
a generic Coordinator[S,E,F]; each machine keeps its public API via type
aliases, so no sink, call-site, or test moved.

Hierarchy. session_lifecycle.fizz models M1 as the parent region with its
children (M2/M3/M4) as one statechart and asserts ChildrenDieWithParent (conn
torn => all children terminal, none start after teardown). respcoord and
compactcoord gain an absorbing Terminated state + Shutdown event; conncoord's
teardown drives the children terminal. This closes a compaction teardown gap: a
fire-and-forget compaction could outlive a torn session — compactionSink now
takes a session-scoped cancellable context + WaitGroup and joins the in-flight
summarize+evict on shutdown.

Formal verification. formal-verification/ holds one authoritative FizzBee spec
per machine plus the composition spec, each with an always-assertion and a
documented one-line edit that makes the checker fail (verified non-vacuous).
scripts/realtime-conformance.sh is fail-closed: all Go conformance suites under
-race AND a model-check of every .fizz spec; a missing FizzBee is a hard error
(only the loud REALTIME_CONFORMANCE_SKIP_FIZZBEE=1 bypasses it, never in CI).
FizzBee is pinned by sha256 and installed via scripts/install-fizzbee.sh into
.tools/ (gitignored). Wired as make test-realtime-conformance, a CI workflow,
and a pre-commit path filter. Go conformance tests are Ginkgo/Gomega (per the
repo's forbidigo lint): transition tables + fixed-seed property walks +
concurrent/-race specs, no rapid dependency. Design map:
docs/design/realtime-state-machines.md.

Parakeet streaming backend. The same treatment applied to the parakeet-cpp
streaming paths:
- AudioTranscriptionStream returns codes.Unimplemented for non-streaming models
  instead of decoding offline and emitting it as one delta + final. A client
  that asked for streaming learns the model cannot stream rather than receiving
  a batch result shaped like a stream. New grpcerrors.StreamTranscriptionUnsupported
  carries that signal; the HTTP /v1/audio/transcriptions stream path surfaces it
  as an SSE error event. Mirrors AudioTranscriptionLive, which already did this.
- utteranceBoundary (boundary.go): a single definition of the end-of-utterance
  latch, replacing three open-coded finalEou toggles. Modelled as a two-valued
  type so illegal states are unrepresentable.
- Shared decode driver (driver.go): streamFeedResult (one per-feed event) +
  feedChunk (hides the ABI v4 JSON vs text-only split) + feedSlices + flushTail.
  The feed loop is written once.
- AudioTranscriptionLive becomes a bidi adapter: it streams the per-feed
  {delta,eou,eob,words} the realtime turn detector consumes and a terminal
  FinalResult carrying only Text. Segments/duration/eou are offline-only and no
  longer produced (nor read) on the live path; liveTraceState drops the terminal
  eou and keeps the per-feed eou_events count.
- AudioTranscriptionStream + streamJSON merge into one driver-based function;
  streamSegmenter is generalized to the unified event with a text-only fallback
  that preserves the legacy (no-words) library's per-utterance segmentation.

Verified: build/vet/gofumpt clean, golangci-lint 0 issues, all coordinator and
parakeet packages under -race, the fail-closed conformance gate green, and
make test-realtime (12 e2e WS+WebRTC).

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
2026-06-30 09:01:22 +02:00

97 lines
3.5 KiB
Bash
Executable File

#!/bin/sh
# install-fizzbee: download the pinned FizzBee release and verify its checksum.
# FizzBee is pre-1.0 and single-maintainer, so we PIN a version + sha256 rather
# than tracking latest or building from source (its primary build is Bazel).
# See formal-verification/README.md.
#
# Usage:
# scripts/install-fizzbee.sh [dest_dir] (default dest: ./.tools/fizzbee)
#
# First-time pinning: run once with no recorded checksum; the script prints the
# computed sha256. Record it in formal-verification/fizzbee.sha256 as
# "<sha256> <asset>" and commit, then CI verifies it on every run.
set -eu
VERSION=${FIZZBEE_VERSION:-v0.5.2}
DEST=${1:-".tools/fizzbee"}
ROOT=$(CDPATH= cd "$(dirname "$0")/.." && pwd)
SHA_FILE="$ROOT/formal-verification/fizzbee.sha256"
# Detect platform -> release asset name.
os=$(uname -s)
arch=$(uname -m)
case "$os" in
Linux) plat="linux" ;;
Darwin) plat="macos" ;;
*) echo "unsupported OS: $os" >&2; exit 1 ;;
esac
case "$arch" in
x86_64|amd64) cpu="x86" ;;
arm64|aarch64) cpu="arm" ;;
*) echo "unsupported arch: $arch" >&2; exit 1 ;;
esac
asset="fizzbee-${VERSION}-${plat}_${cpu}.tar.gz"
url="https://github.com/fizzbee-io/fizzbee/releases/download/${VERSION}/${asset}"
inner="fizzbee-${VERSION}-${plat}_${cpu}"
# Idempotent: if the pinned version is already extracted (e.g. restored from a
# CI cache), do nothing. This keeps the install step a no-op on cache hits and
# avoids re-downloading the (large) bundle.
if [ -x "$DEST/$inner/fizz" ] && [ -L "$DEST/fizz" ]; then
echo "==> FizzBee $VERSION already installed at $DEST/$inner"
exit 0
fi
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
echo "==> downloading $url"
curl -fL -o "$tmp/$asset" "$url"
got=$(sha256sum "$tmp/$asset" | awk '{print $1}')
if [ -f "$SHA_FILE" ]; then
want=$(awk -v a="$asset" '$2==a {print $1}' "$SHA_FILE")
if [ -z "$want" ]; then
echo "ERROR: no recorded sha256 for $asset in $SHA_FILE" >&2
echo " computed: $got $asset" >&2
exit 1
fi
if [ "$got" != "$want" ]; then
echo "ERROR: sha256 mismatch for $asset" >&2
echo " want: $want" >&2
echo " got: $got" >&2
exit 1
fi
echo "==> checksum verified"
else
echo "WARNING: $SHA_FILE not found -- record this line and commit it:" >&2
echo "$got $asset"
fi
# The tarball unpacks to a self-contained, version+platform-named directory:
# fizzbee-<version>-<plat>_<cpu>/
# fizz <- the CLI wrapper (entrypoint; invoke THIS)
# parser/parser_bin <- the .fizz frontend (bundled; no system Python needed)
# fizzbee <- the Go model-checker binary
# fizz.env <- resolves the above RELATIVE to the dir holding `fizz`
# mbt_gen.zip <- MBT generator (only this needs system python)
# The whole directory must stay intact (fizz.env is path-relative), so we keep
# it and publish a STABLE symlink `$DEST/fizz` -> the versioned wrapper. The
# wrapper does readlink -f on itself, so the symlink still resolves fizz.env.
mkdir -p "$DEST"
rm -rf "$DEST/$inner"
tar -xzf "$tmp/$asset" -C "$DEST"
if [ ! -x "$DEST/$inner/fizz" ]; then
echo "ERROR: expected $DEST/$inner/fizz after extraction; tarball layout changed" >&2
exit 1
fi
ln -sfn "$inner/fizz" "$DEST/fizz"
echo "==> installed FizzBee $VERSION to $DEST/$inner"
echo " frontend: $DEST/$inner/parser/parser_bin"
echo " entrypoint: $DEST/fizz (stable symlink -> $inner/fizz)"
echo
echo "The realtime-conformance gate auto-detects \$DEST/fizz; nothing else needed."
echo "To use it directly, run: $DEST/fizz <spec.fizz>"