qwentts.cpp batches concurrent qt_synthesize calls on one context when it
is initialized with max_batch > 1 (its compute worker coalesces them into
batched decode steps; qt_synthesize stays blocking and thread safe). The
LocalAI wrapper never set max_batch and serialized every request with
base.SingleThread, so concurrent /v1/audio/speech requests always ran one
after another.
- new model option `max_batch:N` (alias `parallel:N`, default 1 = current
behavior), passed to qt_init_params.max_batch via qt3_load
- Locking() keeps LocalAI's per-backend request lock only for max_batch 1;
with batching TTS/TTSStream reach the native worker concurrently
- TTSStream routes chunks per request: a fresh id travels as the C
callback's user_data (one shared purego callback, as before), each request
has its own queue drained into its channel, so the native worker never
blocks on a slow consumer and concurrent streams cannot mix
- TTSLive rejects max_batch > 1 (qwentts incremental mode requires a
max_batch=1 context, docs/INCREMENTAL.md)
- README: option, measurements, TTSLive limit, ROCm/gfx1201 hipBLASLt note
- unit tests: option parsing, Locking, concurrent stream routing (fails if
requests were serialized), TTSLive rejection
Measured with qwentts' own tts-server and the 1.7B Base Q4_K_M talker,
four concurrent streaming requests, time to first PCM: RTX 5060 Ti (CUDA)
0.02/0.36/0.70/1.04 s at max_batch 1 vs 0.06 s for all four at 4;
Radeon AI PRO R9700 (ROCm, with ROCBLAS_USE_HIPBLASLT=0) 0.03/0.71/1.36/
2.02 s vs <= 0.11 s.
Signed-off-by: nexxtmobile.de <git@nexxtmobile.de>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(tts): stream Qwen text through audio pipelines
Feed cleaned LLM deltas into one persistent Qwen synthesis session.
Return audio before input ends and preserve the live RPC through model
cache, staging and in-flight wrappers. Stop generation on sink failures
or cancellation without replaying speech through a fallback.
Pin the incremental native fork and verify early audio, late text,
finalization and cancellation with lifecycle and real-model checks.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* build(tts): use the organization Qwen fork
Fetch the reviewed incremental synthesis commit from localai-org so
backend builds use the organization-maintained fork.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add a Qwen streaming assistant preset
Expose an English pipeline using existing VAD, ASR, LLM and named-voice
TTS models. Enable incremental text and audio delivery, and document
separate component installs because the gallery has no dependency
installer.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): use Parakeet TDT v3 for the assistant
Select the existing Parakeet TDT 0.6B v3 transcription entry and update
its prerequisite install command. Keep the preset's English voice and
incremental Qwen synthesis configuration.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(kokoros): reject unsupported backend RPCs
Implement the MotionStream and SlotCache methods required by the shared
protobuf service. Return Unimplemented so Kokoros compiles without
advertising support for either operation.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(tts): mark cleanup and logging results ignored
Explicitly ignore connection shutdown, test server shutdown and
Ginkgo diagnostic writer errors. These operations do not replace the
stream's inference or sink errors, and errcheck requires their return
values to be acknowledged.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(tts): stream buffered clauses in realtime pipelines
Start clause synthesis while LLM text is still arriving when both streaming flags are enabled. Keep explicit clause opt-out and native Qwen sessions, bound automatic queued work, and choose streaming support on resolved failover targets.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(tts): bound synthetic voice clone generation
The sine-wave reference checks speaker conditioning and WAV output, but does not provide a meaningful speech EOS target. Limit that smoke request to 128 frames and verify the request limit reaches native synthesis. Leave production sampling defaults unchanged.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(tts): require explicit clause buffering opt-in
Keep whole-reply synthesis for nonnative engines when clause_chunking is unset or false. Use the bounded cancellable worker when explicitly enabled, with unary clauses when TTS streaming is disabled. Preserve native Qwen precedence for every clause setting.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add VITS and Piper streaming assistants
Provide comparable full pipeline configurations with shared VAD, Parakeet transcription, and Qwen chat components. Enable clause buffering explicitly and document the separate prerequisite model installs and Talk selection.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
