mirror of
https://github.com/mudler/LocalAI.git
synced 2026-11-06 04:23:15 -05:00
nexxtmobile.de
9eb1687bcf
qwentts.cpp batches concurrent qt_synthesize calls on one context when it
is initialized with max_batch > 1 (its compute worker coalesces them into
batched decode steps; qt_synthesize stays blocking and thread safe). The
LocalAI wrapper never set max_batch and serialized every request with
base.SingleThread, so concurrent /v1/audio/speech requests always ran one
after another.
- new model option `max_batch:N` (alias `parallel:N`, default 1 = current
behavior), passed to qt_init_params.max_batch via qt3_load
- Locking() keeps LocalAI's per-backend request lock only for max_batch 1;
with batching TTS/TTSStream reach the native worker concurrently
- TTSStream routes chunks per request: a fresh id travels as the C
callback's user_data (one shared purego callback, as before), each request
has its own queue drained into its channel, so the native worker never
blocks on a slow consumer and concurrent streams cannot mix
- TTSLive rejects max_batch > 1 (qwentts incremental mode requires a
max_batch=1 context, docs/INCREMENTAL.md)
- README: option, measurements, TTSLive limit, ROCm/gfx1201 hipBLASLt note
- unit tests: option parsing, Locking, concurrent stream routing (fails if
requests were serialized), TTSLive rejection
Measured with qwentts' own tts-server and the 1.7B Base Q4_K_M talker,
four concurrent streaming requests, time to first PCM: RTX 5060 Ti (CUDA)
0.02/0.36/0.70/1.04 s at max_batch 1 vs 0.06 s for all four at 4;
Radeon AI PRO R9700 (ROCm, with ROCBLAS_USE_HIPBLASLT=0) 0.03/0.71/1.36/
2.02 s vs <= 0.11 s.
Signed-off-by: nexxtmobile.de <git@nexxtmobile.de>
Assisted-by: Claude:claude-opus-5-5
(cherry picked from commit da0c5435b5)