mirror of
https://github.com/mudler/LocalAI.git
synced 2026-11-06 04:23:15 -05:00
nexxtmobile.deandClaude Opus 5.5
da0c5435b5
qwentts.cpp batches concurrent qt_synthesize calls on one context when it is initialized with max_batch > 1 (its compute worker coalesces them into batched decode steps; qt_synthesize stays blocking and thread safe). The LocalAI wrapper never set max_batch and serialized every request with base.SingleThread, so concurrent /v1/audio/speech requests always ran one after another. - new model option `max_batch:N` (alias `parallel:N`, default 1 = current behavior), passed to qt_init_params.max_batch via qt3_load - Locking() keeps LocalAI's per-backend request lock only for max_batch 1; with batching TTS/TTSStream reach the native worker concurrently - TTSStream routes chunks per request: a fresh id travels as the C callback's user_data (one shared purego callback, as before), each request has its own queue drained into its channel, so the native worker never blocks on a slow consumer and concurrent streams cannot mix - TTSLive rejects max_batch > 1 (qwentts incremental mode requires a max_batch=1 context, docs/INCREMENTAL.md) - README: option, measurements, TTSLive limit, ROCm/gfx1201 hipBLASLt note - unit tests: option parsing, Locking, concurrent stream routing (fails if requests were serialized), TTSLive rejection Measured with qwentts' own tts-server and the 1.7B Base Q4_K_M talker, four concurrent streaming requests, time to first PCM: RTX 5060 Ti (CUDA) 0.02/0.36/0.70/1.04 s at max_batch 1 vs 0.06 s for all four at 4; Radeon AI PRO R9700 (ROCm, with ROCBLAS_USE_HIPBLASLT=0) 0.03/0.71/1.36/ 2.02 s vs <= 0.11 s. Signed-off-by: nexxtmobile.de <git@nexxtmobile.de> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>