From 074bc84f0c0c522ffd628a6be0dfdf37aca2fd07 Mon Sep 17 00:00:00 2001 From: "nexxtmobile.de" Date: Sun, 25 Oct 2026 12:39:23 +0000 Subject: [PATCH] docs(text-to-audio): document qwen3-tts-cpp max_batch Signed-off-by: nexxtmobile.de Co-Authored-By: Claude Opus 5.5 --- backend/go/qwen3-tts-cpp/README.md | 7 ++++--- docs/content/features/text-to-audio.md | 13 +++++++++++++ 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/backend/go/qwen3-tts-cpp/README.md b/backend/go/qwen3-tts-cpp/README.md index cf137e6579..39b881c04f 100644 --- a/backend/go/qwen3-tts-cpp/README.md +++ b/backend/go/qwen3-tts-cpp/README.md @@ -76,9 +76,10 @@ requests (time to the first PCM chunk): RTX 5060 Ti 0.02/0.36/0.70/1.04 s with - Incremental synthesis (`TTSLive`, the realtime pipeline with `streaming.tts: true`) requires `max_batch:1` and is rejected otherwise (qwentts `docs/INCREMENTAL.md`). -- ROCm on gfx1201 (RDNA4) hangs the GPU in hipBLASLt for batched matmuls; - `env: {ROCBLAS_USE_HIPBLASLT: "0"}` in the model config routes them to - rocBLAS. CUDA needs no workaround. +- ROCm before 10.1 on gfx1201 (RDNA4) hangs the GPU in hipBLASLt for the + batched matmuls (fixed upstream by ROCm/rocm-libraries#8909; ROCm 10.1 runs + clean). On older stacks `env: {ROCBLAS_USE_HIPBLASLT: "0"}` in the model + config routes them to rocBLAS. CUDA needs no workaround. Request precedence is: a request `voice`, then `tts.voice`, then `tts.audio_path`. A saved profile supplies its private WAV and exact transcript diff --git a/docs/content/features/text-to-audio.md b/docs/content/features/text-to-audio.md index 180d4eaeab..ca113a2496 100644 --- a/docs/content/features/text-to-audio.md +++ b/docs/content/features/text-to-audio.md @@ -740,6 +740,19 @@ tts: audio_path: voices/default-reference.wav # optional fallback ``` +##### Concurrent requests + +By default `qwen3-tts-cpp` synthesizes one request at a time. Set `max_batch` to let qwentts.cpp batch up to N concurrent `/v1/audio/speech` requests (streaming or not) into shared decode steps; further requests queue inside the engine: + +```yaml +options: + - max_batch:4 +``` + +With four concurrent streaming requests on an RTX 5060 Ti (1.7B Base Q4_K_M), the first audio arrives after 0.06 seconds for all four instead of 0.02, 0.36, 0.70 and 1.04 seconds one after another. VRAM grows with `max_batch` (roughly 2.6 GB at 1, 6 GB at 4, 10.5 GB at 8 for that talker). At load the backend warms up every batch width so the first concurrent requests do not stall; `max_batch_warmup:false` skips this. + +Incremental synthesis (`TTSLive`, used by a realtime pipeline with `streaming.tts: true`) requires `max_batch:1`. On ROCm before 10.1 with RDNA4 (gfx1201), batched requests can hang the GPU in hipBLASLt (fixed in ROCm 10.1 by ROCm/rocm-libraries#8909); on those stacks add `env: {ROCBLAS_USE_HIPBLASLT: "0"}` to the model configuration. + #### llama.cpp gallery variants llama.cpp gained native Qwen3-TTS support in [ggml-org/llama.cpp#26254](https://github.com/ggml-org/llama.cpp/pull/26254), so the `llama-cpp` backend can serve it on the same accelerator matrix it already uses for text generation: CUDA, ROCm, SYCL, Vulkan and Metal.