docs(text-to-audio): document qwen3-tts-cpp max_batch

Signed-off-by: nexxtmobile.de <git@nexxtmobile.de>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
nexxtmobile.deandClaude Opus 5.5 committed 2026-10-25 12:55:18 +00:00
1 parent 0b7d0b40ed
commit 074bc84f0c
2 files changed
+17 -3

No files matched your search

+4 -3
View File
@@ -76,9 +76,10 @@ requests (time to the first PCM chunk): RTX 5060 Ti 0.02/0.36/0.70/1.04 s with
- Incremental synthesis (`TTSLive`, the realtime pipeline with
`streaming.tts: true`) requires `max_batch:1` and is rejected otherwise
(qwentts `docs/INCREMENTAL.md`).
- ROCm on gfx1201 (RDNA4) hangs the GPU in hipBLASLt for batched matmuls;
`env: {ROCBLAS_USE_HIPBLASLT: "0"}` in the model config routes them to
rocBLAS. CUDA needs no workaround.
- ROCm before 10.1 on gfx1201 (RDNA4) hangs the GPU in hipBLASLt for the
batched matmuls (fixed upstream by ROCm/rocm-libraries#8909; ROCm 10.1 runs
clean). On older stacks `env: {ROCBLAS_USE_HIPBLASLT: "0"}` in the model
config routes them to rocBLAS. CUDA needs no workaround.
Request precedence is: a request `voice`, then `tts.voice`, then
`tts.audio_path`. A saved profile supplies its private WAV and exact transcript
+13
View File
@@ -740,6 +740,19 @@ tts:
audio_path: voices/default-reference.wav # optional fallback
```
##### Concurrent requests
By default `qwen3-tts-cpp` synthesizes one request at a time. Set `max_batch` to let qwentts.cpp batch up to N concurrent `/v1/audio/speech` requests (streaming or not) into shared decode steps; further requests queue inside the engine:
```yaml
options:
- max_batch:4
```
With four concurrent streaming requests on an RTX 5060 Ti (1.7B Base Q4_K_M), the first audio arrives after 0.06 seconds for all four instead of 0.02, 0.36, 0.70 and 1.04 seconds one after another. VRAM grows with `max_batch` (roughly 2.6 GB at 1, 6 GB at 4, 10.5 GB at 8 for that talker). At load the backend warms up every batch width so the first concurrent requests do not stall; `max_batch_warmup:false` skips this.
Incremental synthesis (`TTSLive`, used by a realtime pipeline with `streaming.tts: true`) requires `max_batch:1`. On ROCm before 10.1 with RDNA4 (gfx1201), batched requests can hang the GPU in hipBLASLt (fixed in ROCm 10.1 by ROCm/rocm-libraries#8909); on those stacks add `env: {ROCBLAS_USE_HIPBLASLT: "0"}` to the model configuration.
#### llama.cpp gallery variants
llama.cpp gained native Qwen3-TTS support in [ggml-org/llama.cpp#26254](https://github.com/ggml-org/llama.cpp/pull/26254), so the `llama-cpp` backend can serve it on the same accelerator matrix it already uses for text generation: CUDA, ROCm, SYCL, Vulkan and Metal.