mirror of
https://github.com/mudler/LocalAI.git
synced 2026-11-06 12:33:16 -05:00
docs(text-to-audio): document qwen3-tts-cpp max_batch
Signed-off-by: nexxtmobile.de <git@nexxtmobile.de> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0b7d0b40ed
commit
074bc84f0c
2 files changed
+17
-3
No files matched your search
@@ -76,9 +76,10 @@ requests (time to the first PCM chunk): RTX 5060 Ti 0.02/0.36/0.70/1.04 s with
|
||||
- Incremental synthesis (`TTSLive`, the realtime pipeline with
|
||||
`streaming.tts: true`) requires `max_batch:1` and is rejected otherwise
|
||||
(qwentts `docs/INCREMENTAL.md`).
|
||||
- ROCm on gfx1201 (RDNA4) hangs the GPU in hipBLASLt for batched matmuls;
|
||||
`env: {ROCBLAS_USE_HIPBLASLT: "0"}` in the model config routes them to
|
||||
rocBLAS. CUDA needs no workaround.
|
||||
- ROCm before 10.1 on gfx1201 (RDNA4) hangs the GPU in hipBLASLt for the
|
||||
batched matmuls (fixed upstream by ROCm/rocm-libraries#8909; ROCm 10.1 runs
|
||||
clean). On older stacks `env: {ROCBLAS_USE_HIPBLASLT: "0"}` in the model
|
||||
config routes them to rocBLAS. CUDA needs no workaround.
|
||||
|
||||
Request precedence is: a request `voice`, then `tts.voice`, then
|
||||
`tts.audio_path`. A saved profile supplies its private WAV and exact transcript
|
||||
|
||||
@@ -740,6 +740,19 @@ tts:
|
||||
audio_path: voices/default-reference.wav # optional fallback
|
||||
```
|
||||
|
||||
##### Concurrent requests
|
||||
|
||||
By default `qwen3-tts-cpp` synthesizes one request at a time. Set `max_batch` to let qwentts.cpp batch up to N concurrent `/v1/audio/speech` requests (streaming or not) into shared decode steps; further requests queue inside the engine:
|
||||
|
||||
```yaml
|
||||
options:
|
||||
- max_batch:4
|
||||
```
|
||||
|
||||
With four concurrent streaming requests on an RTX 5060 Ti (1.7B Base Q4_K_M), the first audio arrives after 0.06 seconds for all four instead of 0.02, 0.36, 0.70 and 1.04 seconds one after another. VRAM grows with `max_batch` (roughly 2.6 GB at 1, 6 GB at 4, 10.5 GB at 8 for that talker). At load the backend warms up every batch width so the first concurrent requests do not stall; `max_batch_warmup:false` skips this.
|
||||
|
||||
Incremental synthesis (`TTSLive`, used by a realtime pipeline with `streaming.tts: true`) requires `max_batch:1`. On ROCm before 10.1 with RDNA4 (gfx1201), batched requests can hang the GPU in hipBLASLt (fixed in ROCm 10.1 by ROCm/rocm-libraries#8909); on those stacks add `env: {ROCBLAS_USE_HIPBLASLT: "0"}` to the model configuration.
|
||||
|
||||
#### llama.cpp gallery variants
|
||||
|
||||
llama.cpp gained native Qwen3-TTS support in [ggml-org/llama.cpp#26254](https://github.com/ggml-org/llama.cpp/pull/26254), so the `llama-cpp` backend can serve it on the same accelerator matrix it already uses for text generation: CUDA, ROCm, SYCL, Vulkan and Metal.
|
||||
|
||||
Reference in new issue
Block a user