go vet reported "possible misuse of unsafe.Pointer" for the uintptr that
the C library returns, and the //nolint:govet comment does not silence
vet itself. The line dates from the move to qwentts.cpp, not from the
batching work.
It is not a bug: the buffer is malloc'd by the C side, so the Go GC
neither moves nor frees it, and it is copied out before CppPCMFree runs.
Read the address through a pointer to the variable instead of converting
the uintptr, which is the form vet accepts and gives the same address.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
The spec only checks that warmupBatch returns once every request ends;
errcheck flags the dropped error value.
Assisted-by: Claude:claude-sonnet-5-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Report failed warmup requests without failing model load. Document the
warmup limits and use the reported LocalAI benchmark measurements.
Assisted-by: Codex:gpt-6
(cherry picked from commit ec50871dd6)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
qwentts builds its code-predictor and codec-stream graphs lazily per batch
width. With max_batch > 1 the first request that hits a new width (for
example two requests finishing in the same step) stalled the whole batch
for ~5 s on an RTX 5060 Ti; afterwards the same pattern took ~1 s.
When max_batch > 1, Load now synthesizes N short utterances of increasing
length concurrently through the streaming path and discards the audio, so
the slots finish one by one and every width from N down to 1 is built
before the first real request. `max_batch_warmup:false` skips it.
Signed-off-by: nexxtmobile.de <git@nexxtmobile.de>
Assisted-by: Claude:claude-opus-5-5
(cherry picked from commit 0b7d0b40ed)
qwentts.cpp batches concurrent qt_synthesize calls on one context when it
is initialized with max_batch > 1 (its compute worker coalesces them into
batched decode steps; qt_synthesize stays blocking and thread safe). The
LocalAI wrapper never set max_batch and serialized every request with
base.SingleThread, so concurrent /v1/audio/speech requests always ran one
after another.
- new model option `max_batch:N` (alias `parallel:N`, default 1 = current
behavior), passed to qt_init_params.max_batch via qt3_load
- Locking() keeps LocalAI's per-backend request lock only for max_batch 1;
with batching TTS/TTSStream reach the native worker concurrently
- TTSStream routes chunks per request: a fresh id travels as the C
callback's user_data (one shared purego callback, as before), each request
has its own queue drained into its channel, so the native worker never
blocks on a slow consumer and concurrent streams cannot mix
- TTSLive rejects max_batch > 1 (qwentts incremental mode requires a
max_batch=1 context, docs/INCREMENTAL.md)
- README: option, measurements, TTSLive limit, ROCm/gfx1201 hipBLASLt note
- unit tests: option parsing, Locking, concurrent stream routing (fails if
requests were serialized), TTSLive rejection
Measured with qwentts' own tts-server and the 1.7B Base Q4_K_M talker,
four concurrent streaming requests, time to first PCM: RTX 5060 Ti (CUDA)
0.02/0.36/0.70/1.04 s at max_batch 1 vs 0.06 s for all four at 4;
Radeon AI PRO R9700 (ROCm, with ROCBLAS_USE_HIPBLASLT=0) 0.03/0.71/1.36/
2.02 s vs <= 0.11 s.
Signed-off-by: nexxtmobile.de <git@nexxtmobile.de>
Assisted-by: Claude:claude-opus-5-5
(cherry picked from commit da0c5435b5)
whisper passed -DGGML_HIPBLAS=ON, which ggml no longer knows, so the
hipblas image was a CPU-only build (same issue as #10666 in
qwen3-tts-cpp). Use GGML_HIP with the ROCm clang and forward the gfx
targets. parakeet-cpp enabled PARAKEET_GGML_HIP but did not forward
the gfx targets; forward them the same way.
Docs-Not-Needed: build flags only, no user-facing behaviour or option changes.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit 0cb6c908da)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(proxy): accept direct upstream API keys
Support direct credentials for models created in the UI. Keep env and
file references as mutually exclusive alternatives. Use model env for
frontend probes and omit credentials from structured config logs.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(distributed): isolate model environments in worker processes
Forward complete model environment snapshots through backend installation.
Restart worker children when overrides change or are removed, and include
keyed snapshot identities in frontend install coalescing.
Cover wire compatibility, routing, real worker children, and option copying;
document rollout requirements and control-transport security.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(worker): serialize installs by model replica
Different backend names can target the same process slot. Hold a slot
lock through startup, readiness, and operation attachment so concurrent
installs cannot reuse another configuration or an unready child.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(nodes): update endpoint stub for model env
Keep the endpoint command stub aligned with the environment-aware install
interface. Assert interface conformance so future changes fail at the
stub declaration rather than at endpoint call sites.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update NVIDIA/NeMo-Speech.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(nemo-speech-cpp): use upstream llama.cpp ggml patching
Initialize ggml through the llama.cpp submodule and let upstream CMake apply the patch series with fail-closed error handling.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): recognize upstream OpenFST ownership fix
Accept both WrapUnique assignments unchanged while preserving the legacy header patch and rejecting unknown or incomplete forms.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): bundle shared text normalization runtime
Stage the normalization DSO when ITN is present and include it in the package alongside its dependency closure. Fail staging if the required DSO is missing.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update mudler/vllm.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(vllm-cpp): pin the Queue declaration fix
Advance vllm.cpp to the upstream fix for the mismatched Queue tag.
This fixes the strict Clang compilation failure without local patches
or warning suppression.
Focused declaration probes fail before and pass after the upstream fix.
Full suites and builds are skipped at the owner’s request.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update leejet/stable-diffusion.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(sd): adopt owning generation results
The sd.cpp bump replaces raw image and video outputs with owning results.
Adapt image generation, video generation and upscaling to the new API.
Use upstream destructors to release all pixels, audio and metadata.
Add model-free native checks for result ownership and failure paths.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* chore(parakeet-cpp): bump parakeet.cpp to load Phonon-2.5 models
Move the pin to the current parakeet.cpp master. It includes the
dense and packed CPU paths for the Phonon-2.5 checkpoints, which the
previous pin cannot load.
After the bump, run `make purge` in backend/go/parakeet-cpp so the
next build clones the new revision.
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
* feat(gallery): add Phonon-2.5 parakeet-cpp models
Add the multilingual Phonon-2.5 (25 European languages) and the
English-only Phonon-2.5-English checkpoints as parakeet-cpp entries.
Each comes in four builds: F16, Q8_0, Q4_K and packed. The packed
build runs on the CPU only. The files are in
mudler/parakeet-cpp-gguf, pinned by SHA-256. The weights are
CC-BY-4.0.
The entries need the parakeet.cpp pin from the previous commit.
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(grpc): add a Forecast RPC for time-series models
Add Forecast to the backend service with columnar request and response
messages: flat row-major arrays, NaN for missing values, and one message
per series so a caller can send series of different shapes in one call.
The response keeps the request order.
Wire the RPC through the server, client, embedded client, connection
evicting client, in-flight tracking client and the base backend, which
answers unimplemented. core/backend gains a Forecast helper that stamps
the model identity and records a trace, like the other single-purpose
RPCs.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
* feat(config): add the forecast usecase
FLAG_FORECAST marks a model as a time-series forecaster. The timestral-cpp
backend declares it as its default usecase, so a config with that backend
needs no extra lines. The backend is also listed as a non-text backend, so
a forecast model is never offered as a chat model. It shows up in the
gallery filter and in the capability list.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
* feat(api): add POST /v1/forecast
Take one or more numeric series and return quantile trajectories for the
next prediction_length steps. Targets are [variates][steps] or a flat
array for one variate; JSON null marks a missing value. Past and known
covariates and categorical codes are checked for length before the
backend sees them. Quantiles default to 0.1, 0.5 and 0.9 in the backend.
The route is gated by a new default-on forecast API feature. The endpoint
is listed in the swagger spec, the /api/instructions index, the UI model
filter and the capability symbols.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
* feat(backends): add the timestral-cpp forecast backend
Wrap the timestral.cpp C ABI with purego, the way the other ggml
backends do: libtimestral is built from a pinned commit with ggml
embedded, and the Go binary loads it at start.
Forecast validates the counts against the array lengths, groups the
series by shape, makes one library call per group and puts the results
back in the order the caller sent them. Quantiles default to 0.1, 0.5
and 0.9, and a level the model does not have is refused with the list of
valid ones. A mutex guards the handle because the library is not
reentrant. Request size is bounded by two model options.
The unit specs use a fake library. The integration specs and a gRPC
spec run against the 5M F32 model that test.sh downloads and checks
against a fixed SHA-256.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
* build(backends): register timestral-cpp in the build and the library
Add the CPU image to the backend matrix for amd64 and arm64, plus an
Apple silicon entry that is not built or tested yet. The image has no GPU
keys because the engine runs on the CPU only. Wire the make targets, the
path filter and the dependency bump bot.
The library entries were generated with backendcatalog. The images are
not published yet, so the release entries are unpinned until a
maintainer runs --repin. The gallery reader test now counts the names in
the library instead of a fixed number, so adding a backend does not
break it.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
* feat(gallery): add the Timestral forecast models
Add Timestral 5M, 28M, 123M and 209M as F32 entries, each with its F16
build as a variant. Q8_0 and the K-quants are left out until the quality
question is settled. Each entry runs on the timestral-cpp backend, on
the CPU only.
The weights use Mistral's Modified MIT licence, which bars use by
companies above a monthly revenue threshold. The first paragraph after
the summary says so, the licence file is installed next to the weights,
and the revenue-restricted-license tag makes the entries filterable.
The memory each build needs has not been measured, so no size is set.
Assisted-by: Claude:claude-sonnet-5-5
* docs(forecast): document /v1/forecast and the Timestral models
Add a feature page with the request and response shapes, the error
cases, the model options and the licence note. List the engine in the
README, the compatibility table and the backend categories.
Assisted-by: Claude:claude-sonnet-5-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(tts): stream Qwen text through audio pipelines
Feed cleaned LLM deltas into one persistent Qwen synthesis session.
Return audio before input ends and preserve the live RPC through model
cache, staging and in-flight wrappers. Stop generation on sink failures
or cancellation without replaying speech through a fallback.
Pin the incremental native fork and verify early audio, late text,
finalization and cancellation with lifecycle and real-model checks.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* build(tts): use the organization Qwen fork
Fetch the reviewed incremental synthesis commit from localai-org so
backend builds use the organization-maintained fork.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add a Qwen streaming assistant preset
Expose an English pipeline using existing VAD, ASR, LLM and named-voice
TTS models. Enable incremental text and audio delivery, and document
separate component installs because the gallery has no dependency
installer.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): use Parakeet TDT v3 for the assistant
Select the existing Parakeet TDT 0.6B v3 transcription entry and update
its prerequisite install command. Keep the preset's English voice and
incremental Qwen synthesis configuration.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(kokoros): reject unsupported backend RPCs
Implement the MotionStream and SlotCache methods required by the shared
protobuf service. Return Unimplemented so Kokoros compiles without
advertising support for either operation.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(tts): mark cleanup and logging results ignored
Explicitly ignore connection shutdown, test server shutdown and
Ginkgo diagnostic writer errors. These operations do not replace the
stream's inference or sink errors, and errcheck requires their return
values to be acknowledged.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(tts): stream buffered clauses in realtime pipelines
Start clause synthesis while LLM text is still arriving when both streaming flags are enabled. Keep explicit clause opt-out and native Qwen sessions, bound automatic queued work, and choose streaming support on resolved failover targets.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(tts): bound synthetic voice clone generation
The sine-wave reference checks speaker conditioning and WAV output, but does not provide a meaningful speech EOS target. Limit that smoke request to 128 frames and verify the request limit reaches native synthesis. Leave production sampling defaults unchanged.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(tts): require explicit clause buffering opt-in
Keep whole-reply synthesis for nonnative engines when clause_chunking is unset or false. Use the bounded cancellable worker when explicitly enabled, with unary clauses when TTS streaming is disabled. Preserve native Qwen precedence for every clause setting.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add VITS and Piper streaming assistants
Provide comparable full pipeline configurations with shared VAD, Parakeet transcription, and Qwen chat components. Enable clause buffering explicitly and document the separate prerequisite model installs and Talk selection.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
