* feat(vibevoice-cpp): true streaming TTSStream via vv_capi_tts_stream
Replaces the synth-to-tempfile TTSStream hack with a real streaming path:
binds the new vv_capi_tts_stream callback ABI via a single reusable purego
callback (CGO_ENABLED=0-safe, no runtime/cgo), copies each int16 PCM window
into the gRPC results channel after the streaming WAV header.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* test(vibevoice-cpp): real-model streaming integration test with TTFA measurement
Gated behind VIBEVOICE_IT=1, this Ginkgo spec dlopens the engine .so and
drives the exact Go->purego->C TTSStream/TTS path against the real
vibevoice-realtime-0.5B model. It measures time-to-first-audio for the
streaming path versus the batch path and asserts the streaming win:
44-byte WAV header first, >=2 PCM windows, non-silent audio, and
TTFA < total_stream. Without the env var the spec skips so CI and
normal go test are unaffected.
Measured: TTFA 2.38s vs batch deliver-time 39.96s (first audio in 5.9%
of the batch time, ~17x faster), 18 stream chunks, non-silent 24kHz PCM.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(vibevoice-cpp): pin streaming-decoder engine build
Bumps VIBEVOICE_CPP_VERSION to the streaming-decoder engine commit that
adds vv_capi_tts_stream (localai-org/vibevoice.cpp#8). Re-pin to the
merged master commit once that PR lands.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(vibevoice-cpp): re-pin to merged streaming-decoder commit
localai-org/vibevoice.cpp#8 merged to master as 000e372; move the pin
off the PR branch commit onto the merged master commit.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* test(vibevoice-cpp): check writer errors in TTFA report (errcheck)
golangci-lint errcheck flagged the unchecked fmt.Fprintf calls that
print the streaming TTFA headline. Build the report once with
fmt.Sprintf and write it per destination with an explicitly discarded
error, matching the GinkgoWriter reporting idiom used by the other
backend tests.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-fable-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
C++/ggml transcription + speaker diarization + timestamps backend. Purego
dlopens libmoss-transcribe.so (ggml statically linked) from moss-transcribe.cpp
and serves offline AudioTranscription, parsing the [start][Sxx]text[end] output
into segments with nanosecond timestamps. Adds the importer (surfaces in
GET /backends/known), backend-matrix (Linux + Darwin/metal), backend/index.yaml,
and a gallery entry (default q5_k GGUF from mudler/moss-transcribe.cpp-gguf).
Local L0 smoke (go build + go test ./... = 16 pass, golangci-lint 0 issues)
passed against the real libmoss-transcribe.so. The pre-commit coverage gate
(full pkg/core + tests/e2e) could not run in the authoring sandbox (no live
models, port 9090 held); CI must enforce it before merge.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Link the f5-tts library into the crispasr backend so CrispASR's native
F5-TTS runtime (SWivid F5-TTS, 22-layer DiT flow-matching + built-in Vocos
vocoder) is compiled in. The single self-contained GGUF auto-detects as
f5-tts through the session router, so no explicit backend selector is
needed. Add the f5-tts-crispasr gallery entry (cstr/f5-tts-GGUF) and an
env-gated e2e synthesis spec.
F5-TTS is voice-cloning only and has no baked speaker: it clones from a
reference WAV plus its transcript, supplied via the voice/voice_text
options. The gallery description documents this bring-your-own-reference
requirement.
Verified e2e on the pinned engine (278fb79): the GGUF auto-detects as
f5-tts, the reference voice loads, and synthesis produces a valid 24 kHz
mono WAV.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(backends): pin grpcio-tools to the installed grpcio in runProtogen
runProtogen installed grpcio-tools unpinned, so the protoc it bundles
stamped backend_pb2.py with the newest Protobuf gencode (7.35.0). When a
backend caps the protobuf runtime lower -- vLLM pins protobuf to 6.33.6 --
the import-time guarantee runtime >= gencode fails:
google.protobuf.runtime_version.VersionError: Detected incompatible
Protobuf Gencode/Runtime versions ... gencode 7.35.0 runtime 6.33.6
The backend crashes on `import backend_pb2` before it can serve, which
surfaces to the user as "grpc service not ready". It was mis-reported as a
ROCm/gfx1201 failure in #10718 but is not GPU-specific and affects every
vLLM variant (and any backend that caps protobuf below the latest gencode).
Pin grpcio-tools to the grpcio version the backend already installed --
they release in lockstep -- so the generated gencode stays in step with
the protobuf runtime. Falls back to unpinned when grpcio isn't present.
Closes#10718
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The image backends call PIL Image.save(request.dst) without a format, so
Pillow infers the encoder from the file extension. The core passes an
absolute staging path ending in .tmp (e.g. /staging/localai-output-*.tmp),
which Pillow can't map to a format, raising "unknown file extension: .tmp"
and crashing the worker right after a successful GPU inference.
Pass format="PNG" explicitly. LocalAI serves generated images as PNG
regardless of the temporary path, so this is always correct and no longer
depends on the extension of the destination the core happens to allocate.
diffusers is the reported backend (#10727); vllm-omni and tinygrad carry
the identical latent crash for any .tmp staging destination.
Closes#10727
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The nvidia-l4t-cuda-13-arm64 vLLM backend left `vllm` unpinned, so the
prebuilt image drifted onto whatever aarch64 wheel was latest at build
time (0.23.x). On GB10 / DGX Spark (Grace Blackwell, unified memory),
0.23 crashes deterministically during cold model loads with an empty
"Engine core initialization failed" set and pins GPU memory until a host
reboot.
vLLM 0.24.0 carries vllm-project/vllm#45179 ("release cached device
memory under pressure on UMA GPUs during weight loading"), which the
reporter verified fixes the crash on GB10. Pin the L4T requirements to
0.24.0 to match the already-pinned cublas13 build
(requirements-cublas13-after.txt) and keep the image deterministic.
Editing this file also re-triggers the single-arch L4T image build via
the path filter, republishing the gallery image with 0.24.0 (the
single-arch matrix builds again after #10703).
Closes#10722
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Allow llama.cpp model configs to select the backend devices used for offload, matching upstream --device behavior so users can exclude a display or debug GPU.
Signed-off-by: rvmzes <rvmzes@rvmzess-MacBook-Pro.local>
Co-authored-by: rvmzes <rvmzes@rvmzess-MacBook-Pro.local>
* ⬆️ Update leejet/stable-diffusion.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(stablediffusion-ggml): pass chroma knobs via model_args after upstream API change
Upstream stable-diffusion.cpp bb849711 removed the dedicated
chroma_use_dit_mask / chroma_use_t5_mask / chroma_t5_mask_pad fields from
sd_ctx_params_t and now reads them from the generic model_args key=value
spec (parse_key_value_args). Assigning the old struct members broke the
gosd.cpp build. Emit the three options into model_args instead so the
existing chroma controls keep working. Verified by building
libgosd-fallback.so against the pinned upstream commit.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
qwen3-tts-cpp, omnivoice-cpp, acestep-cpp and vibevoice-cpp shipped
rocm-* variants that silently ran on CPU ([Load] backend: CPU). Two
coupled defects:
- The Makefiles passed -DGGML_HIPBLAS=ON, but the vendored ggml only
understands -DGGML_HIP=ON (GGML_HIPBLAS was removed upstream), so the
ggml-hip backend target was never created and no GPU code was built.
- The CMake foreach that links the ggml GPU backends into the module
listed blas/cuda/metal/vulkan but not hip, so even a built ggml-hip
would not have been linked and its static backend registration would
never run.
CUDA users were unaffected because cublas passes the correct GGML_CUDA=ON
and the foreach already links cuda. Mirror the proven llama-cpp hipblas
block (ROCm clang CC/CXX + AMDGPU_TARGETS) and add hip to each foreach.
Upstream picks the best device via ggml_backend_init_best(), so no
runtime flag is needed once HIP is compiled and linked.
Assisted-by: Claude:claude-opus-4-8[1m] [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The ROCm packager copied rocBLAS kernel data (rocblas/library/*.dat) into the
bundled lib/ dir and run.sh pointed ROCBLAS_TENSILE_LIBPATH at it, but the
parallel hipBLASLt data dir (hipblaslt/library/TensileLibrary_lazy_gfx*.dat)
was never packaged and no HIPBLASLT_TENSILE_LIBPATH was set. The bundled
libhipblaslt.so therefore resolved its per-arch kernel data relative to itself,
found nothing, and silently fell back to slow generic kernels, logging:
rocblaslt error: Cannot read "TensileLibrary_lazy_gfx1201.dat": No such file or directory
rocblaslt error: Could not load "TensileLibrary_lazy_gfx1201.dat"
Fix, mirroring the existing rocBLAS handling:
- package-gpu-libs.sh: extract the rocblas data-dir copy into a reusable
copy_rocm_data_dir helper and call it for both rocblas and hipblaslt.
- llama-cpp/turboquant run.sh: export HIPBLASLT_TENSILE_LIBPATH when the
bundled hipblaslt/library dir exists.
The helper takes an optional ROCM_BASE_DIRS override so the copy is unit
testable without a real ROCm install; add a regression test that runs
package_rocm_libs against a fabricated ROCm tree and asserts both data dirs
are bundled.
Note: this bundles whatever gfx*.dat the build image's ROCm provides. If a
given arch's tensile data is absent from the shipped ROCm, that arch still
needs a ROCm bump; the packaging gap itself is fixed for every supported arch.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Two bugs broke OpenAI-style tool calling on the MLX backend (and any
Python backend sharing backend/python/common), reproduced end-to-end on
LocalAI v4.5.5 with the metal-mlx backend and
mlx-community/Qwen3.5-2B-MLX-8bit.
messages_to_dicts left each tool call's function.arguments as the raw
OpenAI-wire JSON string. HuggingFace chat templates (e.g. Qwen3.5)
iterate arguments as a mapping (.items()), so any request whose history
contained a prior assistant tool_calls message failed with HTTP 500
"Generation failed: Can only get item pairs from a mapping." — breaking
every agent loop on its second turn. Decode the string back into a dict
so the template sees a mapping.
split_reasoning returned ("", text) whenever the opening think tag was
absent. Models like Qwen3.5 open the assistant turn already inside
thinking, so the generated text carries only the closing </think>; the
whole chain-of-thought leaked into content. When the opener is missing
but the closer is present, treat everything before the closer as
reasoning.
Adds platform-independent unit tests under backend/python/common
(stdlib-only, no MLX/venv required, following parent_watch_test.py).
Assisted-by: Claude Code:claude-opus-4-8
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
