Pocket TTS ships one model per language, but LoadModel called
TTSModel.load_model() with no arguments, so the library always fell back
to its English model whatever the model config said. Read a `language`
option and pass it to load_model, keeping the library default when it is
not set. Document the option and add unit tests that mock the loader.
Fixes#12759
Assisted-by: Claude:claude-opus-5-5
The sd_embed branch dropped negative_prompt for every pipeline, which
discarded the user's negative prompt for Flux, where no negative embeds
are produced. Only drop it when negative_prompt_embeds is present, and
move the check to a helper that has unit tests.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
AudioTranscription caught every exception and returned an empty
TranscriptResult. A failed diarization step therefore discarded a
transcript that was already finished: with an HF token that has not
accepted the terms of the gated pyannote pipeline, the download fails
with 403 and every transcription came back as an empty text with
HTTP 200.
Diarization now degrades: if it fails, the transcript is returned
without speaker labels and the reason is logged. Any other failure
aborts the call with INTERNAL instead of pretending success with an
empty text.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
FunASR left Transformers and Hugging Face Hub unconstrained, which let uv backtrack to tokenizers 0.10.3 without Python 3.12 wheels. Keep Transformers on the supported 4.x range, including the Intel upgrade profile.\n\nAssisted-by: nib:gpt-5.6-sol\nSigned-off-by: Ettore Di Giacinto <mudler@localai.io>
* sglang backend: pass through thinking_budget + require_reasoning
sglang's raw Engine.async_generate() API (which this backend calls
directly, bypassing sglang's own OpenAI server) supports a precise,
tokenizer-derived reasoning-length budget via
sampling_params["custom_params"]["thinking_budget"] plus
require_reasoning=True, gated behind --enable-strict-thinking. Neither
was reachable through LocalAI: this backend built sampling_params only
from a fixed field mapping (temperature, top_p, ...) with no custom_params
key, and never passed require_reasoning to async_generate at all.
- LoadModel now reads a model-level "thinking_budget" option (same
mechanism as the existing tool_parser/reasoning_parser options), and
_build_sampling_params adds it as custom_params.thinking_budget on
every request when configured.
- _new_reasoning_parser already derives, from the rendered prompt, whether
the model's chat template pre-opened a reasoning block (Qwen3-style
templates append <think> to the prompt instead of letting the model
emit it) -- the same signal sglang's own OpenAI server computes from
per-template config to decide require_reasoning. This backend has no
template manager, so it now returns that signal too and _predict
forwards it to async_generate(require_reasoning=...).
Verified against production (NVFP4, sm_121, Qwen3.6-35B-A3B) via a raw
Engine.async_generate() call bypassing this backend: 301 reasoning
tokens against a 300-token budget, clean completion, ~27s. Not yet
verified through this backend's own gRPC path end-to-end (no local
CUDA/sglang environment available here) -- existing + new unit tests in
test.py cover the pure-Python merge/passthrough logic only.
Scope note: require_reasoning is derived only from the existing
prompt-suffix heuristic, not sglang's full per-template
_get_reasoning_from_request decision tree (minimax-m3/hunyuan special
cases etc.) -- this backend has no template manager to evaluate that
tree against, and the prompt-suffix check is the one heuristic already
validated in this file (test_reasoning_parser_forced_when_template_prefills_think_tag).
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
* sglang backend: honour a model-level reasoning_default
A model YAML can already carry "parameters: reasoning_effort:", but that
value only reaches this backend when a *caller* sets it per request (the Go
side turns it into Metadata["enable_thinking"]). As a model-level default it
is silently dropped: a config reading "reasoning_effort: none" still produces
full reasoning on every request, so the config says one thing and the model
does another.
That gap is expensive in practice. On a self-hosted Qwen3.6-35B-A3B the
reasoning phase consumed the entire max_tokens budget before any content was
produced - 90% of code completions came back empty at max_tokens=768, and the
server log filled with "backend produced only reasoning, retrying". The
config looked like reasoning was off the whole time.
This adds "reasoning_default:off" (or ":on") on the same model-level
options: mechanism as thinking_budget. A per-request value always wins; the
default only fills in when the request is silent.
Measured on the stack above (sglang 0.5.20, NVFP4, GB10/sm_121) after
applying it:
default (nothing set) -> 0 chars reasoning, 27 tokens
"reasoning_effort": "none" -> 0 chars reasoning, 27 tokens
metadata enable_thinking=true -> capped at the 512-token thinking_budget,
541 tokens total, finish_reason stop
Tests: three cases added to backend/python/sglang/test.py covering the
default, per-request override in both directions, and the unconfigured case
(which must leave the template untouched).
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
* sglang backend: validate thinking_budget instead of crashing LoadModel
Addresses the review on this PR:
- `int(thinking_budget)` raised on values like "5000.0" or "abc" and took
LoadModel down. The option is now parsed by _parse_thinking_budget():
integral numbers in any spelling are accepted, anything else is ignored
with a warning on stderr.
- Zero and negative budgets are ignored with a warning instead of being
passed to sglang, where they have no defined meaning. Turning reasoning
off is what reasoning_default:off is for.
- A load-time warning when thinking_budget is set but enable_strict_thinking
is not in engine_args, since sglang then ignores the budget silently.
- Tests for integral spellings, unset, zero, negative, non-integer and the
strict-thinking warning.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
* docs(sglang): explain reasoning options
Document the reasoning budget, strict-thinking requirement, and
precedence of request metadata over the model-level default.
Also note that the budget has to stay well below max_tokens (otherwise
it never triggers and the reply can end up empty), and that
POST /models/reload or a backend-only restart does not pick up changed
options; LocalAI itself has to be restarted.
Assisted-by: Codex:GPT-6
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
* docs(sglang): clarify configuration reloads
Distinguish rereading model configuration from updating a running backend. Keep the full LocalAI restart recommendation for changed reasoning options.
Assisted-by: Codex:GPT-6
* sglang backend: only pass require_reasoning when sglang supports it
Engine.async_generate() gained the require_reasoning keyword in sglang
0.5.13 and takes no **kwargs. The CPU profile builds v0.5.11 from source
and the other profiles only set a >=0.5.11 floor, so passing the keyword
unconditionally made every request fail with TypeError. Detect support
once at import time, as the file already does for sampling_seed.
enable_strict_thinking first appears in sglang 0.5.12; fix the comment.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
---------
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <localai-org-maint-bot@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
sglang 0.5.20 moved its config tier from dataclasses to msgspec.Struct
(sgl-project/sglang#38753). _apply_engine_args validates engine_args keys
via dataclasses.fields(ServerArgs), which raises TypeError there. That
call runs on every LoadModel, so no model loads at all on the sglang
backend once sglang >= 0.5.20 is installed, and the error surfaces as a
generic "Unexpected <class 'TypeError'>" that does not name the cause.
Introspect both shapes: msgspec structs carry their field names in
__struct_fields__, so key validation and the close-match suggestion keep
working, and older dataclass-based sglang stays supported.
Adds a test that pins the msgspec path with a stand-in, so it is covered
regardless of which sglang version is installed.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
Use a regular (non-editable) pip/uv install so the package lands in the
venv site-packages. An editable finder records the builder source path,
which breaks after the backend is copied out of the image (#9162).
Adds a regression test (scripts/build/vllm-omni-install_test.sh) that
verifies imports survive relocation with a regular install and fail with
an editable install.
Supersedes #12040 (DCO not signed by contributor).
Assisted-by: MAKI:regolo/glm5.2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The CUDA 13 profile resolves PyTorch 2.14. Its headers cannot build the
current FlashAttention source release under C++17, so the full backend
matrix cannot publish the image.
Leave FlashAttention optional on CUDA 13. The backend already retries
model loading with SDPA when FlashAttention is unavailable.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The CUDA 13 FlashAttention build still exhausts hosted-runner memory with a single ninja worker because nvcc can compile multiple threads internally. Limit nvcc to one thread for that profile and guard the setting in the backend test script.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
SciPy 1.18 requires Python 3.12 or newer. Keep this backend on a
compatible portable Python for Linux and macOS builds.
Assisted-by: Codex:gpt-5.6 [Codex]
(cherry picked from commit 70bf6d4a3a)
Adds FunASR/SenseVoice as a Python backend for speech-to-text with
support for CPU, CUDA 12/13, ROCm, Intel SYCL, L4T, and Apple MPS.
Co-authored-by: xingyifeng <xingyifeng@users.noreply.github.com>
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The editable install records the backend build path, which does not exist after LocalAI relocates the packaged backend. Add the runtime source directory to PYTHONPATH so inference modules remain importable.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Expose diffusers audio pipelines through the existing sound-generation RPC. AudioLDM2 can now return PCM WAV output from the model gallery without a separate backend.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The pinned mlx-video package requires Python 3.11 or newer, while an empty PYTHON_VERSION selected the backend helper default of 3.10. Pin the available 3.11.13 portable runtime for the Darwin package build.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a Darwin-only MLX-Video backend for LTX-2 and converted Wan checkpoints, expose it through the existing video API, and wire packaging, discovery, tests, docs, and an example.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Prefer an explicitly configured Triton assembler, otherwise use the executable ptxas from CUDA_HOME so torch.compile can target GPU architectures newer than Triton bundled tooling.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Keep the relocated upstream source importable, select CUDA 13 PyTorch wheels instead of the aarch64 CPU fallback, and decode reference audio without torchcodec, which has no Linux arm64 wheels.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The standalone Python release used by libbackend does not publish a 3.11.18 artifact. Pin Whisper-Medusa to the available 3.11.13 build and cover the generated download URL.
Assisted-by: Codex:gpt-5 [systematic-debugging]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(faster-whisper): manually install ctranslate2 with rocm support before installing other dependencies
Signed-off-by: Andreas Egli <github@kharan.ch>
* feat(faster-whisper): wire version into bump-deps workflow
Assisted-by: opencode:gpt-5.5
Signed-off-by: Andreas Egli <github@kharan.ch>
---------
Signed-off-by: Andreas Egli <github@kharan.ch>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
The mlx, mlx-vlm, mlx-distributed and vllm-omni backends only forwarded
enable_thinking when the metadata value was "true". A "false" value never
reached apply_chat_template, so requests with thinking disabled (for
example a realtime pipeline with disable_thinking: true) still used the
chat template default. Apply the same coerce that #11715 added to sglang
and vllm.
Assisted-by: Claude:claude-opus-5
Signed-off-by: devv-shayan <shayankhanx1x@gmail.com>
Backend processes shared the host temporary directory, so crashes could leave request images and audio behind until the filesystem filled. Give each process a locked LocalAI-owned runtime, remove scratch on exit, and sweep only marked abandoned runtimes at the next start.
Also close known request error-path leaks in the Python media backends, CrispASR, LongCat Video, and stable-diffusion.cpp.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The device fell back to CPU unless the model config set cuda: true,
while MPS right below was auto-detected — GPU hosts silently rendered
on CPU for any gallery entry missing the flag. Use CUDA whenever torch
reports it available (ROCm builds included), keep cuda: true as an
explicit force, and allow pinning with the device: model option (e.g.
options: ["device:cpu"]). Gallery entries stay untouched.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
Younes Beriane