The CUDA 13 profile resolves PyTorch 2.14. Its headers cannot build the
current FlashAttention source release under C++17, so the full backend
matrix cannot publish the image.
Leave FlashAttention optional on CUDA 13. The backend already retries
model loading with SDPA when FlashAttention is unavailable.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The CUDA 13 FlashAttention build still exhausts hosted-runner memory with a single ninja worker because nvcc can compile multiple threads internally. Limit nvcc to one thread for that profile and guard the setting in the backend test script.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
SciPy 1.18 requires Python 3.12 or newer. Keep this backend on a
compatible portable Python for Linux and macOS builds.
Assisted-by: Codex:gpt-5.6 [Codex]
(cherry picked from commit 70bf6d4a3a)
Adds FunASR/SenseVoice as a Python backend for speech-to-text with
support for CPU, CUDA 12/13, ROCm, Intel SYCL, L4T, and Apple MPS.
Co-authored-by: xingyifeng <xingyifeng@users.noreply.github.com>
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The editable install records the backend build path, which does not exist after LocalAI relocates the packaged backend. Add the runtime source directory to PYTHONPATH so inference modules remain importable.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Expose diffusers audio pipelines through the existing sound-generation RPC. AudioLDM2 can now return PCM WAV output from the model gallery without a separate backend.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The pinned mlx-video package requires Python 3.11 or newer, while an empty PYTHON_VERSION selected the backend helper default of 3.10. Pin the available 3.11.13 portable runtime for the Darwin package build.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a Darwin-only MLX-Video backend for LTX-2 and converted Wan checkpoints, expose it through the existing video API, and wire packaging, discovery, tests, docs, and an example.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Prefer an explicitly configured Triton assembler, otherwise use the executable ptxas from CUDA_HOME so torch.compile can target GPU architectures newer than Triton bundled tooling.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Keep the relocated upstream source importable, select CUDA 13 PyTorch wheels instead of the aarch64 CPU fallback, and decode reference audio without torchcodec, which has no Linux arm64 wheels.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The standalone Python release used by libbackend does not publish a 3.11.18 artifact. Pin Whisper-Medusa to the available 3.11.13 build and cover the generated download URL.
Assisted-by: Codex:gpt-5 [systematic-debugging]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(faster-whisper): manually install ctranslate2 with rocm support before installing other dependencies
Signed-off-by: Andreas Egli <github@kharan.ch>
* feat(faster-whisper): wire version into bump-deps workflow
Assisted-by: opencode:gpt-5.5
Signed-off-by: Andreas Egli <github@kharan.ch>
---------
Signed-off-by: Andreas Egli <github@kharan.ch>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
The mlx, mlx-vlm, mlx-distributed and vllm-omni backends only forwarded
enable_thinking when the metadata value was "true". A "false" value never
reached apply_chat_template, so requests with thinking disabled (for
example a realtime pipeline with disable_thinking: true) still used the
chat template default. Apply the same coerce that #11715 added to sglang
and vllm.
Assisted-by: Claude:claude-opus-5
Signed-off-by: devv-shayan <shayankhanx1x@gmail.com>
Backend processes shared the host temporary directory, so crashes could leave request images and audio behind until the filesystem filled. Give each process a locked LocalAI-owned runtime, remove scratch on exit, and sweep only marked abandoned runtimes at the next start.
Also close known request error-path leaks in the Python media backends, CrispASR, LongCat Video, and stable-diffusion.cpp.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The device fell back to CPU unless the model config set cuda: true,
while MPS right below was auto-detected — GPU hosts silently rendered
on CPU for any gallery entry missing the flag. Use CUDA whenever torch
reports it available (ROCm builds included), keep cuda: true as an
explicit force, and allow pinning with the device: model option (e.g.
options: ["device:cpu"]). Gallery entries stay untouched.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
Qwen3-style chat templates append the opening <think> tag to the *prompt*
when thinking is enabled. The model therefore never generates it and emits
only the reasoning text plus the closing </think>.
sglang's ReasoningParser keys off the opening tag:
in_reasoning = self._in_reasoning or self.think_start_token in text
if not in_reasoning:
return StreamingParseResult(normal_text=text)
so with such a template the entire completion — reasoning and answer, the
raw </think> in between — is returned as content and reasoning_content
stays empty, no matter how reasoning_parser is configured.
sglang's own OpenAI server handles this via
force_reasoning = (self.template_manager.force_reasoning
or self._get_reasoning_from_request(request))
This backend has no template manager, so derive the same signal from the
rendered prompt: if it ends with the detector's think_start_token, the tag
was prefilled and the parser is constructed with force_reasoning=True.
Structured decoding is the exception, and it matters: a grammar applies
from the first token, so the model cannot emit the closing tag even though
the template opened the block. The whole completion is schema output and
belongs in content — forcing there files it as reasoning and returns an
empty answer. Measured against a JSON-schema code audit: 10107 characters
of "reasoning", zero content. sglang's own server keeps the two apart for
the same reason; its grammar backend owns the reasoning prefix when a
reasoning parser is configured.
force_reasoning is only passed when it is meant to be True, so detector
defaults (DeepSeek-R1 already defaults to True) are untouched, and a
prompt without a prefilled tag behaves exactly as before — which matters,
because forcing unconditionally makes an answer generated with thinking
off disappear into reasoning_content.
The construction is factored into _new_reasoning_parser() so the streaming
and non-streaming paths, which previously built the parser separately,
cannot drift apart.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
With `template.use_tokenizer_template: true` the sglang and vllm backends
render the prompt themselves via `tokenizer.apply_chat_template()`, and they
hand it plain string content. A chat template only emits the model's own media
tokens when the content is a list of parts, so the rendered prompt carries no
`<|vision_start|><|image_pad|><|vision_end|>`. The pixels do reach the engine
(`image_data` / `multi_modal_data`), but both engines locate them by scanning
the prompt for that token, so they are discarded silently: HTTP 200, no
warning, and the model answers as if no image had been attached.
Add `attach_media_parts()` to the shared `python_utils` helper and call it in
both backends: the last user turn is rebuilt as
`[{"type": "image"} * n, {"type": "video"} * n, {"type": "text", ...}]` before
templating, which makes the template emit the placeholders. The pixels keep
travelling out of band exactly as before.
Text-only requests are untouched - with no media the helper returns None and
the original string-content path runs unchanged. If a template cannot iterate
content parts (a text-only model), the parts render is caught and the request
falls back to the previous string-content prompt instead of failing.
Signed-off-by: Tai An <antai12232931@outlook.com>
vLLM's engine-based reasoning parsers derive their initial state from the
chat template kwargs. Qwen3Parser:
chat_kwargs = kwargs.get("chat_template_kwargs", {}) or {}
self.thinking_enabled = chat_kwargs.get("enable_thinking", True)
Constructed as ReasoningParser(tokenizer) the flag defaults to True, so the
parser starts in the REASONING state. A completion produced with thinking
disabled contains no tags at all, and every reasoning parser shape then
reports the whole answer as reasoning:
- engine-based parsers classify it by initial state;
- BaseThinkingReasoningParser hits its documented "may not generate start
token" fallback and returns (model_output, None).
Either way `content = c if c is not None else generated_text` turns that
into a duplicate: a Qwen3 model answering "391" with thinking off comes back
as reasoning_content="391" AND content="391".
Measured against Qwen3.5-MoE on vLLM 0.28, non-streaming:
before thinking on reasoning=202 content="391"
thinking off reasoning="391" content="391" <- duplicated
after thinking on reasoning=192 content="391"
thinking off reasoning="" content="391"
Forward the kwargs the prompt was rendered with, which is what vLLM's own
OpenAI server does; parsers that do not accept the argument keep the plain
constructor.
_split_reasoning() covers the older parser shape, which has no initial state
to set. It only reclassifies when the parser exposes a start/end token pair
and neither the completion nor the prompt ever opened a reasoning block.
Truncated reasoning (block open, end token never arrived) stays reasoning,
and parsers without that token pair are left untouched.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
#11772 exempted Temperature from the zero-filter in both backend adapters,
because proto3 has no field presence and an explicit 0 is indistinguishable
from "unset". Seed has exactly the same property and is still filtered:
if proto_field != "Temperature" and value in (None, 0, 0.0, [], False, ""):
continue
A caller pinning `"seed": 0` for a reproducible run therefore gets a random
seed instead, with no error and no log line — the one case where the failure
is invisible precisely because the request looked deliberate.
Both adapters now share a named tuple of fields whose zero is meaningful, so
the next one is added in one place rather than as a second special case.
Deliberately left filtered: top_k, top_p, min_p and the penalties. Their zero
is not a value a caller means — sglang disables top_k with -1, not 0, so
forwarding 0 there would turn a default into an invalid argument.
Verified on the sglang backend (Qwen3.5-MoE, arm64): with the temperature fix
alone, two identical requests at temperature 0 are byte-identical, but pinning
seed 0 has no effect until this change.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
The Intel backend installs PyTorch XPU wheels, but Qwen ASR only
checked CUDA and MPS. Every Intel model therefore loaded on the CPU.
Select XPU when available and place the model on xpu:0. Keep the
existing CUDA, MPS, and CPU placement behavior.
Assisted-by: Codex:GPT-5 [apply_patch] [gh]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
WhisperX silently returned a plain transcript when diarization lacked
the Hugging Face token required to load pyannote. Reject that request
clearly so callers do not mistake missing speaker labels for a
successful diarization.
Convert WhisperX seconds to the nanosecond duration unit used by the
transcription API.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
- the PR template's 'Signed commits' anchor pointed at a CONTRIBUTING
section that does not exist; repointed at the Commit messages
section
- the longcat-video backend README linked a docs page that was never
committed; replaced the dead link with plain text
- formal-verification/README.md used ../../../ for five in-repo
packages (escaping the repo root); fixed to ../
Two independent breakages on master make every open pull request red,
for reasons unrelated to the changes under review.
The e2e backend suite stopped compiling. Reply.message is `bytes` in
backend.proto, so res.GetMessage() returns []byte, and strings.ToUpper
wants a string. Every other call site in the file already converts.
tests/e2e-backends sits behind a build tag, so `go build ./...` never
compiled it and the breakage reached master unnoticed.
The darwin vllm build stopped resolving. Upstream vllm-metal deleted
its old dev tags and re-versioned to track the vLLM release it targets,
so the pinned wheel 404s. The coupled vLLM release also moved out of
upstream's install.sh into .github/vllm-release-tag.commit, and the
wheel's platform tag moved from macosx_11_0 to macosx_15_0.
Read the wheel name from the release's own asset listing rather than
composing it from a hardcoded platform segment, so a platform-tag
change cannot silently 404 again, and resolve the vLLM version from
the new metadata file with a fallback to the legacy installer. The
bump script and the extractor learn the same two-source lookup, so the
next nightly run converges on the pin checked in here instead of
reintroducing the break.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add complete vLLM and SGLang entries with their exact tool parsers. Preserve an explicit zero temperature in both backend adapters.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Those backends only forwarded the flag when it was "true", so "false"
never reached apply_chat_template and Qwen3 kept thinking on.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
dependabot[bot]