The Go side already forwards the request's effort as
Metadata["reasoning_effort"], but the sglang backend only read
enable_thinking: every level got the same model-level thinking_budget,
so low/medium/high produced identical reasoning.
- New model option `effort_budgets:low=…;medium=…;high=…` picks the
budget of the request's level; unlisted levels and requests without an
effort keep `thinking_budget`. Parsing reuses _parse_thinking_budget.
- A level listed in effort_budgets turns reasoning on for that request,
so levels the Go side does not know cannot silently disable it.
- ApplyReasoningEffort now treats `xhigh` and `max` as thinking levels.
Before, they left DisableReasoning unset, so with a model default of
"off" asking for more reasoning produced none.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 879c686ab1)
Mock the request constructor so fake parser tests can run on older
vLLM profiles without ParserManager. Document the streaming version
requirement and fallback behavior.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit ff0130fa89)
With a reasoning parser configured, the streaming path sent every raw
text delta as ChatDelta.content. LocalAI uses the backend's ChatDeltas as
they are, so with thinking enabled the reasoning arrived as
delta.content, and the final chunk then repeated the whole answer and
the reasoning a second time. Non-streaming requests were split correctly.
Use vLLM's composed parser (ParserManager.get_parser, vLLM >= 0.30), the
one vLLM's own OpenAI server streams with: one instance per request,
built with the request's chat_template_kwargs (enable_thinking), fed
each delta via parse_delta. Reasoning, content and tool calls go out in
their own fields as they are produced; the final chunk carries only
token counts and logprobs.
If the parser raises mid-stream, the rest of the output is buffered and
the final chunk sends what the full-text parse has beyond what was
already streamed (suffixes per field and per tool-call index). Where the
full parse disagrees with what went out, nothing is guessed and the
mismatch is logged.
Only active when a reasoning_parser is set; without one, or on a vLLM
without ParserManager, streaming is unchanged.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 5392eb813e)
Both Python backends start from a fixed default (sglang max_new_tokens=200,
vLLM max_tokens=200) and only replace it when the request carries Tokens.
OpenAI clients that leave max_tokens out (several agent frameworks do) were
silently truncated after 200 tokens, and the response still said
finish_reason "stop": the Go side only reports "length" against a budget
the client requested, so it cannot see a backend-side cap.
Use None instead, which allows generation up to the remaining context.
That is what sglang's own OpenAI server does when max_tokens is omitted
(the scheduler turns None into the remaining request capacity; dropping
the key would select SamplingParams' own default of 128), and vLLM treats
max_tokens=None the same way. A requested max_tokens still wins.
Tests check both cases and that the None default passes each library's own
parameter validation.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 5bdab54d97)
_build_sampling_params constructed SamplingParams and then assigned the
request values with setattr. protobuf repeated fields (StopPrompts) stay
RepeatedScalarContainer that way, and __post_init__ never sees them:
- vLLM's argument checks assert `isinstance(self.stop, list)` and the
msgpack encoder that ships the params to the engine core cannot
serialise the container, so every request with `stop` failed
(HTTP 500, EngineGenerateError), streaming or not.
- the stop-string hold-back buffer (output_text_buffer_length) was never
set, so the start of a multi-character stop string could be streamed;
seed -1 was not mapped to None.
Collect the values, turn repeated fields into lists and construct
SamplingParams once, as vLLM's OpenAI server does. Values vLLM rejects
(e.g. a penalty out of range, an empty stop string) now fail with vLLM's
validation error instead of being set unchecked.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 48d11fcf74)
mlx-vlm v0.4.4 attaches the image placeholder to the last message that is
not system or assistant. After a tool call that is the tool result, which
the chat template renders as text, so the next turn fails with "Image
features and image tokens do not match: tokens: 0". v0.6.12 keeps the
placeholder on the user message that carried the image.
The v0.4.4 pin (#9568) worked around the mlx-cuda wheels stopping at
0.31.1. They are at 0.32.4 now, and v0.6.12 (mlx>=0.32.0) resolves with
mlx[cuda12] and mlx[cuda13]. v0.7.x would need backend changes: it drops
mlx_vlm.tool_parsers, which backend.py imports.
Signed-off-by: gman <genx7up@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
* feat(sound): expand simple prompts with chat models
Allow callers to select an installed chat model for sound prompt
expansion. Validate its JSON before audio generation and preserve
explicit instrumental and language controls. Keep backend defaults
when expansion is omitted, including native Python ACE-Step sampling.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(sound): map expansion by loaded audio family
Map neutral expanded fields where the resolved audio family is known.
Keep unmarked requests unchanged and reject unsupported expansion before
inference. Clear YuE2 and HeartMuLa transcripts to prevent lyric fallback.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(sound): choose a prompt expansion model
Require an explicit Simple choice and remember it separately from chat.
Clear unavailable choices instead of silently switching language models.
Preserve the choice in history and restore it when editing a past result.
Keep Backend default and Advanced requests compatible with existing APIs.
Document family support and align the published optional request schema.
Verify selection, request fields, persistence and history in the browser.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(sound): authorize both generation models
Check the resolved audio model and explicit expansion model before
loading either. Permission lookup failures stop generation.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(sound): enforce JSON during prompt expansion
Constrain expansion output with the existing JSON grammar before decoding.
Require a llama-cpp chat model so unsupported backends cannot ignore it.
Keep strict field validation and fail before generating audio.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Pocket TTS ships one model per language, but LoadModel called
TTSModel.load_model() with no arguments, so the library always fell back
to its English model whatever the model config said. Read a `language`
option and pass it to load_model, keeping the library default when it is
not set. Document the option and add unit tests that mock the loader.
Fixes#12759
Assisted-by: Claude:claude-opus-5-5
The sd_embed branch dropped negative_prompt for every pipeline, which
discarded the user's negative prompt for Flux, where no negative embeds
are produced. Only drop it when negative_prompt_embeds is present, and
move the check to a helper that has unit tests.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
AudioTranscription caught every exception and returned an empty
TranscriptResult. A failed diarization step therefore discarded a
transcript that was already finished: with an HF token that has not
accepted the terms of the gated pyannote pipeline, the download fails
with 403 and every transcription came back as an empty text with
HTTP 200.
Diarization now degrades: if it fails, the transcript is returned
without speaker labels and the reason is logged. Any other failure
aborts the call with INTERNAL instead of pretending success with an
empty text.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
FunASR left Transformers and Hugging Face Hub unconstrained, which let uv backtrack to tokenizers 0.10.3 without Python 3.12 wheels. Keep Transformers on the supported 4.x range, including the Intel upgrade profile.\n\nAssisted-by: nib:gpt-5.6-sol\nSigned-off-by: Ettore Di Giacinto <mudler@localai.io>
* sglang backend: pass through thinking_budget + require_reasoning
sglang's raw Engine.async_generate() API (which this backend calls
directly, bypassing sglang's own OpenAI server) supports a precise,
tokenizer-derived reasoning-length budget via
sampling_params["custom_params"]["thinking_budget"] plus
require_reasoning=True, gated behind --enable-strict-thinking. Neither
was reachable through LocalAI: this backend built sampling_params only
from a fixed field mapping (temperature, top_p, ...) with no custom_params
key, and never passed require_reasoning to async_generate at all.
- LoadModel now reads a model-level "thinking_budget" option (same
mechanism as the existing tool_parser/reasoning_parser options), and
_build_sampling_params adds it as custom_params.thinking_budget on
every request when configured.
- _new_reasoning_parser already derives, from the rendered prompt, whether
the model's chat template pre-opened a reasoning block (Qwen3-style
templates append <think> to the prompt instead of letting the model
emit it) -- the same signal sglang's own OpenAI server computes from
per-template config to decide require_reasoning. This backend has no
template manager, so it now returns that signal too and _predict
forwards it to async_generate(require_reasoning=...).
Verified against production (NVFP4, sm_121, Qwen3.6-35B-A3B) via a raw
Engine.async_generate() call bypassing this backend: 301 reasoning
tokens against a 300-token budget, clean completion, ~27s. Not yet
verified through this backend's own gRPC path end-to-end (no local
CUDA/sglang environment available here) -- existing + new unit tests in
test.py cover the pure-Python merge/passthrough logic only.
Scope note: require_reasoning is derived only from the existing
prompt-suffix heuristic, not sglang's full per-template
_get_reasoning_from_request decision tree (minimax-m3/hunyuan special
cases etc.) -- this backend has no template manager to evaluate that
tree against, and the prompt-suffix check is the one heuristic already
validated in this file (test_reasoning_parser_forced_when_template_prefills_think_tag).
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
* sglang backend: honour a model-level reasoning_default
A model YAML can already carry "parameters: reasoning_effort:", but that
value only reaches this backend when a *caller* sets it per request (the Go
side turns it into Metadata["enable_thinking"]). As a model-level default it
is silently dropped: a config reading "reasoning_effort: none" still produces
full reasoning on every request, so the config says one thing and the model
does another.
That gap is expensive in practice. On a self-hosted Qwen3.6-35B-A3B the
reasoning phase consumed the entire max_tokens budget before any content was
produced - 90% of code completions came back empty at max_tokens=768, and the
server log filled with "backend produced only reasoning, retrying". The
config looked like reasoning was off the whole time.
This adds "reasoning_default:off" (or ":on") on the same model-level
options: mechanism as thinking_budget. A per-request value always wins; the
default only fills in when the request is silent.
Measured on the stack above (sglang 0.5.20, NVFP4, GB10/sm_121) after
applying it:
default (nothing set) -> 0 chars reasoning, 27 tokens
"reasoning_effort": "none" -> 0 chars reasoning, 27 tokens
metadata enable_thinking=true -> capped at the 512-token thinking_budget,
541 tokens total, finish_reason stop
Tests: three cases added to backend/python/sglang/test.py covering the
default, per-request override in both directions, and the unconfigured case
(which must leave the template untouched).
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
* sglang backend: validate thinking_budget instead of crashing LoadModel
Addresses the review on this PR:
- `int(thinking_budget)` raised on values like "5000.0" or "abc" and took
LoadModel down. The option is now parsed by _parse_thinking_budget():
integral numbers in any spelling are accepted, anything else is ignored
with a warning on stderr.
- Zero and negative budgets are ignored with a warning instead of being
passed to sglang, where they have no defined meaning. Turning reasoning
off is what reasoning_default:off is for.
- A load-time warning when thinking_budget is set but enable_strict_thinking
is not in engine_args, since sglang then ignores the budget silently.
- Tests for integral spellings, unset, zero, negative, non-integer and the
strict-thinking warning.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
* docs(sglang): explain reasoning options
Document the reasoning budget, strict-thinking requirement, and
precedence of request metadata over the model-level default.
Also note that the budget has to stay well below max_tokens (otherwise
it never triggers and the reply can end up empty), and that
POST /models/reload or a backend-only restart does not pick up changed
options; LocalAI itself has to be restarted.
Assisted-by: Codex:GPT-6
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
* docs(sglang): clarify configuration reloads
Distinguish rereading model configuration from updating a running backend. Keep the full LocalAI restart recommendation for changed reasoning options.
Assisted-by: Codex:GPT-6
* sglang backend: only pass require_reasoning when sglang supports it
Engine.async_generate() gained the require_reasoning keyword in sglang
0.5.13 and takes no **kwargs. The CPU profile builds v0.5.11 from source
and the other profiles only set a >=0.5.11 floor, so passing the keyword
unconditionally made every request fail with TypeError. Detect support
once at import time, as the file already does for sampling_seed.
enable_strict_thinking first appears in sglang 0.5.12; fix the comment.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
---------
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <localai-org-maint-bot@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
sglang 0.5.20 moved its config tier from dataclasses to msgspec.Struct
(sgl-project/sglang#38753). _apply_engine_args validates engine_args keys
via dataclasses.fields(ServerArgs), which raises TypeError there. That
call runs on every LoadModel, so no model loads at all on the sglang
backend once sglang >= 0.5.20 is installed, and the error surfaces as a
generic "Unexpected <class 'TypeError'>" that does not name the cause.
Introspect both shapes: msgspec structs carry their field names in
__struct_fields__, so key validation and the close-match suggestion keep
working, and older dataclass-based sglang stays supported.
Adds a test that pins the msgspec path with a stand-in, so it is covered
regardless of which sglang version is installed.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
Use a regular (non-editable) pip/uv install so the package lands in the
venv site-packages. An editable finder records the builder source path,
which breaks after the backend is copied out of the image (#9162).
Adds a regression test (scripts/build/vllm-omni-install_test.sh) that
verifies imports survive relocation with a regular install and fail with
an editable install.
Supersedes #12040 (DCO not signed by contributor).
Assisted-by: MAKI:regolo/glm5.2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The CUDA 13 profile resolves PyTorch 2.14. Its headers cannot build the
current FlashAttention source release under C++17, so the full backend
matrix cannot publish the image.
Leave FlashAttention optional on CUDA 13. The backend already retries
model loading with SDPA when FlashAttention is unavailable.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The CUDA 13 FlashAttention build still exhausts hosted-runner memory with a single ninja worker because nvcc can compile multiple threads internally. Limit nvcc to one thread for that profile and guard the setting in the backend test script.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
SciPy 1.18 requires Python 3.12 or newer. Keep this backend on a
compatible portable Python for Linux and macOS builds.
Assisted-by: Codex:gpt-5.6 [Codex]
(cherry picked from commit 70bf6d4a3a)
Adds FunASR/SenseVoice as a Python backend for speech-to-text with
support for CPU, CUDA 12/13, ROCm, Intel SYCL, L4T, and Apple MPS.
Co-authored-by: xingyifeng <xingyifeng@users.noreply.github.com>
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The editable install records the backend build path, which does not exist after LocalAI relocates the packaged backend. Add the runtime source directory to PYTHONPATH so inference modules remain importable.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Expose diffusers audio pipelines through the existing sound-generation RPC. AudioLDM2 can now return PCM WAV output from the model gallery without a separate backend.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The pinned mlx-video package requires Python 3.11 or newer, while an empty PYTHON_VERSION selected the backend helper default of 3.10. Pin the available 3.11.13 portable runtime for the Darwin package build.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a Darwin-only MLX-Video backend for LTX-2 and converted Wan checkpoints, expose it through the existing video API, and wire packaging, discovery, tests, docs, and an example.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Prefer an explicitly configured Triton assembler, otherwise use the executable ptxas from CUDA_HOME so torch.compile can target GPU architectures newer than Triton bundled tooling.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Keep the relocated upstream source importable, select CUDA 13 PyTorch wheels instead of the aarch64 CPU fallback, and decode reference audio without torchcodec, which has no Linux arm64 wheels.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The standalone Python release used by libbackend does not publish a 3.11.18 artifact. Pin Whisper-Medusa to the available 3.11.13 build and cover the generated download URL.
Assisted-by: Codex:gpt-5 [systematic-debugging]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
pos-ei-don