Qwen3-style chat templates append the opening <think> tag to the *prompt*
when thinking is enabled. The model therefore never generates it and emits
only the reasoning text plus the closing </think>.
sglang's ReasoningParser keys off the opening tag:
in_reasoning = self._in_reasoning or self.think_start_token in text
if not in_reasoning:
return StreamingParseResult(normal_text=text)
so with such a template the entire completion — reasoning and answer, the
raw </think> in between — is returned as content and reasoning_content
stays empty, no matter how reasoning_parser is configured.
sglang's own OpenAI server handles this via
force_reasoning = (self.template_manager.force_reasoning
or self._get_reasoning_from_request(request))
This backend has no template manager, so derive the same signal from the
rendered prompt: if it ends with the detector's think_start_token, the tag
was prefilled and the parser is constructed with force_reasoning=True.
Structured decoding is the exception, and it matters: a grammar applies
from the first token, so the model cannot emit the closing tag even though
the template opened the block. The whole completion is schema output and
belongs in content — forcing there files it as reasoning and returns an
empty answer. Measured against a JSON-schema code audit: 10107 characters
of "reasoning", zero content. sglang's own server keeps the two apart for
the same reason; its grammar backend owns the reasoning prefix when a
reasoning parser is configured.
force_reasoning is only passed when it is meant to be True, so detector
defaults (DeepSeek-R1 already defaults to True) are untouched, and a
prompt without a prefilled tag behaves exactly as before — which matters,
because forcing unconditionally makes an answer generated with thinking
off disappear into reasoning_content.
The construction is factored into _new_reasoning_parser() so the streaming
and non-streaming paths, which previously built the parser separately,
cannot drift apart.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
With `template.use_tokenizer_template: true` the sglang and vllm backends
render the prompt themselves via `tokenizer.apply_chat_template()`, and they
hand it plain string content. A chat template only emits the model's own media
tokens when the content is a list of parts, so the rendered prompt carries no
`<|vision_start|><|image_pad|><|vision_end|>`. The pixels do reach the engine
(`image_data` / `multi_modal_data`), but both engines locate them by scanning
the prompt for that token, so they are discarded silently: HTTP 200, no
warning, and the model answers as if no image had been attached.
Add `attach_media_parts()` to the shared `python_utils` helper and call it in
both backends: the last user turn is rebuilt as
`[{"type": "image"} * n, {"type": "video"} * n, {"type": "text", ...}]` before
templating, which makes the template emit the placeholders. The pixels keep
travelling out of band exactly as before.
Text-only requests are untouched - with no media the helper returns None and
the original string-content path runs unchanged. If a template cannot iterate
content parts (a text-only model), the parts render is caught and the request
falls back to the previous string-content prompt instead of failing.
Signed-off-by: Tai An <antai12232931@outlook.com>
vLLM's engine-based reasoning parsers derive their initial state from the
chat template kwargs. Qwen3Parser:
chat_kwargs = kwargs.get("chat_template_kwargs", {}) or {}
self.thinking_enabled = chat_kwargs.get("enable_thinking", True)
Constructed as ReasoningParser(tokenizer) the flag defaults to True, so the
parser starts in the REASONING state. A completion produced with thinking
disabled contains no tags at all, and every reasoning parser shape then
reports the whole answer as reasoning:
- engine-based parsers classify it by initial state;
- BaseThinkingReasoningParser hits its documented "may not generate start
token" fallback and returns (model_output, None).
Either way `content = c if c is not None else generated_text` turns that
into a duplicate: a Qwen3 model answering "391" with thinking off comes back
as reasoning_content="391" AND content="391".
Measured against Qwen3.5-MoE on vLLM 0.28, non-streaming:
before thinking on reasoning=202 content="391"
thinking off reasoning="391" content="391" <- duplicated
after thinking on reasoning=192 content="391"
thinking off reasoning="" content="391"
Forward the kwargs the prompt was rendered with, which is what vLLM's own
OpenAI server does; parsers that do not accept the argument keep the plain
constructor.
_split_reasoning() covers the older parser shape, which has no initial state
to set. It only reclassifies when the parser exposes a start/end token pair
and neither the completion nor the prompt ever opened a reasoning block.
Truncated reasoning (block open, end token never arrived) stays reasoning,
and parsers without that token pair are left untouched.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
#11772 exempted Temperature from the zero-filter in both backend adapters,
because proto3 has no field presence and an explicit 0 is indistinguishable
from "unset". Seed has exactly the same property and is still filtered:
if proto_field != "Temperature" and value in (None, 0, 0.0, [], False, ""):
continue
A caller pinning `"seed": 0` for a reproducible run therefore gets a random
seed instead, with no error and no log line — the one case where the failure
is invisible precisely because the request looked deliberate.
Both adapters now share a named tuple of fields whose zero is meaningful, so
the next one is added in one place rather than as a second special case.
Deliberately left filtered: top_k, top_p, min_p and the penalties. Their zero
is not a value a caller means — sglang disables top_k with -1, not 0, so
forwarding 0 there would turn a default into an invalid argument.
Verified on the sglang backend (Qwen3.5-MoE, arm64): with the temperature fix
alone, two identical requests at temperature 0 are byte-identical, but pinning
seed 0 has no effect until this change.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
A quantization job that runs with no client on its progress stream stays
"queued" forever, in the API and in state.json, while the finished artifact
sits on disk. state.json was written once by StartJob, and the only code that
advanced a job afterwards lived inside the stream callback of StreamProgress,
so job state depended on somebody watching it.
The backend's progress stream cannot simply gain a second reader: each job owns
one queue.Queue and QuantizationProgress pops from it, so two consumers split
the updates rather than both seeing them. The stream has to be opened exactly
once per job.
StartJob now starts watchProgress on the application context (the request
context is done as soon as the handler returns). That goroutine is the single
reader: it applies each update to the job -- in the cross-replica store and in
state.json, terminal statuses still winning over late updates -- and republishes
it in-process. StreamProgress becomes a pure reader over that fan-out and no
longer loads a backend or opens a stream. A client attaching to a job that has
already finished, including one hydrated from disk after a restart, gets a final
event built from the stored job instead of blocking.
Two paths used to end a client's stream by breaking the gRPC connection and now
release it explicitly: StopJob kills the backend, so it publishes the stopped
event itself; and a stream that ends without a terminal update means the backend
is gone, so the job is recorded as failed rather than left running forever.
Signed-off-by: Tai An <antai12232931@outlook.com>
The Intel backend installs PyTorch XPU wheels, but Qwen ASR only
checked CUDA and MPS. Every Intel model therefore loaded on the CPU.
Select XPU when available and place the model on xpu:0. Keep the
existing CUDA, MPS, and CPU placement behavior.
Assisted-by: Codex:GPT-5 [apply_patch] [gh]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
WhisperX silently returned a plain transcript when diarization lacked
the Hugging Face token required to load pyannote. Reject that request
clearly so callers do not mistake missing speaker labels for a
successful diarization.
Convert WhisperX seconds to the nanosecond duration unit used by the
transcription API.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
importNpmLock turns the same-version hono override into a file: tarball
that conflicts with the direct dependency (EOVERRIDE). Pass
--legacy-peer-deps so the flake build can proceed without
touching package.json (open #11633).
Fixes#11804
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
LLM-jp 4 provides a recent Japanese and English reasoning model on the supported Llama architecture. Add its official Q4 and BF16 GGUF builds so hosts can select the fidelity that fits.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4_K_XL and Q8_K_XL llama.cpp builds with the shared vision projector. Enable the preserved MTP head for speculative decoding.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the text-only Q4_K_M build for private red-team, blue-team, and security operations workloads. Configure the supported Gemma 4 model for llama.cpp with its verified Hugging Face checksum.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4_K_M and Q8_0 llama.cpp builds plus the official F16 GGUF for Mixedbread mxbai-embed-large-v1.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The new importance-matrix builds reduce Laguna S 2.1 from the existing 96 GB default to 73.9 GB or 54.4 GB. Add both as selectable llama.cpp variants.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
DFM Mimir is a new permissively licensed Danish and English HRM-Text model. LocalAI already ships a vLLM version with native support for its architecture.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the artifact-backed VoxCPM2 model for multilingual speech synthesis, voice design, and controllable voice cloning.
Assisted-by: Codex:gpt-5.6 [Codex]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the Q4_K_M and Q8_0 GGUF builds for a compact Qwen3.5-based reasoning and tool-use distillation. The variant pair lets LocalAI choose higher fidelity when the host has enough memory while keeping the Q4 build as the installable fallback.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a compact NVFP4 build with its embedded MTP head and vision
projector. This gives Blackwell-capable llama.cpp hosts a speculative
option alongside the existing Q4 and Q8 builds.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the original-dtype GGUF for LocalAI\x27s audio.cpp backend. The entry exposes multilingual voice cloning and records the model license.\n\nAssisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the compact LightOnOCR model to the OCR section with Q8_0 and F16 builds. Each build uses its matching vision projector.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the official F16 GGUF build of the compact Apache-2.0 Qwen3-family chat model with its native 2K context limit.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
- api-errors.md documented LOCALAI_SUBTLEKEY_COMPARISON (missing the
KEY underscore); the code defines LOCALAI_SUBTLE_KEY_COMPARISON, so
the documented variable silently did nothing
- cli-reference.md documented a --csrf flag / $LOCALAI_CSRF env that
do not exist, with inverted semantics; the actual flag is
--disable-csrf (LOCALAI_DISABLE_CSRF), 'Disable CSRF middleware
(enabled by default)'
- the PR template's 'Signed commits' anchor pointed at a CONTRIBUTING
section that does not exist; repointed at the Commit messages
section
- the longcat-video backend README linked a docs page that was never
committed; replaced the dead link with plain text
- formal-verification/README.md used ../../../ for five in-repo
packages (escaping the repo root); fixed to ../
- middleware.md: the 'default detector' link used #instance-wide-defaults;
the heading is 'Instance-wide default detector'
- the advanced/reference landing pages linked an ../installation/
directory that does not exist in docs/content; dropped the dead
bullets (deployment content lives under getting-started)
* docs(llama-cpp): clarify multimodal speculative decoding
Update the speculative decoding guidance now that modern llama.cpp backends can combine mmproj-based vision with speculative decoding, including MTP. Document compatibility checks, draft acceptance statistics, VRAM tradeoffs, and a combined configuration example.
Assisted-by: Codex:GPT-5.6-Sol [gh] [OpenStack] [Docker]
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
* docs(llama-cpp): clarify multimodal MTP references
Distinguish the upstream change that removed the general multimodal speculative restriction from the later change that added MTP with explicit vision compatibility.
Assisted-by: Codex:GPT-5.6-Sol [gh] [Docker]
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
---------
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
The AI-assistant policy says an AI agent must never add a Signed-off-by
trailer, because only a human can certify the DCO. That is right for the
case it was written for: an assistant helping a contributor who then
signs off themselves.
It does not fit automation a maintainer runs. Those pull requests have no
human submitter, so nothing ever signs and the DCO check blocks them
permanently. Sixty-one open pull requests from the maintenance bot are in
exactly that state, every one of them correctly following the documented
rule.
Carve out the case: automation a maintainer operates signs off with that
maintainer's identity. The maintainer certifies the DCO, as they do for a
commit they typed by hand, because they configured the automation, own
its output, and take responsibility on merge. The Assisted-by trailer
still records that a model wrote the code, so provenance is unchanged.
Keep the exception narrow. An assistant helping an outside contributor
still must not sign off, and a bot must not sign for anyone but its
operator, including on a contributor's branch it pushes to.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Two independent breakages on master make every open pull request red,
for reasons unrelated to the changes under review.
The e2e backend suite stopped compiling. Reply.message is `bytes` in
backend.proto, so res.GetMessage() returns []byte, and strings.ToUpper
wants a string. Every other call site in the file already converts.
tests/e2e-backends sits behind a build tag, so `go build ./...` never
compiled it and the breakage reached master unnoticed.
The darwin vllm build stopped resolving. Upstream vllm-metal deleted
its old dev tags and re-versioned to track the vLLM release it targets,
so the pinned wheel 404s. The coupled vLLM release also moved out of
upstream's install.sh into .github/vllm-release-tag.commit, and the
wheel's platform tag moved from macosx_11_0 to macosx_15_0.
Read the wheel name from the release's own asset listing rather than
composing it from a hardcoded platform segment, so a platform-tag
change cannot silently 404 again, and resolve the vLLM version from
the new metadata file with a fallback to the legacy installer. The
bump script and the extractor learn the same two-source lookup, so the
next nightly run converges on the pin checked in here instead of
reintroducing the break.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(downloader): make file:// installs reachable again
DownloadFileWithContext already has a branch that copies from a local
file, but it could never run. Before reaching it the function decides
whether the destination is fetchable with
} else if !os.IsNotExist(err) || !URI(url).LooksLikeHTTPURL() {
and LooksLikeHTTPURL is http(s) only, so any URI resolving to a local
path is rejected there. Falling through requires the destination to be
missing AND the source to be an HTTP URL, which a file:// source never
is -- leaving the local-source branch below unreachable.
A first import always has a missing destination, so importing
file:///path/to/model.gguf always failed, with an error that listed
file:// among the supported schemes (#11701).
Name the local-source condition once as URI.hasLocalSource and use it
both to admit the destination and to pick the source, so the two cannot
drift apart again.
Signed-off-by: Tai An <antai12232931@outlook.com>
The recommended sampling parameters for a model family were applied at
install and then never took effect. Two things went wrong on the way to
disk.
They were written as top level keys. ModelConfig embeds PredictionOptions
under the "parameters" yaml key, so temperature, top_p, top_k, min_p,
repeat_penalty and presence_penalty are only read from there. At the top
level they parse without error and are then ignored for the life of the
model.
They were also merged in after the YAML had already been marshalled. The
only re-marshal sat behind the artifact binding, which an entry carrying
files: never reaches, so for those entries the defaults were computed and
then dropped before anything was written.
Neither failure was visible in normal use. ApplyInferenceDefaults runs
again at load time and fills the same values from the same table, so the
model ends up tuned correctly while the file on disk pins nothing. It
surfaces when someone edits one of those values expecting it to win, or
when a family is absent from inference_defaults.json and there is nothing
to refill from.
Both install paths are covered: an entry carrying files:, and one that
binds a primary artifact instead.
The empty base spec asserted that the authored parameters block landed
verbatim. It now checks the authored keys individually, because the family
defaults are merged into that same block.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
pengmin