* ⬆️ Update leejet/stable-diffusion.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(stablediffusion): adapt streaming options
Upstream now selects segmented weight streaming automatically and removes the stream_layers field. Keep the old LocalAI option as a no-op for existing model configurations.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link upstream image helpers
The ds4 bump adds vision calls to the engine object. Link the new image preprocessing object into every backend target.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Qwen3-style chat templates append the opening <think> tag to the *prompt*
when thinking is enabled. The model therefore never generates it and emits
only the reasoning text plus the closing </think>.
sglang's ReasoningParser keys off the opening tag:
in_reasoning = self._in_reasoning or self.think_start_token in text
if not in_reasoning:
return StreamingParseResult(normal_text=text)
so with such a template the entire completion — reasoning and answer, the
raw </think> in between — is returned as content and reasoning_content
stays empty, no matter how reasoning_parser is configured.
sglang's own OpenAI server handles this via
force_reasoning = (self.template_manager.force_reasoning
or self._get_reasoning_from_request(request))
This backend has no template manager, so derive the same signal from the
rendered prompt: if it ends with the detector's think_start_token, the tag
was prefilled and the parser is constructed with force_reasoning=True.
Structured decoding is the exception, and it matters: a grammar applies
from the first token, so the model cannot emit the closing tag even though
the template opened the block. The whole completion is schema output and
belongs in content — forcing there files it as reasoning and returns an
empty answer. Measured against a JSON-schema code audit: 10107 characters
of "reasoning", zero content. sglang's own server keeps the two apart for
the same reason; its grammar backend owns the reasoning prefix when a
reasoning parser is configured.
force_reasoning is only passed when it is meant to be True, so detector
defaults (DeepSeek-R1 already defaults to True) are untouched, and a
prompt without a prefilled tag behaves exactly as before — which matters,
because forcing unconditionally makes an answer generated with thinking
off disappear into reasoning_content.
The construction is factored into _new_reasoning_parser() so the streaming
and non-streaming paths, which previously built the parser separately,
cannot drift apart.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
With `template.use_tokenizer_template: true` the sglang and vllm backends
render the prompt themselves via `tokenizer.apply_chat_template()`, and they
hand it plain string content. A chat template only emits the model's own media
tokens when the content is a list of parts, so the rendered prompt carries no
`<|vision_start|><|image_pad|><|vision_end|>`. The pixels do reach the engine
(`image_data` / `multi_modal_data`), but both engines locate them by scanning
the prompt for that token, so they are discarded silently: HTTP 200, no
warning, and the model answers as if no image had been attached.
Add `attach_media_parts()` to the shared `python_utils` helper and call it in
both backends: the last user turn is rebuilt as
`[{"type": "image"} * n, {"type": "video"} * n, {"type": "text", ...}]` before
templating, which makes the template emit the placeholders. The pixels keep
travelling out of band exactly as before.
Text-only requests are untouched - with no media the helper returns None and
the original string-content path runs unchanged. If a template cannot iterate
content parts (a text-only model), the parts render is caught and the request
falls back to the previous string-content prompt instead of failing.
Signed-off-by: Tai An <antai12232931@outlook.com>
vLLM's engine-based reasoning parsers derive their initial state from the
chat template kwargs. Qwen3Parser:
chat_kwargs = kwargs.get("chat_template_kwargs", {}) or {}
self.thinking_enabled = chat_kwargs.get("enable_thinking", True)
Constructed as ReasoningParser(tokenizer) the flag defaults to True, so the
parser starts in the REASONING state. A completion produced with thinking
disabled contains no tags at all, and every reasoning parser shape then
reports the whole answer as reasoning:
- engine-based parsers classify it by initial state;
- BaseThinkingReasoningParser hits its documented "may not generate start
token" fallback and returns (model_output, None).
Either way `content = c if c is not None else generated_text` turns that
into a duplicate: a Qwen3 model answering "391" with thinking off comes back
as reasoning_content="391" AND content="391".
Measured against Qwen3.5-MoE on vLLM 0.28, non-streaming:
before thinking on reasoning=202 content="391"
thinking off reasoning="391" content="391" <- duplicated
after thinking on reasoning=192 content="391"
thinking off reasoning="" content="391"
Forward the kwargs the prompt was rendered with, which is what vLLM's own
OpenAI server does; parsers that do not accept the argument keep the plain
constructor.
_split_reasoning() covers the older parser shape, which has no initial state
to set. It only reclassifies when the parser exposes a start/end token pair
and neither the completion nor the prompt ever opened a reasoning block.
Truncated reasoning (block open, end token never arrived) stays reasoning,
and parsers without that token pair are left untouched.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
#11772 exempted Temperature from the zero-filter in both backend adapters,
because proto3 has no field presence and an explicit 0 is indistinguishable
from "unset". Seed has exactly the same property and is still filtered:
if proto_field != "Temperature" and value in (None, 0, 0.0, [], False, ""):
continue
A caller pinning `"seed": 0` for a reproducible run therefore gets a random
seed instead, with no error and no log line — the one case where the failure
is invisible precisely because the request looked deliberate.
Both adapters now share a named tuple of fields whose zero is meaningful, so
the next one is added in one place rather than as a second special case.
Deliberately left filtered: top_k, top_p, min_p and the penalties. Their zero
is not a value a caller means — sglang disables top_k with -1, not 0, so
forwarding 0 there would turn a default into an invalid argument.
Verified on the sglang backend (Qwen3.5-MoE, arm64): with the temperature fix
alone, two identical requests at temperature 0 are byte-identical, but pinning
seed 0 has no effect until this change.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
The Intel backend installs PyTorch XPU wheels, but Qwen ASR only
checked CUDA and MPS. Every Intel model therefore loaded on the CPU.
Select XPU when available and place the model on xpu:0. Keep the
existing CUDA, MPS, and CPU placement behavior.
Assisted-by: Codex:GPT-5 [apply_patch] [gh]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
WhisperX silently returned a plain transcript when diarization lacked
the Hugging Face token required to load pyannote. Reject that request
clearly so callers do not mistake missing speaker labels for a
successful diarization.
Convert WhisperX seconds to the nanosecond duration unit used by the
transcription API.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
- the PR template's 'Signed commits' anchor pointed at a CONTRIBUTING
section that does not exist; repointed at the Commit messages
section
- the longcat-video backend README linked a docs page that was never
committed; replaced the dead link with plain text
- formal-verification/README.md used ../../../ for five in-repo
packages (escaping the repo root); fixed to ../
Two independent breakages on master make every open pull request red,
for reasons unrelated to the changes under review.
The e2e backend suite stopped compiling. Reply.message is `bytes` in
backend.proto, so res.GetMessage() returns []byte, and strings.ToUpper
wants a string. Every other call site in the file already converts.
tests/e2e-backends sits behind a build tag, so `go build ./...` never
compiled it and the breakage reached master unnoticed.
The darwin vllm build stopped resolving. Upstream vllm-metal deleted
its old dev tags and re-versioned to track the vLLM release it targets,
so the pinned wheel 404s. The coupled vLLM release also moved out of
upstream's install.sh into .github/vllm-release-tag.commit, and the
wheel's platform tag moved from macosx_11_0 to macosx_15_0.
Read the wheel name from the release's own asset listing rather than
composing it from a hardcoded platform segment, so a platform-tag
change cannot silently 404 again, and resolve the vLLM version from
the new metadata file with a fallback to the legacy installer. The
bump script and the extractor learn the same two-source lookup, so the
next nightly run converges on the pin checked in here instead of
reintroducing the break.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ds4): build CUDA kernels for the target architecture
The ds4 backend compiled its CUDA objects with no -arch. Upstream's Makefile
leaves CUDA_ARCH empty and its `cuda` target refuses to build without one,
offering `cuda-spark` (sm_121) and `cuda-generic` (native) instead. We invoke
its object targets directly, which bypasses that guard, so nvcc fell back to
its default architecture and the kernels ran as JIT'd PTX on the real GPU.
On GB10 (sm_121) that silently corrupted inference: any prompt over roughly 128
tokens produced text unrelated to the input and never closed its thinking
block, so content came back empty and the chat showed only reasoning; longer
prompts failed with "cuda decode failed". It also cost close to two orders of
magnitude of prefill throughput. Measured on one box, same model, same prompt,
same GPU, upstream ds4 at the pinned commit, differing only in the nvcc flags:
make -B ds4 (archless, as we build it) garbage output 4.21 t/s
make cuda-spark (compute_121a/sm_121a) correct output 325.70 t/s
Select an architecture list from CUDA_MAJOR_VERSION, which the backend matrix
already declares for both ds4 cublas entries but Dockerfile.ds4 never forwarded.
Upstream's CUDA_ARCH takes a single value, so it cannot express the fat binary
these images need; NVCC_ARCH_FLAGS is overridden instead, since a command-line
assignment wins over its `:=`. The lists are copied from vllm-cpp rather than
invented so the two CUDA images cover the same GPUs, with l4t/arm64 covering
Orin, Thor and GB10. An empty CUDA_MAJOR_VERSION keeps upstream's `native`
behaviour for local developer builds, and no CI runner has a GPU to enumerate.
DS4_CUDA_HAVE_MXF4 is deliberately left unset: upstream defines it only for
single-arch sm_120/sm_121 builds and guards it with a plain #ifdef rather than
__CUDA_ARCH__, so it cannot be combined with older archs. It gates an optional
MXFP4 indexer fast path whose #ifndef branch returns 0 and falls back cleanly,
so omitting it costs speed on GB10, not correctness.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* test(ds4): cover the multi-batch prefill regression
The architecture fix has no automated guard: every existing e2e spec uses a
short prompt, and the miscompiled backend answered short prompts correctly.
The corruption only appears once a prompt spans more than one prefill batch,
so the whole suite passed against a backend that produced garbage in normal
use.
Add an opt-in "long_prefill" capability to the backend e2e suite that sends a
prompt well past one batch with a known needle and asserts the answer still
reflects it, and document in the ds4 guide why the build must never omit an
nvcc architecture, how to check which flags a configuration resolves to
without compiling, and how to run the new spec.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
---------
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
Propagate gRPC cancellation into DS4 prompt synchronization and poll it at decode boundaries.
Stop on failed stream writes and skip parser finalization and KV persistence for abandoned partial requests.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
Clamp requested generation to the usable context after prompt sync while preserving the legacy 256-token fallback for omitted limits.
Constrain each speculative MTP cycle to the remaining request budget so accepted tokens cannot advance beyond the visible output limit.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): link librdma from the static ggml-rpc build
ggml-rpc gained an Apple RDMA transport in this llama.cpp range and
declares its librdma dependency with target_link_options(ggml-rpc
PRIVATE "LINKER:-weak_library,..."). Link options are not a usage
requirement of a static library, so the llama-cpp-grpc variant, which
builds with BUILD_SHARED_LIBS=OFF, dropped the flag and left every
ibv_* symbol of transport-apple.cpp undefined when grpc-server linked
on darwin.
prepare.sh now re-declares the same weak link as INTERFACE on the
ggml-rpc target, so the flag reaches whoever links the static library.
The append is guarded on a marker for repeat runs, and on
GGML_RPC_RDMA_APPLE, which the turboquant and bonsai forks lack.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
DS4 appends the opening thinking marker to tokenizer-templated prompts, so generated text begins directly with reasoning bytes. Starting DsmlParser in TEXT therefore puts the reasoning and closing marker in visible content.
Start the parser in THINK for structured chat requests with thinking enabled in both Predict and PredictStream. Keep the default TEXT state for raw prompts and reasoning-off requests, and add incremental regression coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
