* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link upstream image helpers
The ds4 bump adds vision calls to the engine object. Link the new image preprocessing object into every backend target.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ds4): build CUDA kernels for the target architecture
The ds4 backend compiled its CUDA objects with no -arch. Upstream's Makefile
leaves CUDA_ARCH empty and its `cuda` target refuses to build without one,
offering `cuda-spark` (sm_121) and `cuda-generic` (native) instead. We invoke
its object targets directly, which bypasses that guard, so nvcc fell back to
its default architecture and the kernels ran as JIT'd PTX on the real GPU.
On GB10 (sm_121) that silently corrupted inference: any prompt over roughly 128
tokens produced text unrelated to the input and never closed its thinking
block, so content came back empty and the chat showed only reasoning; longer
prompts failed with "cuda decode failed". It also cost close to two orders of
magnitude of prefill throughput. Measured on one box, same model, same prompt,
same GPU, upstream ds4 at the pinned commit, differing only in the nvcc flags:
make -B ds4 (archless, as we build it) garbage output 4.21 t/s
make cuda-spark (compute_121a/sm_121a) correct output 325.70 t/s
Select an architecture list from CUDA_MAJOR_VERSION, which the backend matrix
already declares for both ds4 cublas entries but Dockerfile.ds4 never forwarded.
Upstream's CUDA_ARCH takes a single value, so it cannot express the fat binary
these images need; NVCC_ARCH_FLAGS is overridden instead, since a command-line
assignment wins over its `:=`. The lists are copied from vllm-cpp rather than
invented so the two CUDA images cover the same GPUs, with l4t/arm64 covering
Orin, Thor and GB10. An empty CUDA_MAJOR_VERSION keeps upstream's `native`
behaviour for local developer builds, and no CI runner has a GPU to enumerate.
DS4_CUDA_HAVE_MXF4 is deliberately left unset: upstream defines it only for
single-arch sm_120/sm_121 builds and guards it with a plain #ifdef rather than
__CUDA_ARCH__, so it cannot be combined with older archs. It gates an optional
MXFP4 indexer fast path whose #ifndef branch returns 0 and falls back cleanly,
so omitting it costs speed on GB10, not correctness.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* test(ds4): cover the multi-batch prefill regression
The architecture fix has no automated guard: every existing e2e spec uses a
short prompt, and the miscompiled backend answered short prompts correctly.
The corruption only appears once a prompt spans more than one prefill batch,
so the whole suite passed against a backend that produced garbage in normal
use.
Add an opt-in "long_prefill" capability to the backend e2e suite that sends a
prompt well past one batch with a known needle and asserts the answer still
reflects it, and document in the ds4 guide why the build must never omit an
nvcc architecture, how to check which flags a configuration resolves to
without compiling, and how to run the new spec.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
---------
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
Propagate gRPC cancellation into DS4 prompt synchronization and poll it at decode boundaries.
Stop on failed stream writes and skip parser finalization and KV persistence for abandoned partial requests.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
Clamp requested generation to the usable context after prompt sync while preserving the legacy 256-token fallback for omitted limits.
Constrain each speculative MTP cycle to the remaining request budget so accepted tokens cannot advance beyond the visible output limit.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): link librdma from the static ggml-rpc build
ggml-rpc gained an Apple RDMA transport in this llama.cpp range and
declares its librdma dependency with target_link_options(ggml-rpc
PRIVATE "LINKER:-weak_library,..."). Link options are not a usage
requirement of a static library, so the llama-cpp-grpc variant, which
builds with BUILD_SHARED_LIBS=OFF, dropped the flag and left every
ibv_* symbol of transport-apple.cpp undefined when grpc-server linked
on darwin.
prepare.sh now re-declares the same weak link as INTERFACE on the
ggml-rpc target, so the flag reaches whoever links the static library.
The append is guarded on a marker for repeat runs, and on
GGML_RPC_RDMA_APPLE, which the turboquant and bonsai forks lack.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
DS4 appends the opening thinking marker to tokenizer-templated prompts, so generated text begins directly with reasoning bytes. Starting DsmlParser in TEXT therefore puts the reasoning and closing marker in visible content.
Start the parser in THINK for structured chat requests with thinking enabled in both Predict and PredictStream. Keep the default TEXT state for raw prompts and reasoning-off requests, and add incremental regression coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): follow upstream MTMD APIs
The dependency update adds MTMD initialization options to prompt and
bitmap helpers. The gRPC adapter now passes the server options through
each affected path.
The update also replaces the per-layer MoE regex helper. Preparation
probes both APIs because older forks still reuse this adapter.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): adapt to the common JSON API
The llama.cpp bump replaces its nlohmann JSON alias with common_json. Update the gRPC adapter for the new exception, iterator, conversion, and container APIs.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(turboquant): adapt the JSON exception type
The shared gRPC source now follows the upstream common_json API. The
TurboQuant fork still exposes nlohmann JSON and cannot compile the new
exception type.
Translate that exception in the fork-specific source patch so both
llama.cpp variants compile from the shared adapter.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(bonsai): adapt the JSON exception type
The shared gRPC source uses upstream's common_json wrapper. The Bonsai fork still exposes nlohmann JSON and cannot compile that exception type.\n\nTranslate the exception in the fork-specific preparation step and verify that repeated preparation stays idempotent.\n\nAssisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(llama-cpp): let prepare register gRPC
The score patch duplicated the gRPC CMake registration that prepare.sh already owns. Its stale context rejects the current upstream tools file on Darwin before compilation starts.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
