Translate modern speculative fields to the pinned fork API, disable unsupported score and checkpoint features, and cover the compatibility transform with an idempotent regression test.
Assisted-by: Codex:gpt-5 [Codex]
The buun build copies the stock llama.cpp backend directory, including patches that target upstream. Remove that copied patch directory before invoking the shared build so only the explicit buun compatibility series is applied to the fork.
Assisted-by: Codex:gpt-5 [systematic-debugging]
The shared gRPC wrapper stores p_split under the draft sub-structure. Match that exact source spelling so the fork-specific patch stage reaches the build on every architecture.
Assisted-by: Codex:gpt-5 [Codex]
Two more hipblas-only build failures in buun's fattn.cu, fixed under the
same patches/ infrastructure:
1. cudaMemcpyToSymbol / cudaMemcpyFromSymbol — buun's Q² calibration +
TCQ codebook upload paths call the symbol variants of cudaMemcpy.
ggml/src/ggml-cuda/vendors/hip.h aliases every other cudaMemcpy*
name (cudaMemcpy, cudaMemcpyAsync, cudaMemcpy2DAsync, …) but the
symbol pair was never added. 15+ "use of undeclared identifier"
errors across fattn.cu lines 40, 54, 74-76, 94, 100-101, 371, 883,
905, 954, 976, 1449, 1463. Add the two missing aliases alongside
the existing memcpy block.
2. __shfl_xor_sync fwht128 calls — same 3-arg omission pattern as the
earlier argmax top-K fix. Lines 512 (ggml_cuda_fwht128 intra-warp
butterfly) and 536 (fwht128_store_half neighbor fetch) drop the
width argument that hip.h:33 requires. Add WARP_SIZE.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two call sites in ggml/src/ggml-cuda/argmax.cu (the top-K intra-warp
merge added by buun) use the 3-arg CUDA form __shfl_xor_sync(mask, var,
laneMask), omitting the optional width parameter. The hipification shim
at ggml/src/ggml-cuda/vendors/hip.h:33 is a function-like macro that
requires all four arguments, so hipcc fails with:
argmax.cu:265: too few arguments provided to function-like macro
invocation
note: macro '__shfl_xor_sync' defined here:
#define __shfl_xor_sync(mask, var, laneMask, width) \
__shfl_xor(var, laneMask, width)
Every other call in the same file already passes WARP_SIZE explicitly;
aligning these two with that convention fixes the hipblas build without
changing CUDA codegen (warpSize is the CUDA default).
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Buun's Q² calibration path in ggml/src/ggml-cuda/fattn.cu calls
atomicAdd with a double* destination. Native double atomicAdd is only
available on CUDA compute capability 6.0 and later — LocalAI's CUDA 12
Docker image builds for the full published arch range (which includes
sm_50/sm_52), so nvcc fails with:
fattn.cu:812: error: no instance of overloaded function "atomicAdd"
matches the argument list, argument types are: (double *, double)
Add the canonical CAS-loop shim from the CUDA C Programming Guide
(B.15 Atomic Functions) guarded on __CUDA_ARCH__ < 600. On sm_60+ the
guard is false and nvcc picks up the native intrinsic as before.
Patch file lives under backend/cpp/buun-llama-cpp/patches/ and is
applied to the cloned fork tree by apply-patches.sh (the infrastructure
already put in place for exactly this class of backport).
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Previous substitution kept the call as 5 args, but buun predates the
upstream refactor that also *added* the logit_bias_eog parameter to
params_from_json_cmpl — buun's signature is still the 4-arg form
(const llama_vocab*, const common_params&, int, const json&)
and it still derives logit_bias_eog internally from the common_params.
Replace the substitution with a line-delete. Guard matches both the
original call (ctx_server.get_meta().logit_bias_eog) and the previously
substituted form (params_base.sampling.logit_bias_eog) so the script
stays safe across re-runs and whatever state the tree was left in.
Assisted-by: Claude:Opus-4.7 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
LocalAI's shared grpc-server.cpp reaches
ctx_server.get_meta().logit_bias_eog twice (the twin params_from_json_cmpl
callsites). That accessor was added to server_context_meta upstream after
buun's 2026-04-05 fork-point, so compiling against buun errors with
'struct server_context_meta' has no member named 'logit_bias_eog'.
Rewrite the call sites — only in the buun grpc-server.cpp copy — to source
the vector from params_base.sampling.logit_bias_eog instead. That vector is
the underlying data the upstream meta accessor eventually returns (buun
still carries common_params_sampling::logit_bias_eog at common.h:280), so
the substitution yields identical behavior on both trees.
The sed is guarded by a grep for the call site, so this patch is
self-disabling once buun rebases past the upstream refactor.
Assisted-by: Claude:Opus-4.7 [Read] [Edit] [Bash] [WebFetch]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
spiritbuun/buun-llama-cpp is a fork of TheTom/llama-cpp-turboquant that adds
two independent features on top: DFlash block-diffusion speculative decoding
(via a dedicated DFlashDraftModel GGUF arch) and two extra TCQ KV-cache
variants (turbo2_tcq, turbo3_tcq) on top of TurboQuant's turbo2/turbo3/turbo4.
Follows the turboquant thin-wrapper pattern — reuses backend/cpp/llama-cpp
grpc-server sources verbatim, patches only the build copy to extend the KV
allow-list and wire up buun-exclusive tree_budget / draft_topk options.
DraftModel is already wired end-to-end (proto field 39 → params.speculative),
so DFlash activation only needs the existing options passthrough
(spec_type:dflash) plus the drafter path in draft_model.
CacheTypeOptions now surfaces the five turbo* values so the React UI dropdown
shows them — benefits turboquant too (previously users had to type them in
YAML manually).
Assisted-by: Claude:Opus-4.7 [Read] [Edit] [Bash] [WebFetch]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
grpc::ServerWriter::Write() returns false once the peer is gone, and
PredictStream ignored that result at every call site. The handler kept
pulling decoded tokens and writing them into a dead stream, so the
llama.cpp slot stayed busy until the generation ended on its own terms.
A model configured with max_tokens 0 and a large context ends on its own
terms only at the context limit. On a 35B model at ~41 t/s a 120k context
is about fifty minutes, and a slot held that long is a slot every other
request for that model queues behind. Two abandoned requests were enough
to make a node with free VRAM and a healthy control plane serve nothing:
new requests timed out waiting for a slot, each timeout abandoned another
generation, and the node fell further behind the longer it ran.
Track the peer instead. The first failed write retires it for good, since
a stream never recovers, and the RPC's own cancellation flag folds into
the same predicate so the loop has one condition to test. Returning early
is what frees the slot: ~server_response_reader() posts
SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
TTSStream already checked Write(); this brings PredictStream in line.
Cancellation stays cooperative and is checked between decoded results, so
a batch already in flight may finish before the request stops.
Assisted-by: Claude:claude-opus-5
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The generated gRPC source tree omitted the new header and test. Every llama.cpp-derived backend therefore failed when grpc-server.cpp included the missing header.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
llama.cpp reports the same tensor-count error for unsupported model layouts and damaged GGUF files. Add a focused hint so operators can update the backend or verify the model without losing the upstream diagnostic.
Assisted-by: Codex:gpt-5.6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
audio.cpp forwards GPU_TARGETS to CMake as a semicolon-delimited list. The comma-delimited LocalAI value was treated as one invalid HIP architecture during configuration.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The pinned audio.cpp revision supports HIP, but LocalAI neither builds a ROCm image nor accepts its backend option. AMD hosts therefore fall back to the CPU image.
Build and publish the HIP variant, connect it to AMD capability selection, and accept both upstream HIP names.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link upstream image helpers
The ds4 bump adds vision calls to the engine object. Link the new image preprocessing object into every backend target.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ds4): build CUDA kernels for the target architecture
The ds4 backend compiled its CUDA objects with no -arch. Upstream's Makefile
leaves CUDA_ARCH empty and its `cuda` target refuses to build without one,
offering `cuda-spark` (sm_121) and `cuda-generic` (native) instead. We invoke
its object targets directly, which bypasses that guard, so nvcc fell back to
its default architecture and the kernels ran as JIT'd PTX on the real GPU.
On GB10 (sm_121) that silently corrupted inference: any prompt over roughly 128
tokens produced text unrelated to the input and never closed its thinking
block, so content came back empty and the chat showed only reasoning; longer
prompts failed with "cuda decode failed". It also cost close to two orders of
magnitude of prefill throughput. Measured on one box, same model, same prompt,
same GPU, upstream ds4 at the pinned commit, differing only in the nvcc flags:
make -B ds4 (archless, as we build it) garbage output 4.21 t/s
make cuda-spark (compute_121a/sm_121a) correct output 325.70 t/s
Select an architecture list from CUDA_MAJOR_VERSION, which the backend matrix
already declares for both ds4 cublas entries but Dockerfile.ds4 never forwarded.
Upstream's CUDA_ARCH takes a single value, so it cannot express the fat binary
these images need; NVCC_ARCH_FLAGS is overridden instead, since a command-line
assignment wins over its `:=`. The lists are copied from vllm-cpp rather than
invented so the two CUDA images cover the same GPUs, with l4t/arm64 covering
Orin, Thor and GB10. An empty CUDA_MAJOR_VERSION keeps upstream's `native`
behaviour for local developer builds, and no CI runner has a GPU to enumerate.
DS4_CUDA_HAVE_MXF4 is deliberately left unset: upstream defines it only for
single-arch sm_120/sm_121 builds and guards it with a plain #ifdef rather than
__CUDA_ARCH__, so it cannot be combined with older archs. It gates an optional
MXFP4 indexer fast path whose #ifndef branch returns 0 and falls back cleanly,
so omitting it costs speed on GB10, not correctness.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* test(ds4): cover the multi-batch prefill regression
The architecture fix has no automated guard: every existing e2e spec uses a
short prompt, and the miscompiled backend answered short prompts correctly.
The corruption only appears once a prompt spans more than one prefill batch,
so the whole suite passed against a backend that produced garbage in normal
use.
Add an opt-in "long_prefill" capability to the backend e2e suite that sends a
prompt well past one batch with a known needle and asserts the answer still
reflects it, and document in the ds4 guide why the build must never omit an
nvcc architecture, how to check which flags a configuration resolves to
without compiling, and how to run the new spec.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
---------
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
Propagate gRPC cancellation into DS4 prompt synchronization and poll it at decode boundaries.
Stop on failed stream writes and skip parser finalization and KV persistence for abandoned partial requests.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
Clamp requested generation to the usable context after prompt sync while preserving the legacy 256-token fallback for omitted limits.
Constrain each speculative MTP cycle to the remaining request budget so accepted tokens cannot advance beyond the visible output limit.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): link librdma from the static ggml-rpc build
ggml-rpc gained an Apple RDMA transport in this llama.cpp range and
declares its librdma dependency with target_link_options(ggml-rpc
PRIVATE "LINKER:-weak_library,..."). Link options are not a usage
requirement of a static library, so the llama-cpp-grpc variant, which
builds with BUILD_SHARED_LIBS=OFF, dropped the flag and left every
ibv_* symbol of transport-apple.cpp undefined when grpc-server linked
on darwin.
prepare.sh now re-declares the same weak link as INTERFACE on the
ggml-rpc target, so the flag reaches whoever links the static library.
The append is guarded on a marker for repeat runs, and on
GGML_RPC_RDMA_APPLE, which the turboquant and bonsai forks lack.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
DS4 appends the opening thinking marker to tokenizer-templated prompts, so generated text begins directly with reasoning bytes. Starting DsmlParser in TEXT therefore puts the reasoning and closing marker in visible content.
Start the parser in THINK for structured chat requests with thinking enabled in both Predict and PredictStream. Keep the default TEXT state for raw prompts and reasoning-off requests, and add incremental regression coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
