The previous patch only removed DECL_FATTN_VEC_CASE_D512 for turbo2_0
and turbo3_0 V cache types. turbo4_0 also overflows shared memory
(0x10100 bytes > 0xc000 max), causing ptxas errors on CUDA 12/13.
Additionally, the previous patch was incomplete: it only removed the
template instantiations but not the dispatch calls in fattn.cu or the
extern declarations in fattn-vec.cuh. This caused linker errors
(undefined reference to ggml_cuda_flash_attn_ext_vec_case_d512).
This patch removes all three layers for all turbo V types:
- Template instance .cu files (DECL_FATTN_VEC_CASE_D512)
- Dispatch calls in fattn.cu (FATTN_VEC_CASE_D512)
- Extern declarations in fattn-vec.cuh (extern DECL_FATTN_VEC_CASE_D512)
Signed-off-by: mudler <mudler@localai.io>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update TheTom/llama-cpp-turboquant
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(turboquant): patch D512 flash-attn shared memory overflow
turboquant 4deec55 added DECL_FATTN_VEC_CASE_D512 for TURBO2_0 and
TURBO3_0 V cache types. The D=512 kernel template with these types
allocates 65 KB of shared memory, exceeding the 48 KB GPU limit:
ptxas error: Entry function uses too much shared data
(0x10100 bytes, 0xc000 max)
Carry the fix as a patch under backend/cpp/turboquant/patches/ until
TheTom/llama-cpp-turboquant#386 is merged upstream.
TURBO4_0 (4-bit) does not overflow and is left unchanged.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Models whose options carry no explicit backend: open their session on the
CPU backend even in accelerator images. The gallery entries carry
backend:best since #11892; this covers hand-written model configurations
the same way, per deployment: the environment variable supplies the
fallback, an explicit backend: option always wins (merged beside the
existing threads and maingpu fallbacks), and validation reuses the
option parser.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
* ⬆️ Update TheTom/llama-cpp-turboquant
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(turboquant): follow renamed RPC server target
The pinned TurboQuant fork now exposes the RPC server as
ggml-rpc-server. Build that target and copy its renamed artifact while
preserving LocalAI's packaged turboquant-rpc-server name.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): disable PCH for SYCL builds
IntelLLVM cannot bundle the precompiled headers introduced by the latest
llama.cpp update. Disable PCH for both SYCL variants while retaining it
for the other backends.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link engram support in GPU builds
The bumped ds4 engine moved DeepSeek v4.1 engram helpers into a new object. Build and link that object for CUDA and Metal while retaining the upstream CPU object set.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
grpc::ServerWriter::Write() returns false once the peer is gone, and
PredictStream ignored that result at every call site. The handler kept
pulling decoded tokens and writing them into a dead stream, so the
llama.cpp slot stayed busy until the generation ended on its own terms.
A model configured with max_tokens 0 and a large context ends on its own
terms only at the context limit. On a 35B model at ~41 t/s a 120k context
is about fifty minutes, and a slot held that long is a slot every other
request for that model queues behind. Two abandoned requests were enough
to make a node with free VRAM and a healthy control plane serve nothing:
new requests timed out waiting for a slot, each timeout abandoned another
generation, and the node fell further behind the longer it ran.
Track the peer instead. The first failed write retires it for good, since
a stream never recovers, and the RPC's own cancellation flag folds into
the same predicate so the loop has one condition to test. Returning early
is what frees the slot: ~server_response_reader() posts
SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
TTSStream already checked Write(); this brings PredictStream in line.
Cancellation stays cooperative and is checked between decoded results, so
a batch already in flight may finish before the request stops.
Assisted-by: Claude:claude-opus-5
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The generated gRPC source tree omitted the new header and test. Every llama.cpp-derived backend therefore failed when grpc-server.cpp included the missing header.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
llama.cpp reports the same tensor-count error for unsupported model layouts and damaged GGUF files. Add a focused hint so operators can update the backend or verify the model without losing the upstream diagnostic.
Assisted-by: Codex:gpt-5.6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
audio.cpp forwards GPU_TARGETS to CMake as a semicolon-delimited list. The comma-delimited LocalAI value was treated as one invalid HIP architecture during configuration.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The pinned audio.cpp revision supports HIP, but LocalAI neither builds a ROCm image nor accepts its backend option. AMD hosts therefore fall back to the CPU image.
Build and publish the HIP variant, connect it to AMD capability selection, and accept both upstream HIP names.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
