mirror/LocalAI
1
0 Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-09-25 07:34:58 -04:00
Code Issues Packages Projects Releases Wiki Activity

chore: ⬆️ Update TheTom/llama-cpp-turboquant to 4deec5587b2963af00bdf80884f3337e02eb7d64 (#12154)

Browse Source 
* ⬆️ Update TheTom/llama-cpp-turboquant

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix(turboquant): patch D512 flash-attn shared memory overflow

turboquant 4deec55 added DECL_FATTN_VEC_CASE_D512 for TURBO2_0 and
TURBO3_0 V cache types. The D=512 kernel template with these types
allocates 65 KB of shared memory, exceeding the 48 KB GPU limit:

  ptxas error: Entry function uses too much shared data
  (0x10100 bytes, 0xc000 max)

Carry the fix as a patch under backend/cpp/turboquant/patches/ until
TheTom/llama-cpp-turboquant#386 is merged upstream.

TURBO4_0 (4-bit) does not overflow and is left unchanged.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
localai-org-maint-botmudlerEttore Di Giacinto
 authored and GitHub committed 2026-09-23 21:12:38 +00:00
1 parent 9ad18c8c67
commit 2ccda5ba92
2 files changed
+19 -1
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

backend/cpp/turboquant/Makefile
+1 -1
View File
@@ -1,7 +1,7 @@
# Pinned to the HEAD of feature/turboquant-kv-cache on https://github.com/TheTom/llama-cpp-turboquant.
# Auto-bumped nightly by .github/workflows/bump_deps.yaml.
TURBOQUANT_VERSION?=407f3237bfb3eeaff61546797de3d8c1a96be748
TURBOQUANT_VERSION?=4deec5587b2963af00bdf80884f3337e02eb7d64
LLAMA_REPO?=https://github.com/TheTom/llama-cpp-turboquant
CMAKE_ARGS?=
backend/cpp/turboquant/patches/0002-remove-d512-turbo23-shared-mem-overflow.patch
+18
View File
@@ -0,0 +1,18 @@
diff --git a/ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo2_0.cu b/ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo2_0.cu
index a93be56..3630d87 100644
--- a/ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo2_0.cu
+++ b/ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo2_0.cu
@@ -5,4 +5,3 @@
DECL_FATTN_VEC_CASE( 64, GGML_TYPE_Q8_0, GGML_TYPE_TURBO2_0);
DECL_FATTN_VEC_CASE(128, GGML_TYPE_Q8_0, GGML_TYPE_TURBO2_0);
DECL_FATTN_VEC_CASE(256, GGML_TYPE_Q8_0, GGML_TYPE_TURBO2_0);
-DECL_FATTN_VEC_CASE_D512(GGML_TYPE_Q8_0, GGML_TYPE_TURBO2_0);
diff --git a/ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo3_0.cu b/ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo3_0.cu
index 3c806c2..c8a4d9f 100644
--- a/ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo3_0.cu
+++ b/ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo3_0.cu
@@ -5,4 +5,3 @@
DECL_FATTN_VEC_CASE( 64, GGML_TYPE_Q8_0, GGML_TYPE_TURBO3_0);
DECL_FATTN_VEC_CASE(128, GGML_TYPE_Q8_0, GGML_TYPE_TURBO3_0);
DECL_FATTN_VEC_CASE(256, GGML_TYPE_Q8_0, GGML_TYPE_TURBO3_0);
-DECL_FATTN_VEC_CASE_D512(GGML_TYPE_Q8_0, GGML_TYPE_TURBO3_0);