mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-08-04 12:22:22 -04:00
Code Issues Packages Projects Releases Wiki Activity

fix(llama-cpp): retain CPU variants in GPU builds (#11255)

Browse Source 
Build the runtime CPU variant set alongside x86 GPU backends so partial offload uses the host's SIMD kernels instead of the scalar fallback. Keep arm64 GPU images on the portable binary until their builders consistently provide gcc-14.

Assisted-by: Codex:gpt-5

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
This commit is contained in:
localai-org-maint-bot
2026-08-01 09:26:23 +02:00
committed by GitHub
parent 7e4a60c701
commit cedcbf97a9
4 changed files with 51 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
backend/cpp/llama-cpp/run.sh
View File

@@ -12,10 +12,10 @@ grep -e "flags" /proc/cpuinfo | head -1
BINARY=llama-cpp-fallback
# CPU images (x86, arm64, darwin) ship a single llama-cpp-cpu-all built with ggml
# CPU images and x86 GPU images ship a single llama-cpp-cpu-all built with ggml
# CPU_ALL_VARIANTS: ggml's backend registry dlopens the best libggml-cpu-*.so for this
# host, so no shell-side AVX probing. GPU images (cublas/sycl/vulkan/hipblas) ship only
# llama-cpp-fallback (the accelerator does the compute), so fall back to it when absent.
# host, so no shell-side AVX probing. GPU arm64 images still ship llama-cpp-fallback
# until their builder toolchains support ggml's complete arm variant matrix.
if [ -e "$CURDIR"/llama-cpp-cpu-all ]; then
BINARY=llama-cpp-cpu-all
fi
@@ -76,4 +76,4 @@ echo "Using binary: $BINARY"
exec "$CURDIR"/$BINARY "$@"
# We should never reach this point, however just in case we do, run fallback
exec "$CURDIR"/llama-cpp-fallback "$@"
exec "$CURDIR"/llama-cpp-fallback "$@"