mirror of
https://github.com/mudler/LocalAI.git
synced 2026-06-26 09:26:55 -04:00
4ac67d255d
* feat(llama-cpp): single x86 CPU build via ggml CPU_ALL_VARIANTS
Replace the per-microarch avx/avx2/avx512/fallback multi-binary build on
x86 with a single grpc-server plus the dlopen-able libggml-cpu-*.so set
that ggml's backend registry selects at runtime by probing host CPU
features. One build instead of four, broader microarch coverage (adds
alderlake AVX-VNNI, zen4 AVX512-BF16, sapphirerapids AMX), and the
shell-side /proc/cpuinfo probing in run.sh goes away.
Build/link notes:
- CPU_ALL_VARIANTS requires GGML_BACKEND_DL + BUILD_SHARED_LIBS=ON, so
ggml/llama become shared objects. SHARED_LIBS is now a make variable
(default OFF) so the override survives the recursive sub-make into the
VARIANT build dir instead of being re-clobbered by the base flags.
- The cpu-all target also builds "--target ggml": the per-microarch
backends are runtime-dlopened, not link deps, so they only compile via
ggml's add_dependencies().
- hw_grpc_proto is pinned STATIC. Under BUILD_SHARED_LIBS=ON it would
otherwise become a DSO referencing hidden-visibility symbols in the
static libprotobuf.a, which fails to link ("hidden symbol ... is
referenced by DSO"). Keeping it static links gRPC/protobuf into the
executable while only ggml/llama stay shared, so no PIC or base-image
change is required.
- package.sh bundles the libggml-*.so set into package/lib; ggml finds
them by scanning the bundled ld.so directory (/proc/self/exe), which
run.sh launches from.
Scope: x86 only. arm64/darwin keep the single fallback build. The
ik-llama-cpp / turboquant forks and the other ggml C++ backends are
unchanged; the same recipe applies but is out of scope here.
Validated with a full docker build plus a live inference smoke test:
the model loads, ggml selects the AVX512_BF16 variant on a Zen-class
host, and tokens generate correctly.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(llama-cpp,turboquant): extend CPU_ALL_VARIANTS to arm64 + turboquant
- llama-cpp: x86 AND arm64 now use the single llama-cpp-cpu-all build
(only hipblas keeps the fallback build). ggml's arm64 variant table
(armv8.x / armv9.x, plus apple_m* on darwin) is selected at runtime.
- turboquant: same recipe via a turboquant-cpu-all target. turboquant
copies backend/cpp/llama-cpp's CMakeLists.txt + Makefile per flavor, so
the hw_grpc_proto STATIC fix and the SHARED_LIBS / EXTRA_CMAKE_ARGS
make-vars are inherited; the target just passes SHARED_LIBS=ON, the DL
flags and --target ggml through, then collects the .so set. run.sh and
package.sh updated to ship/select turboquant-cpu-all.
- Makefile lib-collection find now also matches *.dylib (for the darwin
build, which emits dylibs rather than .so).
ik-llama-cpp is intentionally left unchanged: its pinned ggml has no
CPU_ALL_VARIANTS support and its IQK kernels require AVX2, so the
per-microarch dynamic backend set does not apply.
Scope still excludes the darwin packaging wiring (separate change).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(llama-cpp,turboquant): arm64 gcc-14 for SME variants + darwin cpu-all packaging
- arm64: ggml CPU_ALL_VARIANTS builds armv9.2 SME variants whose -march=...+sme
is rejected by the Ubuntu 24.04 default gcc-13. Build the arm64 variants with
gcc-14 (installed in the compile step). The host only selects a variant it
actually supports at runtime, but every variant must still compile.
- darwin: scripts/build/llama-cpp-darwin.sh builds llama-cpp-cpu-all instead of
the fallback binary, keeps Metal (GGML_METAL stays ON; --target ggml also builds
ggml-metal). The per-microarch libggml-cpu-*.dylib are placed in the package
root next to the binary (darwin has no bundled ld.so, so ggml's executable-dir
scan looks there), while the other shared dylibs go in lib/ for DYLD_LIBRARY_PATH.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(llama-cpp-darwin): distribute ggml backends by suffix (.so root, .dylib lib)
ggml emits its loadable backends (per-microarch CPU variants, metal, blas) with a
.so suffix even on darwin, while the core libraries (ggml-base/ggml/llama/
llama-common/mtmd) use .dylib. Split the distribution by suffix: .so DL backends
go in the package root for ggml's executable-directory scan, .dylib core libs go
in lib/ for DYLD_LIBRARY_PATH. The previous .dylib name-pattern matched none of the
variants.
Verified on an M4: ggml loads the apple_m4 CPU variant (SME=1) and Metal, model
loads and generates correct tokens.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(llama-cpp,turboquant): only CPU_ALL_VARIANTS for pure-CPU builds, GPU uses fallback
The previous gate sent every non-hipblas build through llama-cpp-cpu-all, so the
GPU image builds (cublas, sycl_f16/f32, vulkan, nvidia l4t) compiled the whole CPU
microarch variant matrix on top of their already-huge GPU backend - blowing the
build time (the sycl job was only 59% done after 2h11m) - and the arm64 l4t build
failed at `apt-get install gcc-14` (exit 100) on the Jetson base.
Gate on an empty BUILD_TYPE instead: only the pure CPU image (build-type: '' in
.github/backend-matrix.yml) builds the CPU_ALL_VARIANTS set; every GPU build gets a
single fallback CPU grpc-server, since the accelerator does the compute. This also
confines the arm64 gcc-14 step (needed for the armv9.2 SME variants) to the CPU
build, away from the GPU base images.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* docs(llama-cpp): correct run.sh comment for arm64/darwin cpu-all
arm64 and darwin CPU images now also ship llama-cpp-cpu-all (not fallback-only);
only GPU images ship fallback-only. Fix the stale comment to match.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
84 lines
2.8 KiB
Bash
84 lines
2.8 KiB
Bash
#!/bin/bash
|
|
|
|
set -ex
|
|
|
|
IMAGE_NAME="${IMAGE_NAME:-localai/llama-cpp-darwin}"
|
|
|
|
pushd backend/cpp/llama-cpp
|
|
|
|
# Single build via ggml CPU_ALL_VARIANTS: one binary plus the per-microarch Apple/arm
|
|
# dylibs (apple_m1/m2_m3/m4, armv8.x) that ggml selects at runtime. GGML_METAL stays ON
|
|
# and --target ggml also builds ggml-metal (via add_dependencies), so the Metal GPU
|
|
# backend is still produced as a loadable libggml-metal.dylib.
|
|
make llama-cpp-cpu-all && \
|
|
make llama-cpp-grpc && \
|
|
make llama-cpp-rpc-server
|
|
|
|
popd
|
|
|
|
mkdir -p build/darwin
|
|
mkdir -p backend-images
|
|
mkdir -p build/darwin/lib
|
|
|
|
cp -rf backend/cpp/llama-cpp/llama-cpp-cpu-all build/darwin/
|
|
cp -rf backend/cpp/llama-cpp/llama-cpp-grpc build/darwin/
|
|
cp -rf backend/cpp/llama-cpp/llama-cpp-rpc-server build/darwin/
|
|
|
|
# Distribute the shared ggml/llama libraries from the CPU_ALL_VARIANTS build. Unlike the
|
|
# old fully-static fallback build, these have @rpath install names, so the otool loop below
|
|
# (which only copies deps that exist on disk) will not pick them up. The split is by suffix:
|
|
# - ggml emits its loadable backends (per-microarch CPU variants, metal, blas) with a .so
|
|
# suffix EVEN ON DARWIN. These go in the package ROOT next to the binary, because darwin
|
|
# run.sh execs the binary directly (no bundled ld.so) so ggml's executable-directory
|
|
# scan looks there.
|
|
# - the core libraries (libggml-base/libggml/libllama/libllama-common/libmtmd) use the
|
|
# platform .dylib suffix and are NEEDED deps; they go in lib/, resolved at load time via
|
|
# the DYLD_LIBRARY_PATH=lib that run.sh exports. -a preserves the version symlinks.
|
|
SHLIBS=backend/cpp/llama-cpp/ggml-shared-libs
|
|
cp -a $SHLIBS/*.so build/darwin/
|
|
cp -a $SHLIBS/*.dylib build/darwin/lib/
|
|
|
|
# Set default additional libs only for Darwin on M chips (arm64)
|
|
if [[ "$(uname -s)" == "Darwin" && "$(uname -m)" == "arm64" ]]; then
|
|
ADDITIONAL_LIBS=${ADDITIONAL_LIBS:-$(ls /opt/homebrew/Cellar/protobuf/**/lib/libutf8_validity*.dylib 2>/dev/null)}
|
|
else
|
|
ADDITIONAL_LIBS=${ADDITIONAL_LIBS:-""}
|
|
fi
|
|
|
|
for file in $ADDITIONAL_LIBS; do
|
|
cp -rfv $file build/darwin/lib
|
|
done
|
|
|
|
for file in build/darwin/*; do
|
|
LIBS="$(otool -L $file | awk 'NR > 1 { system("echo " $1) } ' | xargs echo)"
|
|
for lib in $LIBS; do
|
|
# only libraries ending in dylib
|
|
if [[ "$lib" == *.dylib ]]; then
|
|
if [ -e "$lib" ]; then
|
|
cp -rvf "$lib" build/darwin/lib
|
|
fi
|
|
fi
|
|
done
|
|
done
|
|
|
|
echo "--------------------------------"
|
|
echo "ADDITIONAL_LIBS: $ADDITIONAL_LIBS"
|
|
echo "--------------------------------"
|
|
|
|
echo "Bundled libraries:"
|
|
ls -la build/darwin/lib
|
|
|
|
|
|
cp -rf backend/cpp/llama-cpp/run.sh build/darwin/
|
|
|
|
PLATFORMARCH="${PLATFORMARCH:-darwin/arm64}"
|
|
|
|
./local-ai util create-oci-image \
|
|
build/darwin/. \
|
|
--output ./backend-images/llama-cpp.tar \
|
|
--image-name $IMAGE_NAME \
|
|
--platform $PLATFORMARCH
|
|
|
|
rm -rf build/darwin
|
|
|