mirror of
https://github.com/mudler/LocalAI.git
synced 2026-08-01 11:00:24 -04:00
c089caf320
* feat(sycl): make the intel llama.cpp backend self-contained on any host The SYCL backend shipped an incomplete oneAPI runtime AND relied on a host-provided GPU driver, so it only ran inside the build container. On a bare host it died with "libze_loader.so.1 / libdnnl.so.3: cannot open shared object file", and even with the host's Intel driver installed it SIGSEGV'd during SYCL init when the host driver was built against a newer glibc than the backend's bundled loader (rolling-release distros). package_intel_libs now bundles the complete, coherent oneAPI runtime (the missing MKL ILP64 / sycl_blas / tbb_thread + oneDNN + the dlopen'd UR adapters, plus a sweep of the backend binaries' own direct deps) and the Intel GPU userspace driver (libze_intel_gpu + libigdrcl + IGC + gmm) with its OpenCL ICD manifest, mirroring how package_vulkan_libs bundles Mesa. run.sh points the Level Zero and OpenCL loaders at the bundled driver, and install-base-deps.sh installs it in the SYCL build image. Bundling the driver is safe across kernels because it talks to the host i915/xe via the stable DRM UAPI (unlike NVIDIA's kernel-locked userspace). Validated on Arch (glibc 2.43, i915): the backend loads and runs on an Iris Xe with no host Intel packages installed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(sycl): install a driver that exists, and let the user choose their own The driver install added earlier in this branch asked apt for intel-level-zero-gpu, which is not a package in Ubuntu 24.04. apt fails outright on an unknown name, so neither driver was installed, nothing was there to copy, and the images carried no driver at all. It now comes from Intel's own repository, which has 25.18 for this Ubuntu release, against 23.43 from late 2023 in the Ubuntu archive. The archive driver does not know any card released since, so a machine with a recent Intel GPU would end up carrying a driver that cannot drive it. Anything that goes wrong during that install fails the build on purpose: an unreachable repository is a passing problem that a retry fixes, while quietly carrying a different driver, or none, is a difference nobody would notice until a user reports an idle GPU. run.sh used to overwrite whatever driver the user had chosen. Level Zero uses only the driver it is given, so on a machine with a card too new for the carried driver, the GPU would go unused with no way back. Both that setting and the OpenCL one are now left alone when already set, and the docs say how to point a backend at the machine's own driver. The OpenCL setting also used to be applied whenever the backend held a driver list, even when the driver it named had not been copied, which leaves OpenCL with nothing instead of falling back to the machine's own driver. It now requires the copied driver to be present, and the packaging leaves out the list entry of any driver it did not copy. The oneAPI images list a processor-only OpenCL library, which was being carried with nothing behind it. Two more corrections in the packaging. The scan for libraries a program is linked against only looked at files named llama-cpp-*, so turboquant and bonsai, which are also built for Intel GPUs, were left with the incomplete set of libraries this branch set out to fix; it now looks at every program in the directory. And a build that should carry a driver but ends up without one now says so, which is what a stale prebuilt base image looks like: such a backend still runs on a machine that has its own driver, so nothing fails and the only other symptom is a user reporting an idle GPU. Backends now also ask the driver to report how much graphics memory is free, without which llama.cpp reads zero on an integrated GPU, since such a chip shares the system memory instead of having its own. turboquant and bonsai get the same run.sh handling as llama.cpp. The driver is only carried by the builds that start through run.sh, because run.sh is what points Level Zero and OpenCL at it. The Python backends for Intel GPUs start differently and would never load it, so they keep using the machine's own driver rather than carrying several hundred megabytes they cannot use. Checked in a container on Ubuntu 24.04: the install brings driver 25.18 with the files where the packaging expects them, an unreachable repository fails the build, and the copied set resolves on its own once the machine's Intel packages are moved away. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(ci): rebuild every Linux backend when the GPU packaging script changes scripts/build/package-gpu-libs.sh decides which GPU libraries end up inside an image. The filter that builds the backend matrix listed it as an input of the Python images only, so changing it rebuilt no Go and no C++ backend, even though those run it from their own package.sh. A packaging fix aimed at the Intel llama.cpp backend could merge and reach no image, which is the same failure this rule was written to prevent. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(sycl): carry only the driver Level Zero uses, not the OpenCL one llama.cpp reaches an Intel GPU through Level Zero, which hands the driver programs that are already compiled and so needs only the back end of the graphics compiler. The OpenCL driver can be handed source code instead, so it needs the compiler's front end as well, and that arrives with its own copy of clang. Carrying it cost about 139 MB in every backend built for Intel GPUs, and took the carried set from 123 MB to 261 MB. Nothing here takes that path. No LocalAI code selects an OpenCL device, each backend image holds one backend, and the documentation never described OpenCL as a way to run models: the only mentions are a stale clblas row in the BUILD_TYPE table, for a llama.cpp backend that no longer exists and that no build matrix entry uses, and the sycl-ls troubleshooting hint. Before this branch the packaging carried the OpenCL loader and adapter but no driver, so the path could not work in a released image either. There is nobody to keep working. The driver list that OpenCL reads is no longer carried, and run.sh no longer sets OCL_ICD_VENDORS, so OpenCL inside a container keeps using whatever the image provides rather than being pointed at a directory with no driver in it. Checked in a container against the real 25.18 driver: the carried set is 123 MB with nothing unresolved, and Level Zero still reports the GPU with the machine's own Intel packages moved out of the way. Neither the Level Zero driver nor the compiler back end names the front end or clang among the libraries it opens by name, so the leaner set is complete for this path. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> --------- Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
78 lines
2.9 KiB
Bash
Executable File
78 lines
2.9 KiB
Bash
Executable File
#!/bin/bash
|
|
set -ex
|
|
|
|
# Get the absolute current dir where the script is located
|
|
CURDIR=$(dirname "$(realpath "$0")")
|
|
|
|
cd /
|
|
|
|
echo "CPU info:"
|
|
grep -e "model\sname" /proc/cpuinfo | head -1
|
|
grep -e "flags" /proc/cpuinfo | head -1
|
|
|
|
BINARY=bonsai-fallback
|
|
|
|
# x86/arm64 ship a single bonsai-cpu-all built with ggml CPU_ALL_VARIANTS: ggml's
|
|
# backend registry dlopens the best libggml-cpu-*.so for this host, so no shell-side
|
|
# probing. ROCm ships only bonsai-fallback, so fall back to it when cpu-all is absent.
|
|
if [ -e "$CURDIR"/bonsai-cpu-all ]; then
|
|
BINARY=bonsai-cpu-all
|
|
fi
|
|
|
|
if [ -n "$LLAMACPP_GRPC_SERVERS" ]; then
|
|
if [ -e "$CURDIR"/bonsai-grpc ]; then
|
|
BINARY=bonsai-grpc
|
|
fi
|
|
fi
|
|
|
|
# Extend ld library path with the dir where this script is located/lib
|
|
if [ "$(uname)" == "Darwin" ]; then
|
|
export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
|
|
else
|
|
export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
|
|
# Tell rocBLAS where to find TensileLibrary data (GPU kernel tuning files)
|
|
if [ -d "$CURDIR/lib/rocblas/library" ]; then
|
|
export ROCBLAS_TENSILE_LIBPATH="$CURDIR"/lib/rocblas/library
|
|
fi
|
|
# Same for hipBLASLt (rocblaslt): the bundled libhipblaslt.so resolves its
|
|
# TensileLibrary_lazy_gfx*.dat kernel data relative to itself, so point it at
|
|
# the bundled data or it falls back to slow generic kernels (issue #10660).
|
|
if [ -d "$CURDIR/lib/hipblaslt/library" ]; then
|
|
export HIPBLASLT_TENSILE_LIBPATH="$CURDIR"/lib/hipblaslt/library
|
|
fi
|
|
# Backends built for Intel GPUs carry a copy of the Intel graphics driver,
|
|
# and libze_loader is only there in those builds. Level Zero looks for a
|
|
# driver on its own, so point it at the copy that came with this backend: it
|
|
# was built against the same C library, while the machine's own driver may
|
|
# not have been, and loading that one can crash on start.
|
|
#
|
|
# Anything the user set is left alone, so a machine with a graphics card
|
|
# newer than the driver carried here can still be told to use its own.
|
|
# Nothing is said about OpenCL: no OpenCL driver is carried, so anything we
|
|
# set there would leave OpenCL worse off than the machine's own setup.
|
|
if [ -e "$CURDIR/lib/libze_loader.so.1" ]; then
|
|
if [ -e "$CURDIR/lib/libze_intel_gpu.so.1" ] && [ -z "${ZE_ENABLE_ALT_DRIVERS:-}" ]; then
|
|
export ZE_ENABLE_ALT_DRIVERS="$CURDIR"/lib/libze_intel_gpu.so.1
|
|
fi
|
|
# Ask the driver how much graphics memory is free. Without this, the
|
|
# backend reads zero on an integrated graphics chip, because such a chip
|
|
# shares the system memory instead of having its own.
|
|
if [ -z "${ZES_ENABLE_SYSMAN:-}" ]; then
|
|
export ZES_ENABLE_SYSMAN=1
|
|
fi
|
|
fi
|
|
fi
|
|
|
|
# If there is a lib/ld.so, use it
|
|
if [ -f "$CURDIR"/lib/ld.so ]; then
|
|
echo "Using lib/ld.so"
|
|
echo "Using binary: $BINARY"
|
|
exec "$CURDIR"/lib/ld.so "$CURDIR"/$BINARY "$@"
|
|
fi
|
|
|
|
echo "Using binary: $BINARY"
|
|
exec "$CURDIR"/$BINARY "$@"
|
|
|
|
# We should never reach this point, however just in case we do, run fallback
|
|
exec "$CURDIR"/bonsai-fallback "$@"
|