mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-12 22:33:54 -04:00
1070cb12450328fa684de11717985fdec78e66aa
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dimitris Karakasilisandlocalai-org-maint-bot
|
c089caf320 |
feat(sycl): make the intel llama.cpp backend self-contained on any host (#10991)
* feat(sycl): make the intel llama.cpp backend self-contained on any host The SYCL backend shipped an incomplete oneAPI runtime AND relied on a host-provided GPU driver, so it only ran inside the build container. On a bare host it died with "libze_loader.so.1 / libdnnl.so.3: cannot open shared object file", and even with the host's Intel driver installed it SIGSEGV'd during SYCL init when the host driver was built against a newer glibc than the backend's bundled loader (rolling-release distros). package_intel_libs now bundles the complete, coherent oneAPI runtime (the missing MKL ILP64 / sycl_blas / tbb_thread + oneDNN + the dlopen'd UR adapters, plus a sweep of the backend binaries' own direct deps) and the Intel GPU userspace driver (libze_intel_gpu + libigdrcl + IGC + gmm) with its OpenCL ICD manifest, mirroring how package_vulkan_libs bundles Mesa. run.sh points the Level Zero and OpenCL loaders at the bundled driver, and install-base-deps.sh installs it in the SYCL build image. Bundling the driver is safe across kernels because it talks to the host i915/xe via the stable DRM UAPI (unlike NVIDIA's kernel-locked userspace). Validated on Arch (glibc 2.43, i915): the backend loads and runs on an Iris Xe with no host Intel packages installed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(sycl): install a driver that exists, and let the user choose their own The driver install added earlier in this branch asked apt for intel-level-zero-gpu, which is not a package in Ubuntu 24.04. apt fails outright on an unknown name, so neither driver was installed, nothing was there to copy, and the images carried no driver at all. It now comes from Intel's own repository, which has 25.18 for this Ubuntu release, against 23.43 from late 2023 in the Ubuntu archive. The archive driver does not know any card released since, so a machine with a recent Intel GPU would end up carrying a driver that cannot drive it. Anything that goes wrong during that install fails the build on purpose: an unreachable repository is a passing problem that a retry fixes, while quietly carrying a different driver, or none, is a difference nobody would notice until a user reports an idle GPU. run.sh used to overwrite whatever driver the user had chosen. Level Zero uses only the driver it is given, so on a machine with a card too new for the carried driver, the GPU would go unused with no way back. Both that setting and the OpenCL one are now left alone when already set, and the docs say how to point a backend at the machine's own driver. The OpenCL setting also used to be applied whenever the backend held a driver list, even when the driver it named had not been copied, which leaves OpenCL with nothing instead of falling back to the machine's own driver. It now requires the copied driver to be present, and the packaging leaves out the list entry of any driver it did not copy. The oneAPI images list a processor-only OpenCL library, which was being carried with nothing behind it. Two more corrections in the packaging. The scan for libraries a program is linked against only looked at files named llama-cpp-*, so turboquant and bonsai, which are also built for Intel GPUs, were left with the incomplete set of libraries this branch set out to fix; it now looks at every program in the directory. And a build that should carry a driver but ends up without one now says so, which is what a stale prebuilt base image looks like: such a backend still runs on a machine that has its own driver, so nothing fails and the only other symptom is a user reporting an idle GPU. Backends now also ask the driver to report how much graphics memory is free, without which llama.cpp reads zero on an integrated GPU, since such a chip shares the system memory instead of having its own. turboquant and bonsai get the same run.sh handling as llama.cpp. The driver is only carried by the builds that start through run.sh, because run.sh is what points Level Zero and OpenCL at it. The Python backends for Intel GPUs start differently and would never load it, so they keep using the machine's own driver rather than carrying several hundred megabytes they cannot use. Checked in a container on Ubuntu 24.04: the install brings driver 25.18 with the files where the packaging expects them, an unreachable repository fails the build, and the copied set resolves on its own once the machine's Intel packages are moved away. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(ci): rebuild every Linux backend when the GPU packaging script changes scripts/build/package-gpu-libs.sh decides which GPU libraries end up inside an image. The filter that builds the backend matrix listed it as an input of the Python images only, so changing it rebuilt no Go and no C++ backend, even though those run it from their own package.sh. A packaging fix aimed at the Intel llama.cpp backend could merge and reach no image, which is the same failure this rule was written to prevent. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(sycl): carry only the driver Level Zero uses, not the OpenCL one llama.cpp reaches an Intel GPU through Level Zero, which hands the driver programs that are already compiled and so needs only the back end of the graphics compiler. The OpenCL driver can be handed source code instead, so it needs the compiler's front end as well, and that arrives with its own copy of clang. Carrying it cost about 139 MB in every backend built for Intel GPUs, and took the carried set from 123 MB to 261 MB. Nothing here takes that path. No LocalAI code selects an OpenCL device, each backend image holds one backend, and the documentation never described OpenCL as a way to run models: the only mentions are a stale clblas row in the BUILD_TYPE table, for a llama.cpp backend that no longer exists and that no build matrix entry uses, and the sycl-ls troubleshooting hint. Before this branch the packaging carried the OpenCL loader and adapter but no driver, so the path could not work in a released image either. There is nobody to keep working. The driver list that OpenCL reads is no longer carried, and run.sh no longer sets OCL_ICD_VENDORS, so OpenCL inside a container keeps using whatever the image provides rather than being pointed at a directory with no driver in it. Checked in a container against the real 25.18 driver: the carried set is 123 MB with nothing unresolved, and Level Zero still reports the GPU with the machine's own Intel packages moved out of the way. Neither the Level Zero driver nor the compiler back end names the front end or clang among the libraries it opens by name, so the leaner set is complete for this path. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> --------- Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
963c637130 |
fix(gpu-libs): bundle cuDNN only where it is used, and complete it when it is (#10946)
cuDNN 9 is a dispatcher (libcudnn.so.9) plus seven sublibraries the dispatcher
dlopen()s by bare soname. Only the dispatcher is ever a DT_NEEDED, so ldd finds
it and never the seven. The allowlist force-copied three of them
(libcudnn.so*, libcudnn_ops.so*, libcudnn_cnn.so*) into every CUDA backend,
which is wrong in both directions at once: too few libraries for a backend that
uses cuDNN, and too many for one that does not.
On an L4T fleet, ten of the eleven backends carrying cuDNN were in a broken end
state; the one that was correct was correct by accident, being BUILD_TYPE=cpu
so package_cuda_libs never ran for it.
longcat-video bundled 4 of 8 at 9.24.0 over a complete pip set at 9.20.0.48
in its venv. libbackend.sh puts lib/ on LD_LIBRARY_PATH, searched before
DT_RUNPATH, so the bundle won and the rest still came from the venv:
CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH.
Nine others bundled 3 of 8 and had no venv cuDNN. None bundled
libcudnn_graph, which libcudnn_cnn has a hard DT_NEEDED on, so it resolved
out of the runtime image and the process ran bundled 9.22.0 against system
9.23.2.
Five of those nine - llama-cpp, whisper, rfdetr-cpp, sam3-cpp,
stablediffusion-ggml - do not reference cuDNN at all. ggml goes through cuBLAS.
They were carrying ~57 MB of cuDNN with no consumer, and completing the family
for them would have taken that to ~576 MB for nothing.
Sizes overall: backends with no cuDNN consumer shed ~57 MB each (seven
instances on the fleet measured, plus longcat's ~60 MB), while the ones that
genuinely use cuDNN grow from ~57 MB to ~576 MB, because the five missing
sublibraries are ~517 MB, dominated by libcudnn_engines_precompiled. Net on
that fleet is an increase of roughly 570 MB. That growth is the bug being paid
off, not a regression: those backends only work today by silently borrowing the
missing five from the runtime image. Whether the engines set can be trimmed is
an open question, not addressed here.
So bundle per backend, by what that backend actually needs:
- venv has a complete pip cuDNN -> bundle nothing; $ORIGIN resolves the pip
set, which is the one its torch was built against (longcat-video)
- venv has no pip cuDNN -> bundle the complete family. Stays
conservative rather than detecting consumers: for a Python backend they sit
inside the venv (torch, ctranslate2, onnxruntime) where the sweep does not
look (vllm)
- no venv, nothing references cuDNN -> bundle nothing (llama-cpp, whisper,
rfdetr-cpp, sam3-cpp, stablediffusion-ggml)
- no venv, something references it -> bundle the complete family
(face-detect, voice-detect)
The no-venv case needs no new machinery. Go backends stage their own shared
object into package/lib, which IS the target dir, so sweep_transitive_deps
already pulls the dispatcher when it is a genuine dependency - that is exactly
how libcudnn_graph reached longcat. cuDNN simply comes off the force-copy list,
and complete_cudnn_family fills in the seven dlopen'd sublibraries around
whatever the sweep found. Detection is a string scan rather than ldd, so a
consumer that only dlopen()s cuDNN is seen too; over-matching costs an unused
library, under-matching costs a backend that cannot load.
Keeping bundled and pip versions in agreement instead is not viable: nothing
here pins nvidia-cudnn (zero occurrences), torch is unpinned for l4t13 except
longcat-video, and the fleet already runs five concurrent cuDNN versions -
9.19.0.56, 9.20.0.48, 9.22.0, 9.23.2, 9.24.0.
verify_cudnn_bundle asserts the end state: exactly one complete cuDNN visible to
whoever needs one - never both, never partial, and never zero for a backend that
references it. Zero is correct and common otherwise. It deliberately does not
accept the build image's system cuDNN as completing a partial bundle, which is
the shape that had been shipping silently; the build image is not the runtime
image. A version check alone would have missed longcat too, whose four bundled
libs were all 9.24.0 and mutually consistent.
Match per family for the other components for the same dlopen reason: TensorRT
(libnvinfer_plugin, libnvinfer_builder_resource), cuBLAS, cuFFT, cuSPARSE,
cuSOLVER, nvRTC. Exclusions bind inside copy_lib so they cover the sweep.
The packaging scripts' shell tests ran nowhere in CI. Add make
test-build-scripts and a lint workflow job so they gate every PR.
Fixes #10905
Assisted-by: Claude:claude-opus-4-8 golangci-lint shellcheck
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
LocalAI [bot]andEttore Di Giacinto
|
348f3c87c0 |
fix(gpu-libs): bundle hipBLASLt TensileLibrary data so ROCm backends stop falling back (#10660) (#10672) the
The ROCm packager copied rocBLAS kernel data (rocblas/library/*.dat) into the
bundled lib/ dir and run.sh pointed ROCBLAS_TENSILE_LIBPATH at it, but the
parallel hipBLASLt data dir (hipblaslt/library/TensileLibrary_lazy_gfx*.dat)
was never packaged and no HIPBLASLT_TENSILE_LIBPATH was set. The bundled
libhipblaslt.so therefore resolved its per-arch kernel data relative to itself,
found nothing, and silently fell back to slow generic kernels, logging:
rocblaslt error: Cannot read "TensileLibrary_lazy_gfx1201.dat": No such file or directory
rocblaslt error: Could not load "TensileLibrary_lazy_gfx1201.dat"
Fix, mirroring the existing rocBLAS handling:
- package-gpu-libs.sh: extract the rocblas data-dir copy into a reusable
copy_rocm_data_dir helper and call it for both rocblas and hipblaslt.
- llama-cpp/turboquant run.sh: export HIPBLASLT_TENSILE_LIBPATH when the
bundled hipblaslt/library dir exists.
The helper takes an optional ROCM_BASE_DIRS override so the copy is unit
testable without a real ROCm install; add a regression test that runs
package_rocm_libs against a fabricated ROCm tree and asserts both data dirs
are bundled.
Note: this bundles whatever gfx*.dat the build image's ROCm provides. If a
given arch's tensile data is absent from the shipped ROCm, that arch still
needs a ROCm bump; the packaging gap itself is fixed for every supported arch.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
LocalAI [bot]andEttore Di Giacinto
|
f98b0f1c1e |
fix(gpu-libs): bundle transitive deps of GPU runtime libs (#10537) (#10539)
fix(gpu-libs): bundle transitive deps of GPU runtime libs The per-vendor packagers in package-gpu-libs.sh copy an explicit allowlist of top-level GPU runtime libraries (libamdhip64, libhipblas, librocblas, the CUDA/Intel equivalents, ...) but never resolved their transitive dependencies. Backends run through the bundled lib/ld.so with LD_LIBRARY_PATH=lib, so any transitive dep not in the allowlist is a fatal "cannot open shared object file" at load time. On recent ROCm (base image rocm 7.2.1) the runtime libs link against librocprofiler-register.so.0, which is not in the allowlist, so the rocm llama-cpp backend (and every other GPU backend sharing this script) failed to load with: librocprofiler-register.so.0: cannot open shared object file The Vulkan path already solved this class of problem with copy_elf_deps (ldd-based transitive resolution), but that sweep was only wired into the Vulkan ICD path. This adds a generic sweep_transitive_deps that runs the same ldd resolution over everything the allowlist already bundled, and wires it into the ROCm, CUDA and Intel packagers. ldd returns the full recursive closure, so one pass suffices; core libc-family deps are skipped via is_core_lib so we never shadow the loader's own libc/libstdc++. Adds a self-contained regression test (gcc + ldd) that fabricates a primary lib linking a transitive lib and asserts the sweep bundles the dependency. Fixes #10537 Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
606128e4e9 |
feat(vulkan): make Vulkan backends self-contained on the GPU (#10404)
Vulkan backends bundled their own loader and ICD manifests but neither the Mesa driver the manifests point at nor a way to make the loader find them, so on a runtime base image without Mesa the loader enumerated zero devices and the GPU silently fell back to CPU (only NVIDIA worked, since its ICD is injected by the container toolkit). - scripts/build/package-gpu-libs.sh: for each installed ICD manifest, bundle the driver .so its library_path names — no hard-coded, platform-dependent soname list — plus that driver's ldd dependencies, skipping manifests whose driver isn't installed. Rewrite each library_path to a bare soname so the bundled driver resolves via the LD_LIBRARY_PATH run.sh already sets. - .docker/install-base-deps.sh, backend/Dockerfile.golang, backend/Dockerfile.python: install mesa-vulkan-drivers in every Vulkan builder so the driver + manifests exist to be packaged (the LunarG SDK ships only the loader and shader tooling). - pkg/model/process.go: when a backend ships vulkan/icd.d/, point the loader at it via VK_DRIVER_FILES/VK_ICD_FILENAMES at launch (no-op otherwise). Covered by pkg/model/process_vulkan_test.go. - backend/go/parakeet-cpp/package.sh: complete the L0 stub (was missing the libc-family ldd walk + GPU-lib packaging) by mirroring whisper, so the vulkan-parakeet image actually bundles its GPU runtime. Assisted-by: Claude Code:claude-opus-4-8 Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Andreas Egli
|
1d0de757c3 |
fix: add hipblaslt library (#9541)
Signed-off-by: Andreas Egli <github@kharan.ch> |
||
|
Ettore Di Giacinto
|
151ad271f2 |
feat(rocm): bump to 7.x (#9323)
feat(rocm): bump to 7.2.1 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
271cc79709 |
chore(backends): do not bundle cuda target directory (#7982)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Copilotandmudler
|
06323df457 |
Optimize GPU library copying to preserve symlinks and avoid duplicates (#7931)
* Initial plan * Optimize library copying to preserve symlinks and avoid duplicates Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> * Address code review feedback: extract get_inode helper, use file type detection for sorting Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> * Simplify implementation by removing inode tracking Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> * Add clarifying comment about basename deduplication Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
|
fd53978a7b |
feat: package GPU libraries inside backend containers for unified base image (#7891)
* Initial plan * Add GPU library packaging for isolated backend environments - Create scripts/build/package-gpu-libs.sh for packaging CUDA, ROCm, SYCL, and Vulkan libraries - Update llama-cpp, whisper, stablediffusion-ggml package.sh to include GPU libraries - Update Dockerfile.python to package GPU libraries into Python backends - Update libbackend.sh to set LD_LIBRARY_PATH for GPU library loading Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> * Address code review feedback: fix variable consistency and quoting Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> * Fix code review issues: improve glob handling and remove redundant variable Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> * Simplify main Dockerfile and workflow to use unified base image - Remove GPU-specific driver installation from Dockerfile (CUDA, ROCm, Vulkan, Intel) - Simplify image.yml workflow to build single unified base image for linux/amd64 and linux/arm64 - GPU libraries are now packaged in individual backend containers Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> |