mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 18:09:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
58112d4b5dfbaaa4c5ef441dd6c61a4c36488d16
LocalAI/backend/Dockerfile.python
mudler's LocalAI [bot] 963c637130 fix(gpu-libs): bundle cuDNN only where it is used, and complete it when it is (#10946) 
cuDNN 9 is a dispatcher (libcudnn.so.9) plus seven sublibraries the dispatcher
dlopen()s by bare soname. Only the dispatcher is ever a DT_NEEDED, so ldd finds
it and never the seven. The allowlist force-copied three of them
(libcudnn.so*, libcudnn_ops.so*, libcudnn_cnn.so*) into every CUDA backend,
which is wrong in both directions at once: too few libraries for a backend that
uses cuDNN, and too many for one that does not.

On an L4T fleet, ten of the eleven backends carrying cuDNN were in a broken end
state; the one that was correct was correct by accident, being BUILD_TYPE=cpu
so package_cuda_libs never ran for it.

  longcat-video bundled 4 of 8 at 9.24.0 over a complete pip set at 9.20.0.48
  in its venv. libbackend.sh puts lib/ on LD_LIBRARY_PATH, searched before
  DT_RUNPATH, so the bundle won and the rest still came from the venv:
  CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH.

  Nine others bundled 3 of 8 and had no venv cuDNN. None bundled
  libcudnn_graph, which libcudnn_cnn has a hard DT_NEEDED on, so it resolved
  out of the runtime image and the process ran bundled 9.22.0 against system
  9.23.2.

Five of those nine - llama-cpp, whisper, rfdetr-cpp, sam3-cpp,
stablediffusion-ggml - do not reference cuDNN at all. ggml goes through cuBLAS.
They were carrying ~57 MB of cuDNN with no consumer, and completing the family
for them would have taken that to ~576 MB for nothing.

Sizes overall: backends with no cuDNN consumer shed ~57 MB each (seven
instances on the fleet measured, plus longcat's ~60 MB), while the ones that
genuinely use cuDNN grow from ~57 MB to ~576 MB, because the five missing
sublibraries are ~517 MB, dominated by libcudnn_engines_precompiled. Net on
that fleet is an increase of roughly 570 MB. That growth is the bug being paid
off, not a regression: those backends only work today by silently borrowing the
missing five from the runtime image. Whether the engines set can be trimmed is
an open question, not addressed here.

So bundle per backend, by what that backend actually needs:

  - venv has a complete pip cuDNN -> bundle nothing; $ORIGIN resolves the pip
    set, which is the one its torch was built against            (longcat-video)
  - venv has no pip cuDNN         -> bundle the complete family. Stays
    conservative rather than detecting consumers: for a Python backend they sit
    inside the venv (torch, ctranslate2, onnxruntime) where the sweep does not
    look                                                                 (vllm)
  - no venv, nothing references cuDNN -> bundle nothing    (llama-cpp, whisper,
                                     rfdetr-cpp, sam3-cpp, stablediffusion-ggml)
  - no venv, something references it   -> bundle the complete family
                                                  (face-detect, voice-detect)

The no-venv case needs no new machinery. Go backends stage their own shared
object into package/lib, which IS the target dir, so sweep_transitive_deps
already pulls the dispatcher when it is a genuine dependency - that is exactly
how libcudnn_graph reached longcat. cuDNN simply comes off the force-copy list,
and complete_cudnn_family fills in the seven dlopen'd sublibraries around
whatever the sweep found. Detection is a string scan rather than ldd, so a
consumer that only dlopen()s cuDNN is seen too; over-matching costs an unused
library, under-matching costs a backend that cannot load.

Keeping bundled and pip versions in agreement instead is not viable: nothing
here pins nvidia-cudnn (zero occurrences), torch is unpinned for l4t13 except
longcat-video, and the fleet already runs five concurrent cuDNN versions -
9.19.0.56, 9.20.0.48, 9.22.0, 9.23.2, 9.24.0.

verify_cudnn_bundle asserts the end state: exactly one complete cuDNN visible to
whoever needs one - never both, never partial, and never zero for a backend that
references it. Zero is correct and common otherwise. It deliberately does not
accept the build image's system cuDNN as completing a partial bundle, which is
the shape that had been shipping silently; the build image is not the runtime
image. A version check alone would have missed longcat too, whose four bundled
libs were all 9.24.0 and mutually consistent.

Match per family for the other components for the same dlopen reason: TensorRT
(libnvinfer_plugin, libnvinfer_builder_resource), cuBLAS, cuFFT, cuSPARSE,
cuSOLVER, nvRTC. Exclusions bind inside copy_lib so they cover the sweep.

The packaging scripts' shell tests ran nowhere in CI. Add make
test-build-scripts and a lint workflow job so they gate every PR.

Fixes #10905


Assisted-by: Claude:claude-opus-4-8 golangci-lint shellcheck

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-07-19 07:48:51 +00:00

244 lines
10 KiB
Docker
Raw Blame History

ARG BASE_IMAGE=ubuntu:24.04
ARG APT_MIRROR=""
ARG APT_PORTS_MIRROR=""
FROM ${BASE_IMAGE} AS builder
ARG BACKEND=rerankers
ARG BUILD_TYPE
ENV BUILD_TYPE=${BUILD_TYPE}
ARG CUDA_MAJOR_VERSION
ARG CUDA_MINOR_VERSION
ARG SKIP_DRIVERS=false
ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
ENV DEBIAN_FRONTEND=noninteractive
ARG TARGETARCH
ARG TARGETVARIANT
ARG UBUNTU_VERSION=2404
ARG APT_MIRROR
ARG APT_PORTS_MIRROR
RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
apt-get update && \
apt-get install -y --no-install-recommends \
build-essential \
ccache \
ca-certificates \
espeak-ng \
curl \
libssl-dev \
git wget \
git-lfs \
unzip clang \
upx-ucl \
curl python3-pip \
python-is-python3 \
python3-dev llvm \
libnuma1 libgomp1 \
python3-venv make cmake && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
RUN <<EOT bash
if [ "${UBUNTU_VERSION}" = "2404" ]; then
pip install --break-system-packages --user --upgrade pip
else
pip install --upgrade pip
fi
EOT
# Cuda
ENV PATH=/usr/local/cuda/bin:${PATH}
# HipBLAS requirements
ENV PATH=/opt/rocm/bin:${PATH}
# Vulkan requirements
RUN <<EOT bash
if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
apt-get update && \
apt-get install -y --no-install-recommends \
software-properties-common pciutils wget gpg-agent && \
apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils && \
apt-get install -y mesa-vulkan-drivers libdrm2
# Mesa Vulkan ICD drivers (ANV/RADV/lavapipe) + their manifests. The
# LunarG SDK below only provides the loader and shader tooling, not
# hardware drivers — without Mesa, package-gpu-libs.sh has no ICD to
# bundle and the packaged backend finds no GPU at runtime.
if [ "amd64" = "$TARGETARCH" ]; then
wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
mkdir -p /opt/vulkan-sdk && \
mv 1.4.335.0 /opt/vulkan-sdk/ && \
cd /opt/vulkan-sdk/1.4.335.0 && \
./vulkansdk --no-deps --maxjobs \
vulkan-loader \
vulkan-validationlayers \
vulkan-extensionlayer \
vulkan-tools \
shaderc && \
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
rm -rf /opt/vulkan-sdk
fi
if [ "arm64" = "$TARGETARCH" ]; then
mkdir vulkan && cd vulkan && \
curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
tar -xvf vulkan-sdk.tar.xz && \
rm vulkan-sdk.tar.xz && \
cd 1.4.335.0 && \
cp -rfv aarch64/bin/* /usr/bin/ && \
cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
cp -rfv aarch64/include/* /usr/include/ && \
cp -rfv aarch64/share/* /usr/share/ && \
cd ../.. && \
rm -rf vulkan
fi
ldconfig && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
fi
EOT
# CuBLAS requirements
RUN <<EOT bash
if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
apt-get update && \
apt-get install -y --no-install-recommends \
software-properties-common pciutils
if [ "amd64" = "$TARGETARCH" ]; then
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
fi
if [ "arm64" = "$TARGETARCH" ]; then
if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
else
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
fi
fi
dpkg -i cuda-keyring_1.1-1_all.deb && \
rm -f cuda-keyring_1.1-1_all.deb && \
apt-get update && \
apt-get install -y --no-install-recommends \
cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
cuda-nvrtc-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
apt-get install -y --no-install-recommends \
libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
fi
apt-get clean && \
rm -rf /var/lib/apt/lists/*
fi
EOT
# https://github.com/NVIDIA/Isaac-GR00T/issues/343
RUN <<EOT bash
if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
apt-get update && apt-get install -y nvpl
fi
EOT
# If we are building with clblas support, we need the libraries for the builds
RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
apt-get update && \
apt-get install -y --no-install-recommends \
libclblast-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* \
; fi
RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
apt-get update && \
apt-get install -y --no-install-recommends \
hipblas-dev \
hipblaslt-dev \
rocblas-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
# I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
# to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
ldconfig \
; fi
RUN if [ "${BUILD_TYPE}" = "hipblas" ]; then \
ln -s /opt/rocm-**/lib/llvm/lib/libomp.so /usr/lib/libomp.so \
; fi
# Install uv as a system package
RUN curl -LsSf https://astral.sh/uv/install.sh | UV_INSTALL_DIR=/usr/bin sh
ENV PATH="/root/.cargo/bin:${PATH}"
# Increase timeout for uv installs behind slow networks
ENV UV_HTTP_TIMEOUT=180
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
# Install grpcio-tools (the version in 22.04 is too old)
RUN <<EOT bash
if [ "${UBUNTU_VERSION}" = "2404" ]; then
pip install --break-system-packages --user grpcio-tools==1.71.0 grpcio==1.71.0
else
pip install grpcio-tools==1.71.0 grpcio==1.71.0
fi
EOT
COPY backend/python/${BACKEND} /${BACKEND}
COPY backend/backend.proto /${BACKEND}/backend.proto
COPY backend/python/common/ /${BACKEND}/common
COPY scripts/build/package-gpu-libs.sh /package-gpu-libs.sh
# Optional per-backend source build toggle (e.g. vllm on CPU can set
# FROM_SOURCE=true to compile against the build host SIMD instead of
# pulling a prebuilt wheel). Default empty — most backends ignore it.
ARG FROM_SOURCE=""
ENV FROM_SOURCE=${FROM_SOURCE}
# Cache-buster for the per-backend `make` step. Most Python backends list
# unpinned deps (torch, transformers, vllm, ...), so a warm registry cache
# would otherwise freeze upstream versions indefinitely. CI passes a value
# that rolls weekly so the install layer is rebuilt at most once per week
# and picks up newer wheels from PyPI / nightly indexes.
ARG DEPS_REFRESH=initial
RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
# Package GPU libraries into the backend's lib directory.
#
# Must stay after the venv is built above: package-gpu-libs.sh inspects
# /${BACKEND}/venv to decide whether this backend already carries a complete
# cuDNN from pip, and bundles one only when it does not (issue #10905).
RUN mkdir -p /${BACKEND}/lib && \
TARGET_LIB_DIR="/${BACKEND}/lib" BUILD_TYPE="${BUILD_TYPE}" CUDA_MAJOR_VERSION="${CUDA_MAJOR_VERSION}" \
bash /package-gpu-libs.sh "/${BACKEND}/lib"
# Run backend-specific packaging if a package.sh exists
RUN if [ -f "/${BACKEND}/package.sh" ]; then \
cd /${BACKEND} && bash package.sh; \
fi
FROM scratch
ARG BACKEND=rerankers
COPY --from=builder /${BACKEND}/ /
Reference in New Issue View Git Blame Copy Permalink