backend(audio-cpp): package the backend image

Bundles the dependency closure for the from-scratch image, the dlopened ggml
CPU-variant shared objects that ldd cannot see, and upstream's bundled
silero_vad and marblenet_vad assets so VAD works with no download.

The bundled loader sits in the package ROOT rather than at lib/ld.so. run.sh
execs it, which makes /proc/self/exe name the loader, and this backend has two
consumers of that path: ggml discovers the libggml-cpu-*.so by listing
dirname(/proc/self/exe), and resolve_model_path expands bundled:<name> under the
same directory. Rooting the loader makes the binary, the ggml objects and
assets/ share the one directory all three resolution mechanisms agree on.
llama-cpp's lib/ld.so layout would need assets/ moved into lib/ as well.

The image builds against apt gRPC and protobuf, like Dockerfile.ds4 and unlike
Dockerfile.privacy-filter. The from-source gRPC that install-base-deps.sh and
the base-grpc-* images supply vendors protobuf 26, which pulls abseil into
message_lite.h; with SPM_PROTOBUF_PROVIDER=package that collides with
sentencepiece's vendored mini-abseil and every absl::internal reference becomes
ambiguous. Noble's protobuf 3.21.12 predates the abseil dependency and is the
pair every earlier verification of this backend ran against.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.dockerignore

@@ -40,6 +40,16 @@ backend/cpp/privacy-filter/build
 backend/cpp/privacy-filter/grpc-server
 backend/cpp/privacy-filter/package
 
+# audio-cpp: same in-place pattern. The Makefile clones audio.cpp at the pinned
+# AUDIO_CPP_VERSION and the `audio.cpp:` target is the directory itself, so a
+# stale host checkout COPY'd in makes the build compile against whatever commit
+# the host had. build/ is worse than stale: its CMakeCache.txt records the host
+# source, prefix and compiler paths, and cmake refuses to reconfigure from it.
+backend/cpp/audio-cpp/audio.cpp
+backend/cpp/audio-cpp/build
+backend/cpp/audio-cpp/grpc-server
+backend/cpp/audio-cpp/package
+
 # Rust backend build output (sources are tracked; target/ is generated)
 backend/rust/*/target
 

backend/Dockerfile.audio-cpp

@@ -0,0 +1,92 @@
+ARG BASE_IMAGE=ubuntu:24.04
+ARG APT_MIRROR=""
+ARG APT_PORTS_MIRROR=""
+
+# audio-cpp: 0xShug0/audio.cpp, a ggml audio inference framework covering TTS,
+# ASR, VAD, diarization, source separation and music generation, wrapped as a
+# LocalAI gRPC backend.
+#
+# BASE_IMAGE is ubuntu:24.04 for cpu and vulkan builds, or
+# nvidia/cuda:<ver>-devel-ubuntu24.04 for cublas builds; both ship apt and
+# Ubuntu Noble packages, and the CUDA base additionally provides
+# /usr/local/cuda. BUILD_TYPE selects the engine backend in the Makefile:
+# "" = portable CPU with all ggml CPU variants, "cublas" ->
+# -DENGINE_ENABLE_CUDA=ON, "vulkan" -> -DENGINE_ENABLE_VULKAN=ON. Darwin
+# (Metal) builds bypass this Dockerfile entirely.
+#
+# Upstream needs GCC 13 or newer, which ubuntu:24.04 and the CUDA 12/13
+# devel-ubuntu24.04 images all provide.
+#
+# THIS BACKEND CANNOT USE .docker/install-base-deps.sh OR THE PREBUILT
+# quay.io/go-skynet/ci-cache:base-grpc-* IMAGES, AND THAT IS NOT A STYLE CHOICE.
+#
+# Both supply gRPC v1.65 built from source at /opt/grpc, which downstream
+# Dockerfiles copy to /usr/local. That gRPC vendors protobuf v26, and protobuf
+# has depended on abseil since v22: google/protobuf/message_lite.h includes
+# absl/strings/cord.h. audio.cpp links sentencepiece, and our CMakeLists sets
+# SPM_PROTOBUF_PROVIDER=package so sentencepiece uses the same protobuf the
+# generated backend.pb.cc was built against (the alternative broke every
+# nested-message parse; the full account is in backend/cpp/audio-cpp/CMakeLists.txt).
+# That makes sentencepiece's init.h include the external message_lite.h while it
+# still includes its own vendored mini-abseil from third_party/absl. The vendored
+# copy declares `namespace absl { namespace internal { ... } }` and real abseil
+# declares `namespace absl { inline namespace lts_20240116 { namespace internal
+# { ... } } }`, so every `absl::internal::` reference becomes ambiguous and the
+# compile dies in absl/base/casts.h. Verified, not theorised: building this image
+# against the base-grpc-amd64 prebuilt fails at
+# sentencepiece-static/error.cc.o with "reference to 'internal' is ambiguous".
+#
+# Ubuntu Noble's apt protobuf is 3.21.12, which predates the abseil dependency,
+# so message_lite.h pulls in no abseil and the vendored copy is the only one in
+# scope. That is also the exact protobuf/gRPC pair every unit and end-to-end run
+# of this backend has been verified against. Keep it: a from-source gRPC here
+# does not buy a faster build, it buys a broken one.
+#
+# The install-base-deps path is additionally unsafe because it drops protoc 27.1
+# into /usr/local/bin, which shadows apt's protoc on PATH and would generate
+# protobuf-27 sources to be compiled against 3.21 headers.
+FROM ${BASE_IMAGE} AS builder
+ARG BUILD_TYPE
+ARG TARGETARCH
+ARG TARGETVARIANT
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
+
+ENV BUILD_TYPE=${BUILD_TYPE} \
+    APT_MIRROR=${APT_MIRROR} \
+    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
+    DEBIAN_FRONTEND=noninteractive \
+    PATH=/usr/local/cuda/bin:${PATH}
+
+WORKDIR /build
+
+# gRPC/protobuf from apt, deliberately; see the block above. libgrpc++-dev ships
+# a CMake config so find_package(gRPC CONFIG) resolves, and libprotobuf-dev
+# lands in the layout CMake's FindProtobuf module expects, which matters because
+# sentencepiece runs a bare find_package(Protobuf REQUIRED) with no CONFIG
+# fallback of its own.
+#
+# BUILD_TYPE=vulkan additionally needs the loader headers and glslc; both are in
+# Noble. The CUDA toolkit for BUILD_TYPE=cublas comes from BASE_IMAGE.
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
+    apt-get install -y --no-install-recommends \
+        git cmake build-essential pkg-config ca-certificates \
+        libgrpc++-dev libprotobuf-dev protobuf-compiler protobuf-compiler-grpc && \
+    if [ "${BUILD_TYPE}" = "vulkan" ]; then \
+        apt-get install -y --no-install-recommends libvulkan-dev glslc; \
+    fi && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+COPY . /LocalAI
+
+RUN --mount=type=cache,target=/root/.ccache,id=audio-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
+    make -C /LocalAI/backend/cpp/audio-cpp BUILD_TYPE=${BUILD_TYPE} NATIVE=false grpc-server package
+
+# The package directory is the whole image: run.sh, grpc-server, the dlopened
+# ggml CPU variants, the bundled loader and its library closure, and the
+# bundled silero_vad / marblenet_vad assets. Nothing else exists at run time.
+FROM scratch
+COPY --from=builder /LocalAI/backend/cpp/audio-cpp/package/. ./

backend/cpp/audio-cpp/package.sh

@@ -0,0 +1,146 @@
+#!/bin/bash
+# Assemble backend/cpp/audio-cpp/package, which becomes the whole content of the
+# FROM scratch backend image. Nothing outside this directory exists at run time.
+set -euo pipefail
+
+CURDIR=$(dirname "$(realpath "$0")")
+REPO_ROOT="${CURDIR}/../../.."
+PACKAGE_DIR="$CURDIR/package"
+BUILD_DIR="$CURDIR/build"
+
+rm -rf "$PACKAGE_DIR"
+mkdir -p "$PACKAGE_DIR/lib" "$PACKAGE_DIR/assets"
+
+cp -avf "$CURDIR/grpc-server" "$PACKAGE_DIR/"
+cp -fv  "$CURDIR/run.sh"      "$PACKAGE_DIR/"
+
+# ENGINE_ENABLE_CPU_ALL_VARIANTS builds the ggml backends as shared objects that
+# are dlopened at run time, so ldd cannot see them and the dependency walk below
+# would leave the image with no CPU backend at all. They also cannot go in lib/:
+# ggml DISCOVERS them by listing dirname(/proc/self/exe) and the current
+# directory, so being on a library path is not enough, they have to be in a
+# directory ggml scans. run.sh execs the bundled loader from the package root
+# exactly so that directory is this one. cmake writes them to build/bin, not
+# next to build/grpc-server, which is why this reads from bin/.
+#
+# -a keeps the libggml.so -> libggml.so.0 -> libggml.so.0.12.0 symlink chain,
+# so the SONAME the binary asks for still names a file here.
+for pattern in '*.so*' '*.dylib*'; do
+    if compgen -G "$BUILD_DIR/bin/$pattern" > /dev/null; then
+        # shellcheck disable=SC2086
+        cp -avf "$BUILD_DIR/bin/"$pattern "$PACKAGE_DIR/"
+    fi
+done
+
+# Upstream ships silero_vad and marblenet_vad as small runtime assets.
+# resolve_model_path() expands "bundled:<name>" to
+# dirname(/proc/self/exe)/assets/<name>, so copying them here is what makes VAD
+# work with nothing downloaded.
+for asset in silero_vad marblenet_vad; do
+    src="$CURDIR/audio.cpp/assets/framework/models/$asset"
+    if [ -d "$src" ]; then
+        cp -rfv "$src" "$PACKAGE_DIR/assets/"
+    else
+        echo "package.sh: bundled asset missing: $src" >&2
+        echo "package.sh: run 'make audio.cpp' before packaging" >&2
+        exit 1
+    fi
+done
+
+UNAME_S=$(uname -s)
+if [ "$UNAME_S" = "Darwin" ]; then
+    echo "package.sh: Darwin dylib bundling is handled by the darwin build script"
+    ls -lah "$PACKAGE_DIR/" "$PACKAGE_DIR/assets/"
+    exit 0
+fi
+
+# The loader goes in the package ROOT, not in lib/. run.sh explains why at
+# length; the short version is that exec'ing it makes dirname(/proc/self/exe)
+# the directory it sits in, and both the ggml backend scan and the bundled:
+# asset lookup need that to be the package root.
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 "$PACKAGE_DIR/ld.so"
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    cp -arfLv /lib/ld-linux-aarch64.so.1 "$PACKAGE_DIR/ld.so"
+else
+    echo "package.sh: unknown architecture" >&2
+    exit 1
+fi
+
+# Libraries the host GPU driver stack owns. package_gpu_libs deliberately ships
+# the CUDA/Vulkan runtime but not the driver, because the driver has to match
+# the kernel module on whatever host runs the image. Copying the build host's
+# copy in would pin it to the build host instead.
+is_driver_lib() {
+    case "$(basename "$1")" in
+        libcuda.so*|libnvidia-*) return 0 ;;
+        *) return 1 ;;
+    esac
+}
+
+# Bundle the full dependency closure. grpc-server links the distro gRPC,
+# protobuf and absl stack; copying only the C/C++ runtime leaves the scratch
+# image unable to start. The walk runs over the PACKAGED binary, not the one in
+# $CURDIR, because its RUNPATH is $ORIGIN: only from inside the package does
+# libggml.so.0 resolve to the copy shipped above rather than to nothing.
+# The dlopened ggml objects are walked too, since a dependency of theirs that
+# grpc-server does not itself link would otherwise be missed.
+{
+    ldd "$PACKAGE_DIR/grpc-server"
+    for so in "$PACKAGE_DIR"/*.so*; do
+        [ -f "$so" ] || continue
+        ldd "$so"
+    done
+} | awk '$2 == "=>" && $3 ~ /^\// { print $3 }' | sort -u | \
+while read -r so; do
+    # Skip what is already inside the package: the ggml objects resolve through
+    # $ORIGIN and re-copying them into lib/ would ship two copies of each.
+    case "$so" in "$PACKAGE_DIR"/*) continue ;; esac
+    if is_driver_lib "$so"; then
+        echo "package.sh: leaving driver-owned library to the host: $so"
+        continue
+    fi
+    cp -arfLv "$so" "$PACKAGE_DIR/lib/"
+done
+
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    # shellcheck source=/dev/null
+    source "$GPU_LIB_SCRIPT" "$PACKAGE_DIR/lib"
+    package_gpu_libs
+fi
+
+# Resolve every dependency through the same loader and library path the
+# from-scratch image uses. Two distinct failures are rejected, because the
+# loader can still fall back to the host's default directories: a dependency it
+# could not resolve at all, and one it resolved to a file OUTSIDE the package,
+# which would validate here and be absent in the image.
+validation_failed=0
+validate_object() {
+    local object="$1"
+    "$PACKAGE_DIR/ld.so" --library-path "$PACKAGE_DIR/lib:$PACKAGE_DIR" \
+        --list "$object" | awk -v pkg="$PACKAGE_DIR/" -v obj="$object" '
+        $2 == "=>" && $3 == "not" {
+            print "package.sh: unresolved dependency of " obj ": " $1 > "/dev/stderr"
+            bad = 1
+        }
+        $2 == "=>" && $3 ~ /^\// && index($3, pkg) != 1 {
+            print "package.sh: dependency of " obj " resolved outside the package: " $0 > "/dev/stderr"
+            bad = 1
+        }
+        END { exit bad }
+    '
+}
+
+validate_object "$PACKAGE_DIR/grpc-server" || validation_failed=1
+for so in "$PACKAGE_DIR"/*.so*; do
+    [ -f "$so" ] || continue
+    validate_object "$so" || validation_failed=1
+done
+if [ "$validation_failed" -ne 0 ]; then
+    exit 1
+fi
+
+echo "audio-cpp package contents:"
+ls -lah "$PACKAGE_DIR/" "$PACKAGE_DIR/lib/" "$PACKAGE_DIR/assets/"

backend/cpp/audio-cpp/run.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+# Entry point for the audio-cpp backend image and for BACKEND_BINARY mode.
+#
+# The image's final stage is FROM scratch, so the package root is / and there is
+# no system loader, no system libc and no fallback library path. Everything the
+# process opens has to be inside the package, and it has to be findable by the
+# two mechanisms that actually do the finding: the dynamic linker, and
+# audio.cpp's own directory scans.
+set -e
+
+CURDIR=$(dirname "$(realpath "$0")")
+
+if [ "$(uname -s)" = "Darwin" ]; then
+	export DYLD_LIBRARY_PATH="$CURDIR/lib:$CURDIR:$DYLD_LIBRARY_PATH"
+	exec "$CURDIR/grpc-server" "$@"
+fi
+
+# $CURDIR is on the path as well as $CURDIR/lib: the ggml shared objects the
+# CPU-all-variants build produces sit in the package root, next to the binary,
+# not in lib/. See the comment below for why they cannot live in lib/.
+export LD_LIBRARY_PATH="$CURDIR/lib:$CURDIR:$LD_LIBRARY_PATH"
+
+# THE BUNDLED LOADER IS AT THE PACKAGE ROOT, NOT AT lib/ld.so. DO NOT MOVE IT.
+#
+# Exec'ing the loader is what pins the bundled glibc to the matching ld.so, and
+# every other C++ backend here does it. The cost is that /proc/self/exe then
+# names the LOADER rather than grpc-server, and this backend has two consumers
+# of /proc/self/exe that both have to land on the package root:
+#
+#   - ggml's backend registry DISCOVERS the per-microarch libggml-cpu-*.so by
+#     scanning dirname(/proc/self/exe) and the current directory. Those files
+#     are dlopened, never linked, so no library path and no RUNPATH reaches
+#     them: they have to be in a directory ggml scans.
+#   - resolve_model_path() turns "bundled:<name>" into
+#     dirname(/proc/self/exe)/assets/<name>, which is how the bundled
+#     silero_vad and marblenet_vad models resolve with nothing downloaded.
+#
+# backend/cpp/llama-cpp/package.sh answers the first of these by keeping
+# lib/ld.so and moving the ggml objects INTO lib/. That does not generalise
+# here, because it would also drag assets/ into lib/ to keep the second
+# consumer working. Putting the loader in the package root instead makes
+# dirname(/proc/self/exe) the package root, so the binary, the ggml objects and
+# assets/ all sit in the one directory that all three mechanisms agree on.
+if [ -f "$CURDIR/ld.so" ]; then
+	exec "$CURDIR/ld.so" "$CURDIR/grpc-server" "$@"
+fi
+
+exec "$CURDIR/grpc-server" "$@"