mirror of
https://github.com/mudler/LocalAI.git
synced 2026-04-19 06:20:42 -04:00
b4e30692a2
* feat(backends): add sglang Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(sglang): force AVX-512 CXXFLAGS and disable CI e2e job sgl-kernel's shm.cpp uses __m512 AVX-512 intrinsics unconditionally; -march=native fails on CI runners without AVX-512 in /proc/cpuinfo. Force -march=sapphirerapids so the build always succeeds, matching sglang upstream's docker/xeon.Dockerfile recipe. The resulting binary still requires an AVX-512 capable CPU at runtime, so disable tests-sglang-grpc in test-extra.yml for the same reason tests-vllm-grpc is disabled. Local runs with make test-extra-backend-sglang still work on hosts with the right SIMD baseline. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(sglang): patch CMakeLists.txt instead of CXXFLAGS for AVX-512 CXXFLAGS with -march=sapphirerapids was being overridden by add_compile_options(-march=native) in sglang's CPU CMakeLists.txt, since CMake appends those flags after CXXFLAGS. Sed-patch the CMakeLists.txt directly after cloning to replace -march=native. --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
30 lines
952 B
Bash
Executable File
30 lines
952 B
Bash
Executable File
#!/bin/bash
|
|
|
|
backend_dir=$(dirname $(realpath $0))
|
|
|
|
if [ -d $backend_dir/common ]; then
|
|
source $backend_dir/common/libbackend.sh
|
|
else
|
|
source $backend_dir/../common/libbackend.sh
|
|
fi
|
|
|
|
# sglang's CPU kernel references LLVM OpenMP (__kmpc_*) symbols that are
|
|
# not declared in its NEEDED list — they get resolved through LD_PRELOAD
|
|
# of libiomp5.so in sglang's own docker/xeon.Dockerfile. Do the same here.
|
|
# Harmless on GPU builds where libiomp5.so is absent.
|
|
if [ -f "${backend_dir}/lib/libiomp5.so" ]; then
|
|
if [ -n "${LD_PRELOAD:-}" ]; then
|
|
export LD_PRELOAD="${backend_dir}/lib/libiomp5.so:${LD_PRELOAD}"
|
|
else
|
|
export LD_PRELOAD="${backend_dir}/lib/libiomp5.so"
|
|
fi
|
|
fi
|
|
|
|
# sglang CPU engine requires this env var to switch to the CPU backend.
|
|
# No-op on GPU builds. See docker/xeon.Dockerfile in sglang upstream.
|
|
if [ -f "${backend_dir}/lib/libiomp5.so" ]; then
|
|
export SGLANG_USE_CPU_ENGINE=1
|
|
fi
|
|
|
|
startBackend $@
|