mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
a4c9698daf
vllm.cpp's CUDA kernels need the CUDA 13 toolchain: 12.x nvcc lacks compute_121a and its ptxas rejects the sm_120a NVFP4 MMA kernels, so backend/go/vllm-cpp/Makefile ships no CUDA 12 variant and the arm64 CUDA build targets sm_121a (GB10 / DGX Spark) only. backend/index.yaml declared nvidia-l4t and nvidia-l4t-cuda-13 but no nvidia-l4t-cuda-12, so a Jetson AGX Orin (sm_87, JetPack 6) fell through SystemState.Capability's "default" catch-all and silently installed cpu-vllm-cpp. That is the right build for that host, but it was indistinguishable from an oversight both to a reader of the index and to a user wondering why their GPU box pulled a CPU backend. Map nvidia-cuda-12 and nvidia-l4t-cuda-12 explicitly at the CPU build, state the Blackwell-only constraint in the gallery description and the backend README, and add specs that pin the routing. No behaviour change: these hosts already resolved to the CPU build via the catch-all. The README also records the image-tag trap behind the same symptom on a supported host: /run/localai/capability is baked in at image build time, so a DGX Spark on the CUDA 12 -nvidia-l4t-arm64 image reports nvidia-l4t-cuda-12 and gets the CPU build; -nvidia-l4t-arm64-cuda-13 (or LOCALAI_FORCE_META_BACKEND_CAPABILITY) gets the GPU one. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>