feat(backends): add CUDA 13 + L4T arm64 CUDA 13 variants for vllm/vllm-omni/sglang (#9553)

* feat(backends): add CUDA 13 + L4T arm64 CUDA 13 variants for vllm/vllm-omni/sglang

Adds new build profiles mirroring the diffusers/ace-step pattern so vLLM
serving (and SGLang on arm64) can be deployed on CUDA 13 hosts and
JetPack 7 boards:

- vllm: cublas13 (PyPI cu130 channel) + l4t13 (jetson-ai-lab SBSA cu130
  prebuilt vllm + flash-attn).
- vllm-omni: cublas13 + l4t13. Floats vllm version on cu13 since vllm
  0.19+ ships cu130 wheels by default and vllm-omni tracks vllm master;
  cu12 path keeps the 0.14.0 pin to avoid disturbing existing images.
- sglang: l4t13 arm64 only — uses the prebuilt sglang wheel from the
  jetson-ai-lab SBSA cu130 index, so no source build is needed.
  Cublas13 sglang on x86_64 is intentionally deferred.

CI matrix gains five new images (-gpu-nvidia-cuda-13-vllm{,-omni},
-nvidia-l4t-cuda-13-arm64-{vllm,vllm-omni,sglang}); backend/index.yaml
gains the matching capability keys (nvidia-cuda-13, nvidia-l4t-cuda-13)
and latest/development merge entries.

Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Write] [Bash]

* fix(backends): use unsafe-best-match index strategy on l4t13 builds

The jetson-ai-lab SBSA cu130 index lists transitive deps (decord, etc.)
at limited versions / older Python ABIs. uv defaults to the first index
that contains a package and refuses to fall through to PyPI, so sglang
l4t13 build fails resolving decord. Mirror the existing cpu sglang
profile by setting --index-strategy=unsafe-best-match on l4t13 across
the three backends, and apply it to the explicit vllm install line in
vllm-omni's install.sh (which doesn't honor EXTRA_PIP_INSTALL_FLAGS).

Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Bash]

* fix(sglang): drop [all] extras on l4t13, floor version at 0.5.0

The [all] extra brings in outlines→decord, and decord has no aarch64
cp312 wheel on PyPI nor the jetson-ai-lab index (only legacy cp35-cp37
tags). With unsafe-best-match enabled, uv backtracked through sglang
versions trying to satisfy decord and silently landed on
sglang==0.1.16, an ancient version with an entirely different dep
tree (cloudpickle/outlines 0.0.44, etc.).

Drop [all] so decord is no longer required, and floor sglang at 0.5.0
to prevent any future resolver misfire from degrading the version
again.

Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

backend/python/sglang/install.sh

@@ -23,6 +23,19 @@ if [ "x${BUILD_PROFILE}" == "xcpu" ]; then
     EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
 fi
 
+# JetPack 7 / L4T arm64 wheels are built for cp312 and shipped via
+# pypi.jetson-ai-lab.io. Bump the venv Python so the prebuilt sglang
+# wheel resolves cleanly. unsafe-best-match is required because the
+# jetson-ai-lab index lists transitive deps (e.g. decord) at older
+# versions only — without it uv refuses to fall through to PyPI for a
+# compatible wheel and resolution fails.
+if [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
+    PYTHON_VERSION="3.12"
+    PYTHON_PATCH="12"
+    PY_STANDALONE_TAG="20251120"
+    EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
+fi
+
 # sglang's CPU path has no prebuilt wheel on PyPI — upstream publishes
 # a separate pyproject_cpu.toml that must be swapped in before `pip install`.
 # Reference: docker/xeon.Dockerfile in the sglang upstream repo.

backend/python/sglang/requirements-l4t13.txt

@@ -0,0 +1,12 @@
+--extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+accelerate
+torch
+torchvision
+torchaudio
+transformers
+# Drop the [all] extra: it pulls outlines/decord, and decord has no
+# aarch64 cp312 wheel anywhere (PyPI nor the jetson-ai-lab index ships
+# only legacy cp35-cp37). With [all] uv backtracks through versions
+# trying to satisfy decord and lands on sglang==0.1.16. Floor at 0.5.0
+# so uv can't silently downgrade if a future resolution misfires.
+sglang>=0.5.0

backend/python/vllm-omni/install.sh

@@ -12,11 +12,15 @@ else
     source $backend_dir/../common/libbackend.sh
 fi
 
-# Handle l4t build profiles (Python 3.12, pip fallback) if needed
+# Handle l4t build profiles (Python 3.12, pip fallback) if needed.
+# unsafe-best-match is required on l4t13 because the jetson-ai-lab index
+# lists transitive deps at limited versions — without it uv pins to the
+# first matching index and fails to resolve a compatible wheel from PyPI.
 if [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
   PYTHON_VERSION="3.12"
   PYTHON_PATCH="12"
   PY_STANDALONE_TAG="20251120"
+  EXTRA_PIP_INSTALL_FLAGS="${EXTRA_PIP_INSTALL_FLAGS:-} --index-strategy=unsafe-best-match"
 fi
 
 if [ "x${BUILD_PROFILE}" == "xl4t12" ]; then
@@ -26,7 +30,11 @@ fi
 # Install base requirements first
 installRequirements
 
-# Install vllm based on build type
+# Install vllm based on build type. vllm-omni tracks vllm master from
+# source (cloned below) so we leave the upstream vllm dependency unpinned
+# — vllm 0.19+ ships cu130 wheels by default, which is what we want for
+# cublas13. Older cuda12/rocm/cpu paths still resolve a compatible wheel
+# from the relevant channel.
 if [ "x${BUILD_TYPE}" == "xhipblas" ]; then
     # ROCm
     if [ "x${USE_PIP}" == "xtrue" ]; then
@@ -34,8 +42,26 @@ if [ "x${BUILD_TYPE}" == "xhipblas" ]; then
     else
         uv pip install vllm==0.14.0 --extra-index-url https://wheels.vllm.ai/rocm/0.14.0/rocm700
     fi
+elif [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
+    # JetPack 7 / L4T arm64 cu130 — vllm comes from the prebuilt SBSA wheel
+    # at jetson-ai-lab. Version is unpinned: the index ships whatever build
+    # matches the cu130/cp312 ABI. unsafe-best-match lets uv fall through
+    # to PyPI for transitive deps not present on the jetson-ai-lab index.
+    if [ "x${USE_PIP}" == "xtrue" ]; then
+        pip install vllm --extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+    else
+        uv pip install --index-strategy=unsafe-best-match vllm --extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+    fi
+elif [ "x${BUILD_PROFILE}" == "xcublas13" ]; then
+    # vllm 0.19+ defaults to cu130 wheels on PyPI, no extra index needed.
+    if [ "x${USE_PIP}" == "xtrue" ]; then
+        pip install vllm --torch-backend=auto
+    else
+        uv pip install vllm --torch-backend=auto
+    fi
 elif [ "x${BUILD_TYPE}" == "xcublas" ] || [ "x${BUILD_TYPE}" == "x" ]; then
-    # CUDA (default) or CPU
+    # cuda12 / CPU — keep the 0.14.0 pin for compatibility with the existing
+    # cuda12 vllm-omni image; bumping should be its own change.
     if [ "x${USE_PIP}" == "xtrue" ]; then
         pip install vllm==0.14.0 --torch-backend=auto
     else

backend/python/vllm-omni/requirements-cublas13.txt

@@ -0,0 +1,5 @@
+--extra-index-url https://download.pytorch.org/whl/cu130
+accelerate
+torch
+transformers
+bitsandbytes

backend/python/vllm-omni/requirements-l4t13.txt

@@ -0,0 +1,13 @@
+--extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+accelerate
+torch
+torchvision
+torchaudio
+transformers
+bitsandbytes
+flash-attn
+diffusers
+librosa
+soundfile
+pillow
+numpy

backend/python/vllm/install.sh

@@ -32,6 +32,22 @@ if [ "x${BUILD_PROFILE}" == "xcpu" ]; then
     EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
 fi
 
+# JetPack 7 / L4T arm64 wheels (torch, vllm, flash-attn) live on
+# pypi.jetson-ai-lab.io and are built for cp312, so bump the venv Python
+# accordingly. JetPack 6 keeps cp310 + USE_PIP=true. unsafe-best-match
+# is required because the jetson-ai-lab index lists transitive deps at
+# limited versions — without it uv pins to the first matching index and
+# fails to resolve a compatible wheel from PyPI.
+if [ "x${BUILD_PROFILE}" == "xl4t12" ]; then
+    USE_PIP=true
+fi
+if [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
+    PYTHON_VERSION="3.12"
+    PYTHON_PATCH="12"
+    PY_STANDALONE_TAG="20251120"
+    EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
+fi
+
 # FROM_SOURCE=true on a CPU build skips the prebuilt vllm wheel in
 # requirements-cpu-after.txt and compiles vllm locally against the host's
 # actual CPU. Not used by default because it takes ~30-40 minutes, but

backend/python/vllm/requirements-cublas13-after.txt

@@ -0,0 +1,2 @@
+--extra-index-url https://download.pytorch.org/whl/cu130
+vllm

backend/python/vllm/requirements-cublas13.txt

@@ -0,0 +1,5 @@
+--extra-index-url https://download.pytorch.org/whl/cu130
+accelerate
+torch
+transformers
+bitsandbytes

backend/python/vllm/requirements-l4t13-after.txt

@@ -0,0 +1,2 @@
+--extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+vllm

backend/python/vllm/requirements-l4t13.txt

@@ -0,0 +1,8 @@
+--extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+accelerate
+torch
+torchvision
+torchaudio
+transformers
+bitsandbytes
+flash-attn