mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-08 15:28:23 -04:00
3a87d9e48f
* feat(vllm): macOS/Metal support via vllm-metal (MLX) Add an additive Apple-Silicon path to the existing vllm Python backend so vLLM runs on macOS via vllm-metal (github.com/vllm-project/vllm-metal). Spike outcome (proven on a real M4 / macOS 26.5, Qwen3-0.6B): - vllm-metal registers through vLLM's platform-plugin entry point (metal -> vllm_metal:register); MetalPlatform activates and runs on the GPU through MLX. - LocalAI's backend.py is UNCHANGED: AsyncEngineArgs(...) -> AsyncLLMEngine.from_engine_args transparently resolves to vLLM 0.23's v1 AsyncLLM MLX engine, and async generate produced correct output. - backend.py is NOT touched: its only empty_cache() call is CUDA-only (guarded by torch.cuda.is_available()), so the benign shutdown-only "Allocator for mps is not a DeviceAllocator" noise comes from vLLM's internal EngineCore teardown, not from our code. Changes (all gated behind a darwin condition; Linux/CUDA/ROCm/Intel paths are byte-for-byte unchanged): - install.sh: darwin branch forces PYTHON_VERSION=3.12 (vllm-metal requirement), creates/activates LocalAI's managed venv via ensureVenv, then reproduces vllm-metal's installer INTO that venv (build vLLM 0.23.0 from the release source tarball against requirements/cpu.txt, then install the prebuilt vllm-metal wheel from its latest GitHub release), and runs runProtogen. installRequirements is skipped on darwin. - backend-matrix.yml: add a vllm includeDarwin entry (mps, python). - index.yaml: add metal capability + concrete metal-vllm / metal-vllm-development child entries mirroring the metal-kitten-tts template. Version coupling: vllm-metal pins vLLM 0.23.0, equal to LocalAI's current vllm pin. Bumping vllm must be coordinated with a supporting vllm-metal release; documented in install.sh and requirements-cublas13-after.txt. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * chore(vllm): track the darwin vllm-metal pin via the autobumper The Apple Silicon build pinned vLLM 0.23.0 as a hidden string in install.sh while floating the vllm-metal wheel on releases/latest - the two could drift apart silently. Make both a tracked, reproducible pair (VLLM_METAL_VERSION + VLLM_VERSION), fetch the wheel by tag, and add .github/bump_vllm_metal.sh wired into bump_deps.yaml. It tracks vllm-project/vllm-metal (not vllm/vllm latest), reading the coupled vLLM source version from vllm-metal's own installer, and opens a bump PR - mirroring the existing bump_vllm_wheel.sh for the cu130 wheel. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * chore(vllm): derive the darwin vLLM version, drop the second pin Follow-up: VLLM_VERSION was still a hardcoded string duplicating what VLLM_METAL_VERSION already determines. Derive it at install time from vllm-metal's own installer (vllm_v=) at the pinned tag - one source of truth, no second value to drift. The bumper now touches only VLLM_METAL_VERSION; the derivation is immutable per tag, so builds stay reproducible. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * fix(vllm): fetch the vllm-metal wheel without the GitHub API The darwin build resolved the wheel URL via api.github.com, whose unauthenticated rate limit (60/hr per IP) 403s on shared macOS runners (observed after the 9-min vLLM source build). Construct the release-asset download URL deterministically from the pinned tag and the cp312/arm64 wheel name instead - no API call, no rate limit. Verified the URL resolves (200). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * fix(vllm): fail Score cleanly when the engine returns no prompt_logprobs Audit of the Score path against vllm-metal (MLX on macOS): the engine accepts SamplingParams(prompt_logprobs=1) but returns an all-None prompt_logprobs list rather than computing it, so scoring is not supported there. The old guard treated the truthy [None] list as valid and silently scored every candidate as 0. Detect the all-None case and return UNIMPLEMENTED instead. No-op on Linux/CUDA, which populate real entries. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
56 lines
2.5 KiB
Bash
Executable File
56 lines
2.5 KiB
Bash
Executable File
#!/bin/bash
|
|
# Bump the single vllm-metal pin (VLLM_METAL_VERSION) in the vLLM backend's
|
|
# darwin (Apple Silicon) install path. The macOS/Metal build
|
|
# (backend/python/vllm/install.sh, Darwin branch) installs vllm-metal, which is
|
|
# version-locked to a specific vLLM source release. install.sh derives that vLLM
|
|
# version at build time from vllm-metal's own installer (`vllm_v=`) at the pinned
|
|
# tag, so there is only ONE value to bump here -- mirroring bump_vllm_wheel.sh,
|
|
# which bumps the Linux cu130 wheel pin.
|
|
#
|
|
# This deliberately tracks vllm-project/vllm-metal, NOT vllm-project/vllm: the
|
|
# darwin build can only use the exact vLLM version vllm-metal supports, so it may
|
|
# lag the Linux pin (requirements-cublas13-after.txt) until vllm-metal catches up.
|
|
set -xe
|
|
REPO=$1 # vllm-project/vllm-metal
|
|
FILE=$2 # backend/python/vllm/install.sh
|
|
VAR=$3 # VLLM_METAL_VERSION (used for the workflow's output file names)
|
|
|
|
if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
|
|
echo "usage: $0 <repo> <install-file> <var-name>" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# vllm-metal ships frequent dev releases, all flagged as non-prerelease, so
|
|
# /releases/latest returns the newest one (with its cp312 wheel asset).
|
|
LATEST_TAG=$(curl -sS -H "Accept: application/vnd.github+json" \
|
|
"https://api.github.com/repos/$REPO/releases/latest" \
|
|
| python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
|
|
|
|
# The coupled vLLM source version lives in vllm-metal's installer at that tag.
|
|
NEW_VLLM_VERSION=$(curl -fsSL \
|
|
"https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \
|
|
| grep -oE 'vllm_v="[0-9]+\.[0-9]+\.[0-9]+"' | head -1 | cut -d'"' -f2)
|
|
|
|
if [ -z "$LATEST_TAG" ] || [ -z "$NEW_VLLM_VERSION" ]; then
|
|
echo "Could not resolve vllm-metal tag ($LATEST_TAG) or its vllm_v ($NEW_VLLM_VERSION)." >&2
|
|
exit 1
|
|
fi
|
|
|
|
set +e
|
|
CURRENT_TAG=$(grep -oE 'VLLM_METAL_VERSION="[^"]*"' "$FILE" | head -1 | cut -d'"' -f2)
|
|
set -e
|
|
|
|
# Rewrite the single pin. install.sh derives VLLM_VERSION from this tag at build
|
|
# time, so there is nothing else to touch. peter-evans/create-pull-request opens
|
|
# no PR on a clean tree, so a no-op rewrite (already current) is safe.
|
|
sed -i "$FILE" \
|
|
-e "s|VLLM_METAL_VERSION=\"[^\"]*\"|VLLM_METAL_VERSION=\"$LATEST_TAG\"|"
|
|
|
|
if [ -z "$CURRENT_TAG" ]; then
|
|
echo "Could not find VLLM_METAL_VERSION=\"...\" in $FILE." >&2
|
|
exit 0
|
|
fi
|
|
|
|
echo "vllm-metal ${CURRENT_TAG} -> ${LATEST_TAG} (builds vLLM ${NEW_VLLM_VERSION}): https://github.com/$REPO/releases/tag/${LATEST_TAG}" >> "${VAR}_message.txt"
|
|
echo "${LATEST_TAG}" >> "${VAR}_commit.txt"
|