mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 01:48:06 -04:00
fix(ci): authenticate the nightly dependency-bump API calls (#11042)
The "Bump Backend dependencies" workflow has failed every night for the last two weeks. #11012 fixed one cause (repos renamed under localai-org); what is left is rate limiting. bump_deps.sh fans out to ~25 parallel matrix jobs that each query api.github.com anonymously. Anonymous calls are capped at 60/hour per source IP and GitHub-hosted runners egress through shared NAT addresses, so a random handful of jobs draw HTTP 403 and die at curl exit 22 with an empty response. Last night that hit ggml-org/whisper.cpp and mudler/depth-anything.cpp -- both public and resolvable, nothing wrong with either pin. Route every bump script through a shared gh_curl helper that sends GITHUB_TOKEN when present (1000/hour instead of 60) and retries transient failures, including the 403s that plain --retry ignores. The helper suppresses xtrace around the call so the Authorization header cannot land in a public job log. bump_docs.sh had a sharper version of the same bug: it piped an unchecked response into `jq -r .tag_name`, so a throttled request resolved to the string "null" and would have been published as the docs version. It now refuses to write anything it cannot resolve to a tag. Verified locally by running all four scripts end to end against their real upstreams: correct SHAs/tags written, exit 0; a nonexistent repo now fails with a named diagnostic instead of a bare exit 22 and leaves the pinned file untouched; the token is absent from the xtrace output; and the scripts still work unauthenticated. Assisted-by: Claude:opus-4.8 [Claude Code] Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
This commit is contained in:
committed by
GitHub
parent
5c96e097ba
commit
47c0e06198
12
.github/bump_deps.sh
vendored
12
.github/bump_deps.sh
vendored
@@ -1,5 +1,8 @@
|
||||
#!/bin/bash
|
||||
set -xe
|
||||
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
|
||||
|
||||
REPO=$1
|
||||
BRANCH=$2
|
||||
VAR=$3
|
||||
@@ -9,10 +12,11 @@ if [ -z "$FILE" ]; then
|
||||
FILE="Makefile"
|
||||
fi
|
||||
|
||||
# -L so a renamed/transferred upstream repo (GitHub answers 301) still
|
||||
# resolves instead of handing us the redirect body, and -f so an HTTP error
|
||||
# aborts the run rather than letting an error page reach sed below.
|
||||
LAST_COMMIT=$(curl -sfL -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH")
|
||||
# gh_curl follows redirects so a renamed/transferred upstream repo (GitHub
|
||||
# answers 301) still resolves, and fails on HTTP errors rather than letting an
|
||||
# error page reach sed below. `|| true` keeps a failed lookup from aborting the
|
||||
# script at exit 22 with no context — the SHA guard below reports it instead.
|
||||
LAST_COMMIT=$(gh_curl -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH" || true)
|
||||
|
||||
# Guard the sed input: anything that is not a bare 40-hex SHA (an API error
|
||||
# body, an empty response) would otherwise be spliced into the Makefile pin —
|
||||
|
||||
13
.github/bump_docs.sh
vendored
13
.github/bump_docs.sh
vendored
@@ -1,7 +1,18 @@
|
||||
#!/bin/bash
|
||||
set -xe
|
||||
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
|
||||
|
||||
REPO=$1
|
||||
|
||||
LATEST_TAG=$(curl -s "https://api.github.com/repos/$REPO/releases/latest" | jq -r '.tag_name')
|
||||
LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
|
||||
"https://api.github.com/repos/$REPO/releases/latest" | jq -r '.tag_name')
|
||||
|
||||
# jq prints the string "null" for a missing key, so a throttled or otherwise
|
||||
# unexpected API response would otherwise be published as the docs version.
|
||||
if [ -z "$LATEST_TAG" ] || [ "$LATEST_TAG" = "null" ]; then
|
||||
echo "Refusing to bump docs version: could not resolve the latest release tag for $REPO." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cat <<< $(jq ".version = \"$LATEST_TAG\"" docs/data/version.json) > docs/data/version.json
|
||||
|
||||
7
.github/bump_vllm_metal.sh
vendored
7
.github/bump_vllm_metal.sh
vendored
@@ -11,6 +11,9 @@
|
||||
# darwin build can only use the exact vLLM version vllm-metal supports, so it may
|
||||
# lag the Linux pin (requirements-cublas13-after.txt) until vllm-metal catches up.
|
||||
set -xe
|
||||
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
|
||||
|
||||
REPO=$1 # vllm-project/vllm-metal
|
||||
FILE=$2 # backend/python/vllm/install.sh
|
||||
VAR=$3 # VLLM_METAL_VERSION (used for the workflow's output file names)
|
||||
@@ -22,12 +25,12 @@ fi
|
||||
|
||||
# vllm-metal ships frequent dev releases, all flagged as non-prerelease, so
|
||||
# /releases/latest returns the newest one (with its cp312 wheel asset).
|
||||
LATEST_TAG=$(curl -sS -H "Accept: application/vnd.github+json" \
|
||||
LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
|
||||
"https://api.github.com/repos/$REPO/releases/latest" \
|
||||
| python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
|
||||
|
||||
# The coupled vLLM source version lives in vllm-metal's installer at that tag.
|
||||
NEW_VLLM_VERSION=$(curl -fsSL \
|
||||
NEW_VLLM_VERSION=$(gh_curl \
|
||||
"https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \
|
||||
| grep -oE 'vllm_v="[0-9]+\.[0-9]+\.[0-9]+"' | head -1 | cut -d'"' -f2)
|
||||
|
||||
|
||||
5
.github/bump_vllm_wheel.sh
vendored
5
.github/bump_vllm_wheel.sh
vendored
@@ -9,6 +9,9 @@
|
||||
# vars in Makefiles; this script handles the two-value rewrite specific to the
|
||||
# vLLM requirements file.
|
||||
set -xe
|
||||
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
|
||||
|
||||
REPO=$1 # vllm-project/vllm
|
||||
FILE=$2 # backend/python/vllm/requirements-cublas13-after.txt
|
||||
VAR=$3 # VLLM_VERSION (used for output file names so the workflow can read them)
|
||||
@@ -19,7 +22,7 @@ if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
|
||||
fi
|
||||
|
||||
# /releases/latest returns the most recent non-prerelease tag.
|
||||
LATEST_TAG=$(curl -sS -H "Accept: application/vnd.github+json" \
|
||||
LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
|
||||
"https://api.github.com/repos/$REPO/releases/latest" \
|
||||
| python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
|
||||
|
||||
|
||||
39
.github/gh_curl.sh
vendored
Executable file
39
.github/gh_curl.sh
vendored
Executable file
@@ -0,0 +1,39 @@
|
||||
#!/bin/bash
|
||||
# Shared curl wrapper for the nightly dependency-bump scripts.
|
||||
#
|
||||
# The bump workflow fans out to ~25 parallel matrix jobs, each querying
|
||||
# api.github.com. Anonymous API calls are capped at 60/hour per source IP and
|
||||
# GitHub-hosted runners egress through shared NAT addresses, so a random handful
|
||||
# of jobs were getting rate-limited (HTTP 403 -> curl exit 22, empty response)
|
||||
# every single night. Authenticating with GITHUB_TOKEN lifts the ceiling to
|
||||
# 1000/hour; the retries absorb whatever transient blips remain.
|
||||
|
||||
# Wraps curl with GitHub auth (when a token is present) plus retry/timeout
|
||||
# hardening. Callers pass their own headers and the URL.
|
||||
gh_curl() {
|
||||
# The bump scripts run under `set -x`; without this the Authorization header
|
||||
# would be echoed into the job log on every call.
|
||||
local had_xtrace=0
|
||||
case "$-" in
|
||||
*x*) had_xtrace=1; set +x ;;
|
||||
esac
|
||||
|
||||
local args=(
|
||||
--silent --show-error --location --fail
|
||||
# --retry-all-errors so 403 rate-limit responses are retried too; plain
|
||||
# --retry only covers 408/429/5xx. curl honours Retry-After when sent.
|
||||
--retry 5 --retry-delay 3 --retry-all-errors
|
||||
--connect-timeout 15 --max-time 60
|
||||
)
|
||||
if [ -n "${GITHUB_TOKEN:-}" ]; then
|
||||
args+=(--header "Authorization: Bearer ${GITHUB_TOKEN}")
|
||||
fi
|
||||
|
||||
curl "${args[@]}" "$@"
|
||||
local rc=$?
|
||||
|
||||
if [ "$had_xtrace" -eq 1 ]; then
|
||||
set -x
|
||||
fi
|
||||
return $rc
|
||||
}
|
||||
10
.github/workflows/bump_deps.yaml
vendored
10
.github/workflows/bump_deps.yaml
vendored
@@ -115,6 +115,12 @@ jobs:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Bump dependencies 🔧
|
||||
id: bump
|
||||
env:
|
||||
# This job fans out to ~25 parallel matrix entries, all querying
|
||||
# api.github.com from runner IPs that share the 60/hour anonymous
|
||||
# rate limit. Authenticating raises it to 1000/hour, which is what
|
||||
# kept a random handful of these red every night.
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
bash .github/bump_deps.sh ${{ matrix.repository }} ${{ matrix.branch }} ${{ matrix.variable }} ${{ matrix.file }}
|
||||
{
|
||||
@@ -151,6 +157,8 @@ jobs:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Bump vLLM cu130 wheel pin 🔧
|
||||
id: bump
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
bash .github/bump_vllm_wheel.sh vllm-project/vllm backend/python/vllm/requirements-cublas13-after.txt VLLM_VERSION
|
||||
{
|
||||
@@ -187,6 +195,8 @@ jobs:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Bump vllm-metal pin 🔧
|
||||
id: bump
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
bash .github/bump_vllm_metal.sh vllm-project/vllm-metal backend/python/vllm/install.sh VLLM_METAL_VERSION
|
||||
{
|
||||
|
||||
4
.github/workflows/bump_docs.yaml
vendored
4
.github/workflows/bump_docs.yaml
vendored
@@ -15,6 +15,10 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Bump dependencies 🔧
|
||||
env:
|
||||
# Authenticated API calls get 1000 req/hour instead of the 60/hour
|
||||
# anonymous cap that is shared across every job on the runner's IP.
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
bash .github/bump_docs.sh ${{ matrix.repository }}
|
||||
- name: Create Pull Request
|
||||
|
||||
Reference in New Issue
Block a user