chore(tests): Avoid network, sleep and more during tests (#11050)

* test: make coverage failures observable Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: parallelize coverage without remote fixtures Assisted-by: Codex:gpt-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: add offline resource infrastructure Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: enforce offline resource replay Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: harden offline resource refresh Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: expose slow coverage waits Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: eliminate avoidable wall-clock waits Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: remove repeated fixture startup waits Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: fix offline resource CI portability Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: cache Go modules before offline tests Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: drop the static network lint in favour of real isolation The offline test suite already prevents tests from reaching the network twice over: run-test-linux-offline.sh puts the test process in a cgroup and REJECTs egress outside the private ranges, and HardenedTransport installs testnetwork.LocalGuard to refuse dials that resolve to a public address. Both fail the test with a precise error at the moment of the dial. test-network-lint.sh added neither. Its diff stage defaulted to a HEAD base, so on a clean checkout it compared the tree against itself and inspected nothing; the branch's own commits were never examined. It only produced output when an earlier job step dirtied the tree, and then it matched a bare https?:// against whatever changed. make react-ui runs npm install rather than npm ci, so CI rewrote core/http/react-ui/package-lock.json and the lint reported an npm registry URL as forbidden test network access: + "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz", Its fingerprint stage was self-defeating in a quieter way: hashing the whole tree's network-mechanism inventory meant every rebase onto a master that touched any _test.go needed a manual baseline bump, so the check mostly caught its own staleness. Remove the script, its make target and the two prerequisite edges, along with the test-network: fixture markers that existed only to suppress it. The isolation itself is untouched. Assisted-by: Claude:claude-opus-5 [go vet] Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: keep hidden files in the offline test bundle artifact Cherry-picked from 15a37b0ac on the remote branch. The offline bundle lives under .cache/, which actions/upload-artifact skips by default, so the Linux job packed an artifact missing the very file the next step restores. The other half of 15a37b0ac moved test-network-lint out of the `test` and `test-coverage` prerequisite lists into a recipe line, so parallel make could not fingerprint the tree while generated fixtures were still changing. That is dropped: the preceding commit removes the lint entirely, and the race it worked around is one more reason a whole-tree fingerprint was the wrong mechanism. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * refactor: share bounded exponential backoff Use overflow-safe saturating arithmetic for retry delays across model import polling, downloads, registration, node operations, and model loading. Keep model import status checks responsive initially while capping their interval at 500ms. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: mirror Jetson Python wheels Keep the CUDA aarch64 wheel subset in GHCR and serve it as a local PEP 503 index during L4T backend builds, preserving last-known-good packages through upstream outages. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * docs(agents): index the Jetson wheels mirror Mention the GHCR-hosted L4T wheel mirror in the CI caching guide summary so maintainers can find its outage and cache documentation. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: add defensive build network proxy Record build destinations and byte counts, retry observable idempotent HTTP downloads, and isolate explorer database tests that race under coverage. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement updated backend trait Return unimplemented for image upscaling, matching the backend's other unsupported modalities after the protobuf API update. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): clear recovered proxy errors Do not mark a request failed when a later safe retry succeeds. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: require HTTPS build interception Inject a short-lived proxy CA into BuildKit and Dockerfile RUN steps, reject plain HTTP and opaque tunnels, and retain method/status/byte telemetry for verified HTTPS traffic. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): preserve system trust in unproxied builds Mount the generated interception CA at a dedicated secret path and add it to the trust bundle only in proxy-aware dependency stages. This prevents optional secret mounts from masking the system CA bundle in ordinary backend test builds. Install the requested Go toolchain before starting the proxy and satisfy cleanup error checks found by CI lint. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): persist build proxy trust Install the generated proxy CA through the system-managed local certificate directory so ca-certificates upgrades retain it. Avoid turning canceled matrix jobs into proxy cleanup failures. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): trust proxy in nested build scripts Install the build proxy CA before nested source fetches, route the DS4 package setup through the HTTPS mirror helper, and avoid repeated OCI setup in gallery behavior tests. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): use HTTPS apt sources for Bonsai Rewrite ARM64 package sources before installing GCC and check gallery fixture cleanup errors so the optimized tests satisfy errcheck. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(privacy-filter): trust build proxy CA Install the mounted build proxy certificate before privacy-filter's make target fetches its HTTPS sources, for both source and prebuilt builder paths.\n\nAssisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: fail on hidden offline egress Count cgroup-scoped firewall rejects and fail the offline test harness with bounded aggregate diagnostics. Inject the gen-audio GGUF probe so fixture-backed importer tests do not attempt real network access. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): preserve system CA trust Build a combined runner certificate bundle instead of replacing public roots with the generated proxy CA. Centralize additive container installation in the shared proxy CA helper. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
2026-09-12 22:33:54 -04:00 · 2026-08-19 10:59:31 +02:00
1 parent de3329e332
commit cb3bf7af3f
@@ -20,20 +20,15 @@ inputs:
  github-hosted-mirror:
    description: 'archive/security mirror URL for github-hosted runners (empty = upstream)'
    required: false
-    default: 'http://azure.archive.ubuntu.com'
+    default: 'https://archive.ubuntu.com'
  github-hosted-ports-mirror:
    description: 'ports.ubuntu.com mirror URL for github-hosted runners (empty = upstream)'
    required: false
-    default: 'http://azure.ports.ubuntu.com'
+    default: 'https://ports.ubuntu.com'
  self-hosted-mirror:
    description: 'archive/security mirror URL for self-hosted runners (empty = upstream)'
    required: false
-    # HTTP, not HTTPS: the bare ubuntu:24.04 builder image doesn't ship
-    # ca-certificates, so the very first apt-get update over TLS would
-    # fail with "No system certificates available" before it can install
-    # anything. apt validates package integrity via GPG signatures, so
-    # plain HTTP is safe for the archive itself.
-    default: 'http://mirrors.edge.kernel.org'
+    default: 'https://mirrors.edge.kernel.org'
  self-hosted-ports-mirror:
    description: 'ports.ubuntu.com mirror URL for self-hosted runners (empty = upstream)'
    required: false
@@ -41,7 +36,7 @@ inputs:
    # main /ubuntu/ archive — so arm64 builds 404 there. Leave ports
    # upstream by default. The original DDoS was on archive.ubuntu.com
    # so ports.ubuntu.com remains the path of least surprise.
-    default: ''
+    default: 'https://ports.ubuntu.com'

 outputs:
  effective-mirror:
@@ -0,0 +1,25 @@
+{
+  "upstream": "https://pypi.jetson-ai-lab.io",
+  "indexes": {
+    "jp6/cu129": [
+      "torch",
+      "torchvision",
+      "torchaudio",
+      "torchcodec",
+      "torchao",
+      "bitsandbytes",
+      "onnxruntime",
+      "ctranslate2"
+    ],
+    "jp7/cu130": [
+      "torch",
+      "torchvision",
+      "torchaudio",
+      "torchcodec",
+      "torchao",
+      "bitsandbytes",
+      "onnxruntime",
+      "ctranslate2"
+    ]
+  }
+}
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ca="${LOCALAI_BUILD_PROXY_CA:?build proxy CA is unset}"
+builder="${BUILDER_NAME:?Buildx builder name is unset}"
+container="$(docker ps --filter "name=buildx_buildkit_${builder}0" --format '{{.ID}}' | head -n1)"
+if test -z "$container"; then
+  echo 'BuildKit container not found' >&2
+  exit 1
+fi
+
+docker exec "$container" mkdir -p /usr/local/share/ca-certificates /etc/ssl/certs
+docker cp "$ca" "$container:/usr/local/share/ca-certificates/localai-build-proxy.crt"
+docker exec "$container" sh -eu -c '
+  if command -v update-ca-certificates >/dev/null 2>&1; then
+    update-ca-certificates
+  else
+    cat /usr/local/share/ca-certificates/localai-build-proxy.crt >>/etc/ssl/certs/ca-certificates.crt
+  fi
+'
+docker restart "$container" >/dev/null
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+set -euo pipefail
+output="${RUNNER_TEMP}/localai-build-proxy"
+mkdir -p "$output"
+CGO_ENABLED=0 GOCACHE="${RUNNER_TEMP}/go-build-cache" go build -o "$output/build-proxy" ./cmd/build-proxy
+nohup "$output/build-proxy" --listen 127.0.0.1:18080 --output "$output" >"$output/proxy.log" 2>&1 &
+echo "$!" >"$output/proxy.pid"
+for _ in $(seq 1 50); do
+  grep -q '^ca=' "$output/proxy.log" && break
+  sleep 0.1
+done
+grep '^proxy=' "$output/proxy.log"
+grep '^ca=' "$output/proxy.log"
+proxy_ca="$output/ca/ca.crt"
+ca_bundle="$output/ca/ca-bundle.crt"
+system_ca=""
+for candidate in \
+  /etc/ssl/certs/ca-certificates.crt \
+  /etc/ssl/cert.pem \
+  /etc/pki/tls/certs/ca-bundle.crt \
+  /etc/openssl/certs/ca-certificates.crt; do
+  if test -s "$candidate"; then
+    system_ca="$candidate"
+    break
+  fi
+done
+if test -z "$system_ca"; then
+  echo 'build proxy: unable to find the runner system CA bundle' >&2
+  exit 1
+fi
+cat "$system_ca" "$proxy_ca" >"$ca_bundle"
+{
+  echo "LOCALAI_BUILD_PROXY=http://127.0.0.1:18080"
+  echo "LOCALAI_BUILD_PROXY_OUTPUT=$output"
+  echo "LOCALAI_BUILD_PROXY_CA=$proxy_ca"
+  echo "LOCALAI_BUILD_PROXY_CA_BUNDLE=$ca_bundle"
+  echo "HTTP_PROXY=http://127.0.0.1:18080"
+  echo "HTTPS_PROXY=http://127.0.0.1:18080"
+  echo "http_proxy=http://127.0.0.1:18080"
+  echo "https_proxy=http://127.0.0.1:18080"
+  echo "SSL_CERT_FILE=$ca_bundle"
+  echo "CURL_CA_BUNDLE=$ca_bundle"
+  echo "REQUESTS_CA_BUNDLE=$ca_bundle"
+  echo "GIT_SSL_CAINFO=$ca_bundle"
+  echo "NODE_EXTRA_CA_CERTS=$proxy_ca"
+  echo "NO_PROXY=localhost,127.0.0.1"
+  echo "no_proxy=localhost,127.0.0.1"
+} >>"$GITHUB_ENV"
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+set -euo pipefail
+if test -z "${LOCALAI_BUILD_PROXY_OUTPUT:-}"; then
+  echo 'Build proxy did not start; skipping inventory finalization'
+  exit 0
+fi
+output="$LOCALAI_BUILD_PROXY_OUTPUT"
+if test -f "$output/proxy.pid"; then
+  kill -TERM "$(cat "$output/proxy.pid")" 2>/dev/null || true
+  for _ in $(seq 1 50); do
+    test -f "$output/summary.json" && break
+    sleep 0.1
+  done
+fi
+
+# Later artifact uploads and action post-hooks must not target a stopped proxy.
+{
+  echo 'HTTP_PROXY='
+  echo 'HTTPS_PROXY='
+  echo 'http_proxy='
+  echo 'https_proxy='
+  echo 'SSL_CERT_FILE='
+  echo 'CURL_CA_BUNDLE='
+  echo 'REQUESTS_CA_BUNDLE='
+  echo 'GIT_SSL_CAINFO='
+  echo 'NODE_EXTRA_CA_CERTS='
+} >>"$GITHUB_ENV"
+if test -f "$output/summary.json"; then
+  {
+    echo '### Build network inventory'
+    echo
+    echo 'HTTPS without the generated CA is reported as CONNECT because its HTTP method is encrypted.'
+    echo
+    echo '```json'
+    cat "$output/summary.json"
+    echo '```'
+  } >>"$GITHUB_STEP_SUMMARY"
+fi
+
+# A matrix cancellation can interrupt checkout or a BuildKit request at any
+# point. Preserve whatever inventory exists, but do not replace the canceled
+# conclusion with a misleading proxy-enforcement failure.
+if test "${LOCALAI_BUILD_JOB_STATUS:-}" = cancelled; then
+  echo 'Build was cancelled; skipping network inventory enforcement'
+  exit 0
+fi
+
+if ! test -s "$output/events.jsonl"; then
+  echo 'Build proxy produced no network inventory' >&2
+  exit 1
+fi
+if grep -qE '"method":"CONNECT"|"error":"plain HTTP is forbidden"' "$output/events.jsonl"; then
+  echo 'Build traffic bypassed HTTPS interception or attempted plain HTTP' >&2
+  exit 1
+fi
@@ -153,9 +153,27 @@ jobs:
        with:
          platforms: all

+      - name: Set up Go for build proxy
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+
+      - name: Start build network proxy
+        run: .github/scripts/start-build-proxy.sh
+
      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@master
+        with:
+          driver-opts: |
+            network=host
+            env.http_proxy=${{ env.LOCALAI_BUILD_PROXY }}
+            env.https_proxy=${{ env.LOCALAI_BUILD_PROXY }}
+
+      - name: Trust build proxy CA in BuildKit
+        env:
+          BUILDER_NAME: ${{ steps.buildx.outputs.name }}
+        run: .github/scripts/inject-build-proxy-ca.sh

      - name: Login to DockerHub
        if: github.event_name != 'pull_request'
@@ -181,6 +199,41 @@ jobs:
        id: deps_refresh
        run: echo "key=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"

+      - name: Login to ghcr.io (jetson wheels mirror)
+        if: inputs.build-type == 'l4t'
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ github.token }}
+
+      # l4t builds pull their CUDA aarch64 torch wheels from
+      # pypi.jetson-ai-lab.io, which has a history of multi-hour 502 outages
+      # that fail every l4t job. jetson-wheels.yml mirrors those wheels into
+      # ghcr weekly; here we hand the mirror image to Dockerfile.python,
+      # which serves it as a local package index during pip install (see
+      # installRequirements in backend/python/common/libbackend.sh). Falls
+      # back to scratch — i.e. building straight against the upstream index —
+      # when the mirror tag doesn't exist yet, so the mirror can bootstrap
+      # without a chicken-and-egg failure.
+      - name: Resolve jetson wheels mirror image
+        id: jetson_wheels
+        if: inputs.build-type == 'l4t'
+        run: |
+          repo="ghcr.io/$(echo "${{ github.repository }}" | tr '[:upper:]' '[:lower:]')/jetson-wheels"
+          case "${{ inputs.cuda-major-version }}" in
+            12) tag="jp6-cu129" ;;
+            13) tag="jp7-cu130" ;;
+            *) tag="" ;;
+          esac
+          img=""
+          if [ -n "$tag" ] && docker buildx imagetools inspect "$repo:$tag" >/dev/null 2>&1; then
+            img="$repo:$tag"
+          else
+            echo "jetson wheels image $repo:$tag not found; building against the upstream index"
+          fi
+          echo "image=$img" >> "$GITHUB_OUTPUT"
+
      - name: Build and push by digest
        id: build
        uses: docker/build-push-action@v7
@@ -188,6 +241,9 @@ jobs:
        with:
          builder: ${{ steps.buildx.outputs.name }}
          build-args: |
+            HTTP_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            HTTPS_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            NO_PROXY=localhost,127.0.0.1
            BUILD_TYPE=${{ inputs.build-type }}
            SKIP_DRIVERS=${{ inputs.skip-drivers }}
            CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
@@ -201,6 +257,9 @@ jobs:
            DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
            BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
            BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
+            JETSON_WHEELS_IMAGE=${{ steps.jetson_wheels.outputs.image || 'scratch' }}
+          secret-files: |
+            build_proxy_ca=${{ env.LOCALAI_BUILD_PROXY_CA }}
          context: ${{ inputs.context }}
          file: ${{ inputs.dockerfile }}
          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
@@ -260,6 +319,9 @@ jobs:
        with:
          builder: ${{ steps.buildx.outputs.name }}
          build-args: |
+            HTTP_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            HTTPS_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            NO_PROXY=localhost,127.0.0.1
            BUILD_TYPE=${{ inputs.build-type }}
            SKIP_DRIVERS=${{ inputs.skip-drivers }}
            CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
@@ -273,6 +335,9 @@ jobs:
            DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
            BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
            BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
+            JETSON_WHEELS_IMAGE=${{ steps.jetson_wheels.outputs.image || 'scratch' }}
+          secret-files: |
+            build_proxy_ca=${{ env.LOCALAI_BUILD_PROXY_CA }}
          context: ${{ inputs.context }}
          file: ${{ inputs.dockerfile }}
          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
@@ -286,3 +351,18 @@ jobs:
      - name: job summary
        run: |
          echo "Built image: ${{ steps.meta.outputs.labels }}" >> $GITHUB_STEP_SUMMARY
+
+      - name: Stop build network proxy
+        if: ${{ always() && env.LOCALAI_BUILD_PROXY_OUTPUT != '' }}
+        env:
+          LOCALAI_BUILD_JOB_STATUS: ${{ job.status }}
+        run: .github/scripts/stop-build-proxy.sh
+
+      - name: Upload build network inventory
+        if: ${{ always() && env.LOCALAI_BUILD_PROXY_OUTPUT != '' }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: build-network-${{ inputs.backend }}-${{ inputs.tag-suffix }}-${{ inputs.platform-tag || 'single' }}
+          path: ${{ env.LOCALAI_BUILD_PROXY_OUTPUT }}
+          if-no-files-found: warn
+          retention-days: 14
@@ -0,0 +1,38 @@
+---
+name: external compatibility probes
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '23 4 * * 1'
+
+permissions:
+  contents: read
+
+jobs:
+  external-probe-huggingface-xet:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.x'
+          cache: false
+      - name: Probe Hugging Face Xet compatibility
+        run: LOCALAI_HF_XET_SMOKE=1 go test ./pkg/huggingface-api -ginkgo.focus='pinned public Xet fixture' -count=1
+
+  external-probe-sigstore:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.x'
+          cache: false
+      - name: Probe public Sigstore compatibility
+        env:
+          LOCALAI_COSIGN_LIVE: '1'
+          LOCALAI_COSIGN_LIVE_IMAGE: ${{ vars.LOCALAI_COSIGN_LIVE_IMAGE }}
+          LOCALAI_COSIGN_LIVE_ISSUER: ${{ vars.LOCALAI_COSIGN_LIVE_ISSUER }}
+          LOCALAI_COSIGN_LIVE_IDENTITY_REGEX: ${{ vars.LOCALAI_COSIGN_LIVE_IDENTITY_REGEX }}
+        run: go test ./pkg/oci/cosignverify -ginkgo.focus='VerifyImage' -count=1
@@ -129,9 +129,27 @@ jobs:
        with:
          platforms: all

+      - name: Set up Go for build proxy
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+
+      - name: Start build network proxy
+        run: .github/scripts/start-build-proxy.sh
+
      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@master
+        with:
+          driver-opts: |
+            network=host
+            env.http_proxy=${{ env.LOCALAI_BUILD_PROXY }}
+            env.https_proxy=${{ env.LOCALAI_BUILD_PROXY }}
+
+      - name: Trust build proxy CA in BuildKit
+        env:
+          BUILDER_NAME: ${{ steps.buildx.outputs.name }}
+        run: .github/scripts/inject-build-proxy-ca.sh

      - name: Login to DockerHub
        if: github.event_name != 'pull_request'
@@ -155,6 +173,9 @@ jobs:
        with:
          builder: ${{ steps.buildx.outputs.name }}
          build-args: |
+            HTTP_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            HTTPS_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            NO_PROXY=localhost,127.0.0.1
            BUILD_TYPE=${{ inputs.build-type }}
            CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
            CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
@@ -165,6 +186,8 @@ jobs:
            UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
            APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
            APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
+          secret-files: |
+            build_proxy_ca=${{ env.LOCALAI_BUILD_PROXY_CA }}
          context: .
          file: ./Dockerfile
          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
@@ -218,6 +241,9 @@ jobs:
        with:
          builder: ${{ steps.buildx.outputs.name }}
          build-args: |
+            HTTP_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            HTTPS_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            NO_PROXY=localhost,127.0.0.1
            BUILD_TYPE=${{ inputs.build-type }}
            CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
            CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
@@ -228,6 +254,8 @@ jobs:
            UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
            APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
            APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
+          secret-files: |
+            build_proxy_ca=${{ env.LOCALAI_BUILD_PROXY_CA }}
          context: .
          file: ./Dockerfile
          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
@@ -239,3 +267,18 @@ jobs:
      - name: job summary
        run: |
          echo "Built image: ${{ steps.meta.outputs.labels }}" >> $GITHUB_STEP_SUMMARY
+
+      - name: Stop build network proxy
+        if: ${{ always() && env.LOCALAI_BUILD_PROXY_OUTPUT != '' }}
+        env:
+          LOCALAI_BUILD_JOB_STATUS: ${{ job.status }}
+        run: .github/scripts/stop-build-proxy.sh
+
+      - name: Upload build network inventory
+        if: ${{ always() && env.LOCALAI_BUILD_PROXY_OUTPUT != '' }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: build-network-localai-${{ inputs.build-type }}-${{ inputs.cuda-major-version }}-${{ inputs.cuda-minor-version }}-${{ inputs.platform-tag || 'single' }}
+          path: ${{ env.LOCALAI_BUILD_PROXY_OUTPUT }}
+          if-no-files-found: warn
+          retention-days: 14
@@ -0,0 +1,131 @@
+---
+name: 'sync jetson wheels mirror'
+
+# Mirrors the CUDA aarch64 wheels our l4t backends need from
+# pypi.jetson-ai-lab.io into scratch OCI images on ghcr
+# (ghcr.io/mudler/localai/jetson-wheels:<tag>, one tag per JetPack index).
+# backend_build.yml hands the matching tag to Dockerfile.python, which
+# bind-mounts it and serves it as a local package index during pip install
+# (see installRequirements in backend/python/common/libbackend.sh), so the
+# upstream index's recurring multi-hour 502 outages can no longer fail l4t
+# builds.
+#
+# The package subset lives in .github/jetson-wheels.json. A package that a
+# build needs from the jetson index but that is missing from that list will
+# resolve from PyPI instead — for compiled CUDA packages that silently means
+# a CPU build, so extend the list when adding an l4t backend with new
+# compiled deps.
+#
+# When upstream is unreachable the sync keeps the previously mirrored wheels
+# and exits green — the mirror serves last-known-good through outages. It
+# only fails when upstream is down and the tag has never been published
+# (bootstrap during an outage: nothing to serve yet).
+#
+# Triggers:
+#   - schedule (Saturdays 03:00 UTC) — refreshes ahead of base-images.yml
+#     (Saturdays 05:00 UTC) and the backend.yml weekly cron (Sundays), whose
+#     DEPS_REFRESH cache-bust re-resolves the python deps.
+#   - workflow_dispatch — manual one-off sync; also the bootstrap run:
+#       gh workflow run jetson-wheels.yml --ref master
+#   - push to master touching the config, the sync script, or this workflow.
+
+on:
+  schedule:
+    - cron: '0 3 * * 6'
+  workflow_dispatch:
+  push:
+    branches: [master]
+    paths:
+      - '.github/jetson-wheels.json'
+      - 'scripts/jetson-wheels-sync.py'
+      - '.github/workflows/jetson-wheels.yml'
+
+permissions:
+  contents: read
+  packages: write
+
+concurrency:
+  group: jetson-wheels-${{ github.repository }}
+  cancel-in-progress: false
+
+jobs:
+  sync:
+    if: github.repository == 'mudler/LocalAI'
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - index: 'jp6/cu129'
+            tag: 'jp6-cu129'
+          - index: 'jp7/cu130'
+            tag: 'jp7-cu130'
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v7
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@master
+
+      - name: Login to ghcr.io
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ github.token }}
+
+      - name: Compute image name
+        id: image
+        run: |
+          repo="ghcr.io/$(echo "${GITHUB_REPOSITORY}" | tr '[:upper:]' '[:lower:]')/jetson-wheels"
+          echo "ref=${repo}:${{ matrix.tag }}" >> "$GITHUB_OUTPUT"
+
+      # Seed the working dir with the current mirror contents so the sync is
+      # incremental and an upstream outage keeps last-known-good wheels.
+      - name: Pull current mirror contents
+        run: |
+          mkdir -p wheels
+          # The image is declared linux/arm64 (its only consumers are arm64
+          # l4t builds); pulling on this amd64 runner needs the explicit
+          # platform. The content is just wheel files — never executed here.
+          if docker pull --platform linux/arm64 "${{ steps.image.outputs.ref }}"; then
+            # scratch images have no command; docker create still needs one,
+            # but the container is never started so any path works.
+            cid="$(docker create "${{ steps.image.outputs.ref }}" /noop)"
+            docker export "${cid}" | tar -x -C wheels
+            docker rm "${cid}"
+            find wheels -name '*.whl' | sed 's/^/  existing: /'
+          else
+            echo "no existing mirror image (bootstrap run)"
+          fi
+
+      - name: Sync from upstream
+        id: sync
+        run: |
+          python3 scripts/jetson-wheels-sync.py \
+            --config .github/jetson-wheels.json \
+            --index '${{ matrix.index }}' \
+            --dest wheels \
+            --changed-file /tmp/jetson-wheels-changed
+          if [ -f /tmp/jetson-wheels-changed ]; then
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Push mirror image
+        if: steps.sync.outputs.changed == 'true'
+        run: |
+          cat > Dockerfile.jetson-wheels <<'EOF'
+          FROM scratch
+          COPY wheels/ /
+          EOF
+          # linux/arm64 because the consumers (l4t builds in
+          # backend_build.yml) build for arm64 and BuildKit refuses a
+          # platform-mismatched FROM; COPY-only, so no emulation is needed.
+          # provenance=false keeps the pushed ref a plain single manifest
+          # instead of an OCI index wrapping an attestation.
+          docker buildx build --push \
+            --platform linux/arm64 \
+            --provenance=false \
+            -f Dockerfile.jetson-wheels \
+            -t "${{ steps.image.outputs.ref }}" \
+            .
@@ -0,0 +1,76 @@
+---
+name: refresh offline test resources
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '17 3 * * 1'
+
+permissions:
+  contents: read
+  issues: write
+  packages: write
+
+jobs:
+  refresh:
+    strategy:
+      fail-fast: false
+      matrix:
+        resource-set: [default, distributed-e2e, aio]
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.x'
+          cache: true
+      - uses: oras-project/setup-oras@v1
+      - name: Verify upstream resources and build compressed cache
+        id: refresh
+        continue-on-error: true
+        env:
+          LOCALAI_TEST_RESOURCES_ONLINE: '1'
+        run: |
+          set -o pipefail
+          make update-offline-test-cache TEST_RESOURCE_SET=${{ matrix.resource-set }} 2>&1 | tee resource-refresh.log
+      - name: Upload investigation evidence
+        if: steps.refresh.outcome == 'failure'
+        uses: actions/upload-artifact@v4
+        with:
+          name: test-resource-investigation-${{ matrix.resource-set }}-${{ github.run_id }}
+          path: resource-refresh.log
+      - name: Open or update investigation issue
+        if: steps.refresh.outcome == 'failure'
+        env:
+          GH_TOKEN: ${{ secrets.LOCALAI_BOT_TOKEN || github.token }}
+          RESOURCE_SET: ${{ matrix.resource-set }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          title="test resource integrity investigation: ${RESOURCE_SET}"
+          body=$(printf '%s\n\n%s\n' \
+            "The scheduled offline-resource refresh failed for \`${RESOURCE_SET}\`." \
+            "Do not update the manifest digest blindly. Download the evidence artifact from ${RUN_URL}, compare upstream checksums/signatures and release notes, inspect redirects, and search the [GitHub Advisory Database](https://github.com/advisories) and [OSV](https://osv.dev). Retry from a declared mirror to distinguish source drift from corruption.")
+          existing=$(gh issue list --state open --search "${title} in:title" --json number --jq '.[0].number // empty')
+          if [ -n "$existing" ]; then
+            gh issue comment "$existing" --body "$body"
+          else
+            gh issue create --title "$title" --body "$body"
+          fi
+      - name: Log in to GHCR
+        if: steps.refresh.outcome == 'success'
+        run: echo "${{ github.token }}" | oras login ghcr.io -u "${{ github.actor }}" --password-stdin
+      - name: Publish compressed cache as an OCI artifact
+        if: steps.refresh.outcome == 'success'
+        env:
+          RESOURCE_SET: ${{ matrix.resource-set }}
+        run: |
+          repository=$(printf '%s' "ghcr.io/${GITHUB_REPOSITORY}/localai-test-resources" | tr '[:upper:]' '[:lower:]')
+          digest=$(jq -r --arg set "$RESOURCE_SET" '.bundles[$set] | sub("sha256:"; "sha256-")' test-resources/manifests/lock.json)
+          oras push \
+            --artifact-type application/vnd.localai.test-resources.v1 \
+            "${repository}:${RESOURCE_SET},${RESOURCE_SET}-${digest}" \
+            ".cache/test-resources/bundles/${RESOURCE_SET}.tar.zst:application/vnd.localai.test-resources.bundle.v1+zstd" \
+            "test-resources/manifests/${RESOURCE_SET}.json:application/vnd.localai.test-resources.manifest.v1+json"
+      - name: Fail after preserving evidence
+        if: steps.refresh.outcome == 'failure'
+        run: exit 1
@@ -39,6 +39,8 @@ jobs:
      # You can test your matrix by printing the current Go version
      - name: Display Go version
        run: go version
+      - name: Download Go modules
+        run: go mod download
      - name: Proto Dependencies
        run: |
          # Install protoc
@@ -58,13 +60,30 @@ jobs:
          node-version: '22'
      - name: Build React UI
        run: make react-ui
+      - name: Record and pack declared test resources
+        run: LOCALAI_TEST_RESOURCES_ONLINE=1 make update-offline-test-cache TEST_RESOURCE_SET=default
+      - name: Transfer local test-resource bundle
+        uses: actions/upload-artifact@v4
+        with:
+          name: test-resources-default-${{ github.run_id }}
+          include-hidden-files: true
+          path: |
+            .cache/test-resources/bundles/default.tar.zst
+            test-resources/manifests/lock.json
+      - name: Clear recorded resource cache
+        run: rm -rf .cache/test-resources
+      - name: Restore local test-resource bundle
+        uses: actions/download-artifact@v4
+        with:
+          name: test-resources-default-${{ github.run_id }}
+          path: .
      # Runs the core suite with coverage and fails if total coverage dropped
      # below the committed baseline (coverage-baseline.txt). The gate is
      # strict — any decrease fails. Raise the baseline with
      # `make test-coverage-baseline` and commit it when coverage rises.
      - name: Test (with coverage gate)
        run: |
-          PATH="$PATH:/root/go/bin" make --jobs 5 --output-sync=target test-coverage-check
+          LOCALAI_TEST_KERNEL_ENFORCE=1 PATH="$PATH:/root/go/bin" make --jobs 5 --output-sync=target test-coverage-check
      # tests/integration is outside the coverage roots because its store specs
      # need a live backend. test-stores builds and installs local-store before
      # running the complete suite, so new local-store specs are collected
@@ -106,6 +125,8 @@ jobs:
      # You can test your matrix by printing the current Go version
      - name: Display Go version
        run: go version
+      - name: Download Go modules
+        run: go mod download
      - name: Dependencies
        run: |
          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm opus ffmpeg
@@ -124,7 +145,7 @@ jobs:
          # Used to run the newer GNUMake version from brew that supports --output-sync
          export PATH="/opt/homebrew/opt/make/libexec/gnubin:$PATH"
          PATH="$PATH:$HOME/go/bin" make protogen-go
-          PATH="$PATH:$HOME/go/bin" BUILD_TYPE="GITHUB_CI_HAS_BROKEN_METAL" CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make --jobs 4 --output-sync=target test
+          PATH="$PATH:$HOME/go/bin" BUILD_TYPE="GITHUB_CI_HAS_BROKEN_METAL" CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make --jobs 4 --output-sync=target TEST_RESOURCE_SET=default-darwin test
      - name: Setup tmate session if tests fail
        if: ${{ failure() }}
        uses: mxschmitt/action-tmate@v3.23
@@ -76,7 +76,9 @@ jobs:
          PATH="$PATH:$HOME/go/bin" make protogen-go
      - name: Test
        run: |
-            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
+            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e
+            LOCALAI_TEST_RESOURCES_ONLINE=1 PATH="$PATH:$HOME/go/bin" make update-offline-test-cache TEST_RESOURCE_SET=aio
+            LOCALAI_BACKEND_DIR="$GITHUB_WORKSPACE/backends" LOCALAI_MODELS_DIR="$GITHUB_WORKSPACE/tests/e2e-aio/models" LOCALAI_IMAGE_TAG=tests LOCALAI_IMAGE=local-ai PATH="$PATH:$HOME/go/bin" make run-e2e-aio
      - name: Setup tmate session if tests fail
        if: ${{ failure() }}
        uses: mxschmitt/action-tmate@v3.23
@@ -60,9 +60,12 @@ jobs:
          node-version: '22'
      - name: Build React UI
        run: make react-ui
+      - name: Record declared distributed test resources
+        run: LOCALAI_TEST_RESOURCES_ONLINE=1 make update-offline-test-cache TEST_RESOURCE_SET=distributed-e2e
      - name: Test Backend E2E
        run: |
          PATH="$PATH:$HOME/go/bin" make build-mock-backend test-e2e
+          PATH="$PATH:$HOME/go/bin" make test-e2e-distributed
      - name: Setup tmate session if tests fail
        if: ${{ failure() }}
        uses: mxschmitt/action-tmate@v3.23