chore(tests): Avoid network, sleep and more during tests (#11050)

* test: make coverage failures observable

Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: parallelize coverage without remote fixtures

Assisted-by: Codex:gpt-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: add offline resource infrastructure

Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: enforce offline resource replay

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: harden offline resource refresh

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: expose slow coverage waits

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: eliminate avoidable wall-clock waits

Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: remove repeated fixture startup waits

Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: fix offline resource CI portability

Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: cache Go modules before offline tests

Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: drop the static network lint in favour of real isolation

The offline test suite already prevents tests from reaching the network
twice over: run-test-linux-offline.sh puts the test process in a cgroup
and REJECTs egress outside the private ranges, and HardenedTransport
installs testnetwork.LocalGuard to refuse dials that resolve to a public
address. Both fail the test with a precise error at the moment of the
dial.

test-network-lint.sh added neither. Its diff stage defaulted to a HEAD
base, so on a clean checkout it compared the tree against itself and
inspected nothing; the branch's own commits were never examined. It only
produced output when an earlier job step dirtied the tree, and then it
matched a bare https?:// against whatever changed. make react-ui runs
npm install rather than npm ci, so CI rewrote
core/http/react-ui/package-lock.json and the lint reported an npm
registry URL as forbidden test network access:

  +      "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz",

Its fingerprint stage was self-defeating in a quieter way: hashing the
whole tree's network-mechanism inventory meant every rebase onto a master
that touched any _test.go needed a manual baseline bump, so the check
mostly caught its own staleness.

Remove the script, its make target and the two prerequisite edges, along
with the test-network: fixture markers that existed only to suppress it.
The isolation itself is untouched.

Assisted-by: Claude:claude-opus-5 [go vet]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: keep hidden files in the offline test bundle artifact

Cherry-picked from 15a37b0ac on the remote branch. The offline bundle lives
under .cache/, which actions/upload-artifact skips by default, so the Linux
job packed an artifact missing the very file the next step restores.

The other half of 15a37b0ac moved test-network-lint out of the `test` and
`test-coverage` prerequisite lists into a recipe line, so parallel make could
not fingerprint the tree while generated fixtures were still changing. That
is dropped: the preceding commit removes the lint entirely, and the race it
worked around is one more reason a whole-tree fingerprint was the wrong
mechanism.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* refactor: share bounded exponential backoff

Use overflow-safe saturating arithmetic for retry delays across model import polling, downloads, registration, node operations, and model loading. Keep model import status checks responsive initially while capping their interval at 500ms.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: mirror Jetson Python wheels

Keep the CUDA aarch64 wheel subset in GHCR and serve it as a local PEP 503 index during L4T backend builds, preserving last-known-good packages through upstream outages.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* docs(agents): index the Jetson wheels mirror

Mention the GHCR-hosted L4T wheel mirror in the CI caching guide summary so maintainers can find its outage and cache documentation.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: add defensive build network proxy

Record build destinations and byte counts, retry observable idempotent HTTP downloads, and isolate explorer database tests that race under coverage.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(kokoros): implement updated backend trait

Return unimplemented for image upscaling, matching the backend's other unsupported modalities after the protobuf API update.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): clear recovered proxy errors

Do not mark a request failed when a later safe retry succeeds.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: require HTTPS build interception

Inject a short-lived proxy CA into BuildKit and Dockerfile RUN steps, reject plain HTTP and opaque tunnels, and retain method/status/byte telemetry for verified HTTPS traffic.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): preserve system trust in unproxied builds

Mount the generated interception CA at a dedicated secret path and add it to the trust bundle only in proxy-aware dependency stages. This prevents optional secret mounts from masking the system CA bundle in ordinary backend test builds.

Install the requested Go toolchain before starting the proxy and satisfy cleanup error checks found by CI lint.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): persist build proxy trust

Install the generated proxy CA through the system-managed local certificate directory so ca-certificates upgrades retain it. Avoid turning canceled matrix jobs into proxy cleanup failures.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): trust proxy in nested build scripts

Install the build proxy CA before nested source fetches, route the DS4 package setup through the HTTPS mirror helper, and avoid repeated OCI setup in gallery behavior tests.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): use HTTPS apt sources for Bonsai

Rewrite ARM64 package sources before installing GCC and check gallery fixture cleanup errors so the optimized tests satisfy errcheck.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(privacy-filter): trust build proxy CA

Install the mounted build proxy certificate before privacy-filter's make target fetches its HTTPS sources, for both source and prebuilt builder paths.\n\nAssisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: fail on hidden offline egress

Count cgroup-scoped firewall rejects and fail the offline test harness with bounded aggregate diagnostics. Inject the gen-audio GGUF probe so fixture-backed importer tests do not attempt real network access.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): preserve system CA trust

Build a combined runner certificate bundle instead of replacing public roots with the generated proxy CA. Centralize additive container installation in the shared proxy CA helper.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>

.github/workflows/backend_build.yml

@@ -153,9 +153,27 @@ jobs:
         with:
           platforms: all
 
+      - name: Set up Go for build proxy
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+
+      - name: Start build network proxy
+        run: .github/scripts/start-build-proxy.sh
+
       - name: Set up Docker Buildx
         id: buildx
         uses: docker/setup-buildx-action@master
+        with:
+          driver-opts: |
+            network=host
+            env.http_proxy=${{ env.LOCALAI_BUILD_PROXY }}
+            env.https_proxy=${{ env.LOCALAI_BUILD_PROXY }}
+
+      - name: Trust build proxy CA in BuildKit
+        env:
+          BUILDER_NAME: ${{ steps.buildx.outputs.name }}
+        run: .github/scripts/inject-build-proxy-ca.sh
 
       - name: Login to DockerHub
         if: github.event_name != 'pull_request'
@@ -181,6 +199,41 @@ jobs:
         id: deps_refresh
         run: echo "key=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
 
+      - name: Login to ghcr.io (jetson wheels mirror)
+        if: inputs.build-type == 'l4t'
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ github.token }}
+
+      # l4t builds pull their CUDA aarch64 torch wheels from
+      # pypi.jetson-ai-lab.io, which has a history of multi-hour 502 outages
+      # that fail every l4t job. jetson-wheels.yml mirrors those wheels into
+      # ghcr weekly; here we hand the mirror image to Dockerfile.python,
+      # which serves it as a local package index during pip install (see
+      # installRequirements in backend/python/common/libbackend.sh). Falls
+      # back to scratch — i.e. building straight against the upstream index —
+      # when the mirror tag doesn't exist yet, so the mirror can bootstrap
+      # without a chicken-and-egg failure.
+      - name: Resolve jetson wheels mirror image
+        id: jetson_wheels
+        if: inputs.build-type == 'l4t'
+        run: |
+          repo="ghcr.io/$(echo "${{ github.repository }}" | tr '[:upper:]' '[:lower:]')/jetson-wheels"
+          case "${{ inputs.cuda-major-version }}" in
+            12) tag="jp6-cu129" ;;
+            13) tag="jp7-cu130" ;;
+            *) tag="" ;;
+          esac
+          img=""
+          if [ -n "$tag" ] && docker buildx imagetools inspect "$repo:$tag" >/dev/null 2>&1; then
+            img="$repo:$tag"
+          else
+            echo "jetson wheels image $repo:$tag not found; building against the upstream index"
+          fi
+          echo "image=$img" >> "$GITHUB_OUTPUT"
+
       - name: Build and push by digest
         id: build
         uses: docker/build-push-action@v7
@@ -188,6 +241,9 @@ jobs:
         with:
           builder: ${{ steps.buildx.outputs.name }}
           build-args: |
+            HTTP_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            HTTPS_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            NO_PROXY=localhost,127.0.0.1
             BUILD_TYPE=${{ inputs.build-type }}
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
@@ -201,6 +257,9 @@ jobs:
             DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
             BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
             BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
+            JETSON_WHEELS_IMAGE=${{ steps.jetson_wheels.outputs.image || 'scratch' }}
+          secret-files: |
+            build_proxy_ca=${{ env.LOCALAI_BUILD_PROXY_CA }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
           cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
@@ -260,6 +319,9 @@ jobs:
         with:
           builder: ${{ steps.buildx.outputs.name }}
           build-args: |
+            HTTP_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            HTTPS_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            NO_PROXY=localhost,127.0.0.1
             BUILD_TYPE=${{ inputs.build-type }}
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
@@ -273,6 +335,9 @@ jobs:
             DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
             BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
             BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
+            JETSON_WHEELS_IMAGE=${{ steps.jetson_wheels.outputs.image || 'scratch' }}
+          secret-files: |
+            build_proxy_ca=${{ env.LOCALAI_BUILD_PROXY_CA }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
           cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
@@ -286,3 +351,18 @@ jobs:
       - name: job summary
         run: |
           echo "Built image: ${{ steps.meta.outputs.labels }}" >> $GITHUB_STEP_SUMMARY
+
+      - name: Stop build network proxy
+        if: ${{ always() && env.LOCALAI_BUILD_PROXY_OUTPUT != '' }}
+        env:
+          LOCALAI_BUILD_JOB_STATUS: ${{ job.status }}
+        run: .github/scripts/stop-build-proxy.sh
+
+      - name: Upload build network inventory
+        if: ${{ always() && env.LOCALAI_BUILD_PROXY_OUTPUT != '' }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: build-network-${{ inputs.backend }}-${{ inputs.tag-suffix }}-${{ inputs.platform-tag || 'single' }}
+          path: ${{ env.LOCALAI_BUILD_PROXY_OUTPUT }}
+          if-no-files-found: warn
+          retention-days: 14

.github/workflows/external-probes.yml

@@ -0,0 +1,38 @@
+---
+name: external compatibility probes
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '23 4 * * 1'
+
+permissions:
+  contents: read
+
+jobs:
+  external-probe-huggingface-xet:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.x'
+          cache: false
+      - name: Probe Hugging Face Xet compatibility
+        run: LOCALAI_HF_XET_SMOKE=1 go test ./pkg/huggingface-api -ginkgo.focus='pinned public Xet fixture' -count=1
+
+  external-probe-sigstore:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.x'
+          cache: false
+      - name: Probe public Sigstore compatibility
+        env:
+          LOCALAI_COSIGN_LIVE: '1'
+          LOCALAI_COSIGN_LIVE_IMAGE: ${{ vars.LOCALAI_COSIGN_LIVE_IMAGE }}
+          LOCALAI_COSIGN_LIVE_ISSUER: ${{ vars.LOCALAI_COSIGN_LIVE_ISSUER }}
+          LOCALAI_COSIGN_LIVE_IDENTITY_REGEX: ${{ vars.LOCALAI_COSIGN_LIVE_IDENTITY_REGEX }}
+        run: go test ./pkg/oci/cosignverify -ginkgo.focus='VerifyImage' -count=1

.github/workflows/image_build.yml

@@ -129,9 +129,27 @@ jobs:
         with:
           platforms: all
 
+      - name: Set up Go for build proxy
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+
+      - name: Start build network proxy
+        run: .github/scripts/start-build-proxy.sh
+
       - name: Set up Docker Buildx
         id: buildx
         uses: docker/setup-buildx-action@master
+        with:
+          driver-opts: |
+            network=host
+            env.http_proxy=${{ env.LOCALAI_BUILD_PROXY }}
+            env.https_proxy=${{ env.LOCALAI_BUILD_PROXY }}
+
+      - name: Trust build proxy CA in BuildKit
+        env:
+          BUILDER_NAME: ${{ steps.buildx.outputs.name }}
+        run: .github/scripts/inject-build-proxy-ca.sh
 
       - name: Login to DockerHub
         if: github.event_name != 'pull_request'
@@ -155,6 +173,9 @@ jobs:
         with:
           builder: ${{ steps.buildx.outputs.name }}
           build-args: |
+            HTTP_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            HTTPS_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            NO_PROXY=localhost,127.0.0.1
             BUILD_TYPE=${{ inputs.build-type }}
             CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
             CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
@@ -165,6 +186,8 @@ jobs:
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
             APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
             APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
+          secret-files: |
+            build_proxy_ca=${{ env.LOCALAI_BUILD_PROXY_CA }}
           context: .
           file: ./Dockerfile
           cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
@@ -218,6 +241,9 @@ jobs:
         with:
           builder: ${{ steps.buildx.outputs.name }}
           build-args: |
+            HTTP_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            HTTPS_PROXY=${{ env.LOCALAI_BUILD_PROXY }}
+            NO_PROXY=localhost,127.0.0.1
             BUILD_TYPE=${{ inputs.build-type }}
             CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
             CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
@@ -228,6 +254,8 @@ jobs:
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
             APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
             APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
+          secret-files: |
+            build_proxy_ca=${{ env.LOCALAI_BUILD_PROXY_CA }}
           context: .
           file: ./Dockerfile
           cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
@@ -239,3 +267,18 @@ jobs:
       - name: job summary
         run: |
           echo "Built image: ${{ steps.meta.outputs.labels }}" >> $GITHUB_STEP_SUMMARY
+
+      - name: Stop build network proxy
+        if: ${{ always() && env.LOCALAI_BUILD_PROXY_OUTPUT != '' }}
+        env:
+          LOCALAI_BUILD_JOB_STATUS: ${{ job.status }}
+        run: .github/scripts/stop-build-proxy.sh
+
+      - name: Upload build network inventory
+        if: ${{ always() && env.LOCALAI_BUILD_PROXY_OUTPUT != '' }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: build-network-localai-${{ inputs.build-type }}-${{ inputs.cuda-major-version }}-${{ inputs.cuda-minor-version }}-${{ inputs.platform-tag || 'single' }}
+          path: ${{ env.LOCALAI_BUILD_PROXY_OUTPUT }}
+          if-no-files-found: warn
+          retention-days: 14

.github/workflows/jetson-wheels.yml

@@ -0,0 +1,131 @@
+---
+name: 'sync jetson wheels mirror'
+
+# Mirrors the CUDA aarch64 wheels our l4t backends need from
+# pypi.jetson-ai-lab.io into scratch OCI images on ghcr
+# (ghcr.io/mudler/localai/jetson-wheels:<tag>, one tag per JetPack index).
+# backend_build.yml hands the matching tag to Dockerfile.python, which
+# bind-mounts it and serves it as a local package index during pip install
+# (see installRequirements in backend/python/common/libbackend.sh), so the
+# upstream index's recurring multi-hour 502 outages can no longer fail l4t
+# builds.
+#
+# The package subset lives in .github/jetson-wheels.json. A package that a
+# build needs from the jetson index but that is missing from that list will
+# resolve from PyPI instead — for compiled CUDA packages that silently means
+# a CPU build, so extend the list when adding an l4t backend with new
+# compiled deps.
+#
+# When upstream is unreachable the sync keeps the previously mirrored wheels
+# and exits green — the mirror serves last-known-good through outages. It
+# only fails when upstream is down and the tag has never been published
+# (bootstrap during an outage: nothing to serve yet).
+#
+# Triggers:
+#   - schedule (Saturdays 03:00 UTC) — refreshes ahead of base-images.yml
+#     (Saturdays 05:00 UTC) and the backend.yml weekly cron (Sundays), whose
+#     DEPS_REFRESH cache-bust re-resolves the python deps.
+#   - workflow_dispatch — manual one-off sync; also the bootstrap run:
+#       gh workflow run jetson-wheels.yml --ref master
+#   - push to master touching the config, the sync script, or this workflow.
+
+on:
+  schedule:
+    - cron: '0 3 * * 6'
+  workflow_dispatch:
+  push:
+    branches: [master]
+    paths:
+      - '.github/jetson-wheels.json'
+      - 'scripts/jetson-wheels-sync.py'
+      - '.github/workflows/jetson-wheels.yml'
+
+permissions:
+  contents: read
+  packages: write
+
+concurrency:
+  group: jetson-wheels-${{ github.repository }}
+  cancel-in-progress: false
+
+jobs:
+  sync:
+    if: github.repository == 'mudler/LocalAI'
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - index: 'jp6/cu129'
+            tag: 'jp6-cu129'
+          - index: 'jp7/cu130'
+            tag: 'jp7-cu130'
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v7
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@master
+
+      - name: Login to ghcr.io
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ github.token }}
+
+      - name: Compute image name
+        id: image
+        run: |
+          repo="ghcr.io/$(echo "${GITHUB_REPOSITORY}" | tr '[:upper:]' '[:lower:]')/jetson-wheels"
+          echo "ref=${repo}:${{ matrix.tag }}" >> "$GITHUB_OUTPUT"
+
+      # Seed the working dir with the current mirror contents so the sync is
+      # incremental and an upstream outage keeps last-known-good wheels.
+      - name: Pull current mirror contents
+        run: |
+          mkdir -p wheels
+          # The image is declared linux/arm64 (its only consumers are arm64
+          # l4t builds); pulling on this amd64 runner needs the explicit
+          # platform. The content is just wheel files — never executed here.
+          if docker pull --platform linux/arm64 "${{ steps.image.outputs.ref }}"; then
+            # scratch images have no command; docker create still needs one,
+            # but the container is never started so any path works.
+            cid="$(docker create "${{ steps.image.outputs.ref }}" /noop)"
+            docker export "${cid}" | tar -x -C wheels
+            docker rm "${cid}"
+            find wheels -name '*.whl' | sed 's/^/  existing: /'
+          else
+            echo "no existing mirror image (bootstrap run)"
+          fi
+
+      - name: Sync from upstream
+        id: sync
+        run: |
+          python3 scripts/jetson-wheels-sync.py \
+            --config .github/jetson-wheels.json \
+            --index '${{ matrix.index }}' \
+            --dest wheels \
+            --changed-file /tmp/jetson-wheels-changed
+          if [ -f /tmp/jetson-wheels-changed ]; then
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Push mirror image
+        if: steps.sync.outputs.changed == 'true'
+        run: |
+          cat > Dockerfile.jetson-wheels <<'EOF'
+          FROM scratch
+          COPY wheels/ /
+          EOF
+          # linux/arm64 because the consumers (l4t builds in
+          # backend_build.yml) build for arm64 and BuildKit refuses a
+          # platform-mismatched FROM; COPY-only, so no emulation is needed.
+          # provenance=false keeps the pushed ref a plain single manifest
+          # instead of an OCI index wrapping an attestation.
+          docker buildx build --push \
+            --platform linux/arm64 \
+            --provenance=false \
+            -f Dockerfile.jetson-wheels \
+            -t "${{ steps.image.outputs.ref }}" \
+            .

.github/workflows/test-resource-refresh.yml

@@ -0,0 +1,76 @@
+---
+name: refresh offline test resources
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '17 3 * * 1'
+
+permissions:
+  contents: read
+  issues: write
+  packages: write
+
+jobs:
+  refresh:
+    strategy:
+      fail-fast: false
+      matrix:
+        resource-set: [default, distributed-e2e, aio]
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.x'
+          cache: true
+      - uses: oras-project/setup-oras@v1
+      - name: Verify upstream resources and build compressed cache
+        id: refresh
+        continue-on-error: true
+        env:
+          LOCALAI_TEST_RESOURCES_ONLINE: '1'
+        run: |
+          set -o pipefail
+          make update-offline-test-cache TEST_RESOURCE_SET=${{ matrix.resource-set }} 2>&1 | tee resource-refresh.log
+      - name: Upload investigation evidence
+        if: steps.refresh.outcome == 'failure'
+        uses: actions/upload-artifact@v4
+        with:
+          name: test-resource-investigation-${{ matrix.resource-set }}-${{ github.run_id }}
+          path: resource-refresh.log
+      - name: Open or update investigation issue
+        if: steps.refresh.outcome == 'failure'
+        env:
+          GH_TOKEN: ${{ secrets.LOCALAI_BOT_TOKEN || github.token }}
+          RESOURCE_SET: ${{ matrix.resource-set }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          title="test resource integrity investigation: ${RESOURCE_SET}"
+          body=$(printf '%s\n\n%s\n' \
+            "The scheduled offline-resource refresh failed for \`${RESOURCE_SET}\`." \
+            "Do not update the manifest digest blindly. Download the evidence artifact from ${RUN_URL}, compare upstream checksums/signatures and release notes, inspect redirects, and search the [GitHub Advisory Database](https://github.com/advisories) and [OSV](https://osv.dev). Retry from a declared mirror to distinguish source drift from corruption.")
+          existing=$(gh issue list --state open --search "${title} in:title" --json number --jq '.[0].number // empty')
+          if [ -n "$existing" ]; then
+            gh issue comment "$existing" --body "$body"
+          else
+            gh issue create --title "$title" --body "$body"
+          fi
+      - name: Log in to GHCR
+        if: steps.refresh.outcome == 'success'
+        run: echo "${{ github.token }}" | oras login ghcr.io -u "${{ github.actor }}" --password-stdin
+      - name: Publish compressed cache as an OCI artifact
+        if: steps.refresh.outcome == 'success'
+        env:
+          RESOURCE_SET: ${{ matrix.resource-set }}
+        run: |
+          repository=$(printf '%s' "ghcr.io/${GITHUB_REPOSITORY}/localai-test-resources" | tr '[:upper:]' '[:lower:]')
+          digest=$(jq -r --arg set "$RESOURCE_SET" '.bundles[$set] | sub("sha256:"; "sha256-")' test-resources/manifests/lock.json)
+          oras push \
+            --artifact-type application/vnd.localai.test-resources.v1 \
+            "${repository}:${RESOURCE_SET},${RESOURCE_SET}-${digest}" \
+            ".cache/test-resources/bundles/${RESOURCE_SET}.tar.zst:application/vnd.localai.test-resources.bundle.v1+zstd" \
+            "test-resources/manifests/${RESOURCE_SET}.json:application/vnd.localai.test-resources.manifest.v1+json"
+      - name: Fail after preserving evidence
+        if: steps.refresh.outcome == 'failure'
+        run: exit 1

.github/workflows/test.yml

@@ -39,6 +39,8 @@ jobs:
       # You can test your matrix by printing the current Go version
       - name: Display Go version
         run: go version
+      - name: Download Go modules
+        run: go mod download
       - name: Proto Dependencies
         run: |
           # Install protoc
@@ -58,13 +60,30 @@ jobs:
           node-version: '22'
       - name: Build React UI
         run: make react-ui
+      - name: Record and pack declared test resources
+        run: LOCALAI_TEST_RESOURCES_ONLINE=1 make update-offline-test-cache TEST_RESOURCE_SET=default
+      - name: Transfer local test-resource bundle
+        uses: actions/upload-artifact@v4
+        with:
+          name: test-resources-default-${{ github.run_id }}
+          include-hidden-files: true
+          path: |
+            .cache/test-resources/bundles/default.tar.zst
+            test-resources/manifests/lock.json
+      - name: Clear recorded resource cache
+        run: rm -rf .cache/test-resources
+      - name: Restore local test-resource bundle
+        uses: actions/download-artifact@v4
+        with:
+          name: test-resources-default-${{ github.run_id }}
+          path: .
       # Runs the core suite with coverage and fails if total coverage dropped
       # below the committed baseline (coverage-baseline.txt). The gate is
       # strict — any decrease fails. Raise the baseline with
       # `make test-coverage-baseline` and commit it when coverage rises.
       - name: Test (with coverage gate)
         run: |
-          PATH="$PATH:/root/go/bin" make --jobs 5 --output-sync=target test-coverage-check
+          LOCALAI_TEST_KERNEL_ENFORCE=1 PATH="$PATH:/root/go/bin" make --jobs 5 --output-sync=target test-coverage-check
       # tests/integration is outside the coverage roots because its store specs
       # need a live backend. test-stores builds and installs local-store before
       # running the complete suite, so new local-store specs are collected
@@ -106,6 +125,8 @@ jobs:
       # You can test your matrix by printing the current Go version
       - name: Display Go version
         run: go version
+      - name: Download Go modules
+        run: go mod download
       - name: Dependencies
         run: |
           brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm opus ffmpeg
@@ -124,7 +145,7 @@ jobs:
           # Used to run the newer GNUMake version from brew that supports --output-sync
           export PATH="/opt/homebrew/opt/make/libexec/gnubin:$PATH"
           PATH="$PATH:$HOME/go/bin" make protogen-go
-          PATH="$PATH:$HOME/go/bin" BUILD_TYPE="GITHUB_CI_HAS_BROKEN_METAL" CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make --jobs 4 --output-sync=target test
+          PATH="$PATH:$HOME/go/bin" BUILD_TYPE="GITHUB_CI_HAS_BROKEN_METAL" CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make --jobs 4 --output-sync=target TEST_RESOURCE_SET=default-darwin test
       - name: Setup tmate session if tests fail
         if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23

.github/workflows/tests-aio.yml

@@ -76,7 +76,9 @@ jobs:
           PATH="$PATH:$HOME/go/bin" make protogen-go
       - name: Test
         run: |
-            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
+            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e
+            LOCALAI_TEST_RESOURCES_ONLINE=1 PATH="$PATH:$HOME/go/bin" make update-offline-test-cache TEST_RESOURCE_SET=aio
+            LOCALAI_BACKEND_DIR="$GITHUB_WORKSPACE/backends" LOCALAI_MODELS_DIR="$GITHUB_WORKSPACE/tests/e2e-aio/models" LOCALAI_IMAGE_TAG=tests LOCALAI_IMAGE=local-ai PATH="$PATH:$HOME/go/bin" make run-e2e-aio
       - name: Setup tmate session if tests fail
         if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23

.github/workflows/tests-e2e.yml

@@ -60,9 +60,12 @@ jobs:
           node-version: '22'
       - name: Build React UI
         run: make react-ui
+      - name: Record declared distributed test resources
+        run: LOCALAI_TEST_RESOURCES_ONLINE=1 make update-offline-test-cache TEST_RESOURCE_SET=distributed-e2e
       - name: Test Backend E2E
         run: |
           PATH="$PATH:$HOME/go/bin" make build-mock-backend test-e2e
+          PATH="$PATH:$HOME/go/bin" make test-e2e-distributed
       - name: Setup tmate session if tests fail
         if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23