chore(tests): Avoid network, sleep and more during tests (#11050)

* test: make coverage failures observable

Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: parallelize coverage without remote fixtures

Assisted-by: Codex:gpt-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: add offline resource infrastructure

Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: enforce offline resource replay

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: harden offline resource refresh

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: expose slow coverage waits

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: eliminate avoidable wall-clock waits

Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: remove repeated fixture startup waits

Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: fix offline resource CI portability

Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: cache Go modules before offline tests

Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: drop the static network lint in favour of real isolation

The offline test suite already prevents tests from reaching the network
twice over: run-test-linux-offline.sh puts the test process in a cgroup
and REJECTs egress outside the private ranges, and HardenedTransport
installs testnetwork.LocalGuard to refuse dials that resolve to a public
address. Both fail the test with a precise error at the moment of the
dial.

test-network-lint.sh added neither. Its diff stage defaulted to a HEAD
base, so on a clean checkout it compared the tree against itself and
inspected nothing; the branch's own commits were never examined. It only
produced output when an earlier job step dirtied the tree, and then it
matched a bare https?:// against whatever changed. make react-ui runs
npm install rather than npm ci, so CI rewrote
core/http/react-ui/package-lock.json and the lint reported an npm
registry URL as forbidden test network access:

  +      "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz",

Its fingerprint stage was self-defeating in a quieter way: hashing the
whole tree's network-mechanism inventory meant every rebase onto a master
that touched any _test.go needed a manual baseline bump, so the check
mostly caught its own staleness.

Remove the script, its make target and the two prerequisite edges, along
with the test-network: fixture markers that existed only to suppress it.
The isolation itself is untouched.

Assisted-by: Claude:claude-opus-5 [go vet]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: keep hidden files in the offline test bundle artifact

Cherry-picked from 15a37b0ac on the remote branch. The offline bundle lives
under .cache/, which actions/upload-artifact skips by default, so the Linux
job packed an artifact missing the very file the next step restores.

The other half of 15a37b0ac moved test-network-lint out of the `test` and
`test-coverage` prerequisite lists into a recipe line, so parallel make could
not fingerprint the tree while generated fixtures were still changing. That
is dropped: the preceding commit removes the lint entirely, and the race it
worked around is one more reason a whole-tree fingerprint was the wrong
mechanism.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* refactor: share bounded exponential backoff

Use overflow-safe saturating arithmetic for retry delays across model import polling, downloads, registration, node operations, and model loading. Keep model import status checks responsive initially while capping their interval at 500ms.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: mirror Jetson Python wheels

Keep the CUDA aarch64 wheel subset in GHCR and serve it as a local PEP 503 index during L4T backend builds, preserving last-known-good packages through upstream outages.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* docs(agents): index the Jetson wheels mirror

Mention the GHCR-hosted L4T wheel mirror in the CI caching guide summary so maintainers can find its outage and cache documentation.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: add defensive build network proxy

Record build destinations and byte counts, retry observable idempotent HTTP downloads, and isolate explorer database tests that race under coverage.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(kokoros): implement updated backend trait

Return unimplemented for image upscaling, matching the backend's other unsupported modalities after the protobuf API update.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): clear recovered proxy errors

Do not mark a request failed when a later safe retry succeeds.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: require HTTPS build interception

Inject a short-lived proxy CA into BuildKit and Dockerfile RUN steps, reject plain HTTP and opaque tunnels, and retain method/status/byte telemetry for verified HTTPS traffic.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): preserve system trust in unproxied builds

Mount the generated interception CA at a dedicated secret path and add it to the trust bundle only in proxy-aware dependency stages. This prevents optional secret mounts from masking the system CA bundle in ordinary backend test builds.

Install the requested Go toolchain before starting the proxy and satisfy cleanup error checks found by CI lint.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): persist build proxy trust

Install the generated proxy CA through the system-managed local certificate directory so ca-certificates upgrades retain it. Avoid turning canceled matrix jobs into proxy cleanup failures.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): trust proxy in nested build scripts

Install the build proxy CA before nested source fetches, route the DS4 package setup through the HTTPS mirror helper, and avoid repeated OCI setup in gallery behavior tests.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): use HTTPS apt sources for Bonsai

Rewrite ARM64 package sources before installing GCC and check gallery fixture cleanup errors so the optimized tests satisfy errcheck.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(privacy-filter): trust build proxy CA

Install the mounted build proxy certificate before privacy-filter's make target fetches its HTTPS sources, for both source and prebuilt builder paths.\n\nAssisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: fail on hidden offline egress

Count cgroup-scoped firewall rejects and fail the offline test harness with bounded aggregate diagnostics. Inject the gen-audio GGUF probe so fixture-backed importer tests do not attempt real network access.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): preserve system CA trust

Build a combined runner certificate bundle instead of replacing public roots with the generated proxy CA. Centralize additive container installation in the shared proxy CA helper.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>

scripts/jetson-wheels-sync.py

@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+"""Mirror the jetson-only wheel subset from pypi.jetson-ai-lab.io.
+
+Downloads the wheels for the packages listed in .github/jetson-wheels.json
+(one list per JetPack index path) into a local directory laid out exactly
+like the upstream index (<dest>/<index>/<project>/<wheel>). The jetson-wheels
+CI workflow publishes that directory as a scratch OCI image on ghcr, and
+backend builds serve it as a local package index during pip install — see
+pypi_mirror_server.py and installRequirements in
+backend/python/common/libbackend.sh for the consuming side and the
+motivation (recurring multi-hour upstream outages).
+
+The sync is additive-with-pruning against a *successfully fetched* project
+page: files no longer listed upstream are removed, but a package whose page
+cannot be fetched is left exactly as mirrored last time. When the whole
+upstream is unreachable the existing mirror is kept as-is (exit 0) so a CI
+run during an outage never destroys the last known-good wheels; it only
+fails (exit 2) when upstream is down AND there is nothing mirrored yet,
+i.e. the bootstrap run has nothing to publish.
+
+Standard library only. Usage:
+    python3 scripts/jetson-wheels-sync.py --config .github/jetson-wheels.json \
+        --index jp6/cu129 --dest wheels [--changed-file /tmp/changed]
+"""
+
+import argparse
+import hashlib
+import html.parser
+import json
+import os
+import re
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+
+DIST_SUFFIXES = (".whl", ".tar.gz", ".zip")
+TIMEOUT = 120
+
+
+def normalize(name):
+    """PEP 503 project-name normalization."""
+    return re.sub(r"[-_.]+", "-", name).lower()
+
+
+class _LinkParser(html.parser.HTMLParser):
+    def __init__(self):
+        super().__init__()
+        self.hrefs = []
+
+    def handle_starttag(self, tag, attrs):
+        if tag == "a":
+            for key, value in attrs:
+                if key == "href" and value:
+                    self.hrefs.append(value)
+
+
+def parse_links(page, base_url):
+    """Extract (filename, absolute_url, sha256|None) for each dist link."""
+    parser = _LinkParser()
+    parser.feed(page)
+    links = []
+    for href in parser.hrefs:
+        split = urllib.parse.urlsplit(href)
+        filename = os.path.basename(urllib.parse.unquote(split.path))
+        if not filename.endswith(DIST_SUFFIXES):
+            continue
+        sha256 = None
+        if split.fragment.startswith("sha256="):
+            sha256 = split.fragment[len("sha256="):]
+        url = urllib.parse.urljoin(base_url, split._replace(fragment="").geturl())
+        links.append((filename, url, sha256))
+    return links
+
+
+def _fetch(url):
+    request = urllib.request.Request(url, headers={"User-Agent": "localai-jetson-wheels-sync"})
+    return urllib.request.urlopen(request, timeout=TIMEOUT)
+
+
+def _download(url, dest_path, sha256):
+    digest = hashlib.sha256()
+    tmp = dest_path + ".tmp"
+    with _fetch(url) as resp, open(tmp, "wb") as out:
+        for chunk in iter(lambda: resp.read(1 << 20), b""):
+            digest.update(chunk)
+            out.write(chunk)
+    if sha256 and digest.hexdigest() != sha256:
+        os.unlink(tmp)
+        raise RuntimeError(f"sha256 mismatch for {url}: expected {sha256}, got {digest.hexdigest()}")
+    os.replace(tmp, dest_path)
+
+
+def _has_wheels(dest):
+    for _, _, files in os.walk(dest):
+        if any(f.endswith(DIST_SUFFIXES) for f in files):
+            return True
+    return False
+
+
+def sync_package(base_url, package, dest_dir):
+    """Returns (fetched_ok, changed)."""
+    page_url = urllib.parse.urljoin(base_url, normalize(package) + "/")
+    try:
+        with _fetch(page_url) as resp:
+            page = resp.read().decode("utf-8", "replace")
+    except urllib.error.HTTPError as err:
+        if err.code == 404:
+            # Upstream simply doesn't host this package for this index —
+            # normal (the config list is a superset across JetPack versions).
+            print(f"  {package}: not hosted upstream (404), skipping")
+            return True, False
+        print(f"  {package}: upstream error {err.code}, keeping mirrored files")
+        return False, False
+    except (urllib.error.URLError, TimeoutError, OSError) as err:
+        print(f"  {package}: upstream unreachable ({err}), keeping mirrored files")
+        return False, False
+
+    links = parse_links(page, page_url)
+    listed = {name for name, _, _ in links}
+    changed = False
+    os.makedirs(dest_dir, exist_ok=True)
+    for name, url, sha256 in links:
+        path = os.path.join(dest_dir, name)
+        if os.path.exists(path):
+            continue
+        print(f"  {package}: downloading {name}")
+        _download(url, path, sha256)
+        changed = True
+    # Prune only against a page we actually fetched: upstream removing a
+    # wheel is tracked, an outage never empties the mirror.
+    for existing in os.listdir(dest_dir):
+        if existing.endswith(DIST_SUFFIXES) and existing not in listed:
+            print(f"  {package}: pruning {existing} (no longer listed upstream)")
+            os.unlink(os.path.join(dest_dir, existing))
+            changed = True
+    return True, changed
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--config", required=True)
+    parser.add_argument("--index", required=True, help="index path, e.g. jp6/cu129")
+    parser.add_argument("--dest", required=True)
+    parser.add_argument("--upstream", help="override the config's upstream (for tests)")
+    parser.add_argument("--changed-file", help="created iff the mirror content changed")
+    args = parser.parse_args()
+
+    with open(args.config) as f:
+        config = json.load(f)
+    packages = config["indexes"][args.index]
+    upstream = args.upstream or config["upstream"]
+    base_url = upstream.rstrip("/") + "/" + args.index.strip("/") + "/"
+
+    print(f"syncing {args.index} from {base_url}: {', '.join(packages)}")
+    any_fetched = False
+    any_changed = False
+    for package in packages:
+        dest_dir = os.path.join(args.dest, args.index, normalize(package))
+        fetched, changed = sync_package(base_url, package, dest_dir)
+        any_fetched = any_fetched or fetched
+        any_changed = any_changed or changed
+
+    if not any_fetched:
+        if _has_wheels(os.path.join(args.dest, args.index)):
+            print("upstream unreachable; keeping existing mirror unchanged")
+            return 0
+        print("upstream unreachable and nothing mirrored yet — nothing to publish")
+        return 2
+    if any_changed and args.changed_file:
+        with open(args.changed_file, "w") as f:
+            f.write("changed\n")
+    print("sync complete" + (" (changes)" if any_changed else " (no changes)"))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/run-coverage.sh

@@ -21,6 +21,13 @@
 #                        "!real-models" (those specs need a downloaded model).
 #   COVERAGE_EXCLUDE_RE  egrep pattern of profile lines to drop before merging,
 #                        e.g. generated protobuf (grpc/proto/.*\.pb\.go).
+#   COVERAGE_PROCS       parallel Ginkgo processes; 0 lets Ginkgo detect CPUs.
+#   COVERAGE_SUITE_TIMEOUT maximum duration of each recursive root (default 5m).
+#   COVERAGE_PROGRESS_AFTER emit diagnostics when a spec is slow (default 30s).
+#
+# Verbose Ginkgo output is retained in OUTPUT_DIR/logs. The previous run's log
+# for each root is kept with a .previous suffix, so a noisy failure remains
+# available without flooding the commit-hook output.
 #
 # Why one ginkgo invocation per root: passing several recursive roots to a
 # single ginkgo run only merges ONE root's coverprofile into --output-dir
@@ -41,11 +48,43 @@ shift 3
 unit_roots="$*" # space-free tokens (./pkg ./core)
 
 mkdir -p "$out_dir"
+lock_dir="$out_dir/.run-coverage.lock"
+if ! mkdir "$lock_dir" 2>/dev/null; then
+	echo "run-coverage: another coverage run is using $out_dir" >&2
+	echo "run-coverage: wait for it to finish; if none is running, remove stale lock $lock_dir" >&2
+	exit 2
+fi
+cleanup() {
+	for root in $unit_roots ${COVERAGE_E2E_ROOTS:-}; do
+		# --keep-separate-coverprofiles leaves Go's package test binaries behind.
+		# They are deterministic runner artifacts, never source inputs.
+		find "$root" -type f -name '*.test' -delete 2>/dev/null || :
+	done
+	rmdir "$lock_dir" 2>/dev/null || :
+}
+trap cleanup EXIT
+trap 'exit 130' HUP INT TERM
+
+log_dir="$out_dir/logs"
+mkdir -p "$log_dir"
 # Clear per-root profiles from a previous run: the merge collects them by glob,
 # so a stale profile (e.g. from a root that failed to rebuild this run) must not
 # leak into the merged result.
 rm -f "$out_dir"/cover-*.out
+rm -f "$out_dir"/*_cover-*.out
+rm -f "$merged"
 fail=0
+timings="$out_dir/timings.tsv"
+: > "$timings"
+run_started="$(date +%s)"
+
+procs="${COVERAGE_PROCS:-0}"
+suite_timeout="${COVERAGE_SUITE_TIMEOUT:-5m}"
+progress_after="${COVERAGE_PROGRESS_AFTER:-30s}"
+parallel_flags="-p --keep-going --timeout=$suite_timeout --poll-progress-after=$progress_after --poll-progress-interval=10s"
+if [ "$procs" -gt 0 ] 2>/dev/null; then
+	parallel_flags="$parallel_flags --procs=$procs --compilers=$procs"
+fi
 
 # Common optional flags go into "$@"; unquoted ${VAR:+...} would word-split a
 # --tags value that contains a space. The unit roots were captured above, so
@@ -59,26 +98,133 @@ profile_name() {
 	printf 'cover-%s.out' "$(printf '%s' "$1" | sed 's#[./][./]*#_#g; s#^_##; s#_$##')"
 }
 
+log_name() {
+	printf '%s.log' "$(printf '%s' "$1" | sed 's#[./][./]*#_#g; s#^_##; s#_$##')"
+}
+
+rotate_log() {
+	log="$1"
+	if [ -f "$log" ]; then
+		mv -f "$log" "$log.previous"
+	fi
+}
+
+# Ginkgo's recursive-run merger can fail after every suite has passed when a
+# large --coverpkg run produces many profiles. Keep its profiles separate and
+# merge them here using the same block-summing rule as the cross-root merge.
+consolidate_root_profiles() {
+	base="$1"
+	set -- "$out_dir"/*_"$base"
+	if [ ! -e "$1" ]; then
+		echo "run-coverage: no per-package profiles produced for $base" >&2
+		return 1
+	fi
+	tmp="$out_dir/.${base}.tmp"
+	{
+		echo "mode: atomic"
+		awk '
+			/^mode:/ { next }
+			{ stmts[$1] = $2; cnt[$1] += $3 }
+			END { for (k in stmts) print k, stmts[k], cnt[k] }
+		' "$@"
+	} > "$tmp"
+	mv "$tmp" "$out_dir/$base"
+	rm -f "$@"
+}
+
+report_failure() {
+	root="$1"
+	log="$2"
+	echo "run-coverage: FAIL — tests under coverage failed for $root" >&2
+	echo "run-coverage: full output: $log" >&2
+	echo "run-coverage: relevant tail:" >&2
+	# Keep the terminal useful even when Ginkgo emits thousands of verbose lines.
+	# The complete log remains available when this short extract is insufficient.
+	summary="$(grep -E 'Summarizing|\[FAIL(ED)?\]|FAIL!|--- FAIL:|Test Suite Failed|could not finalize|Status code: 429|HTTP 429|rate limit|timed out|panic:|fork/exec|no such file or directory|Expected.*(but got|success)' "$log" \
+		| tail -n 30)"
+	if [ -n "$summary" ]; then
+		printf '%s\n' "$summary" >&2
+	else
+		tail -n 30 "$log" >&2
+	fi
+}
+
+record_timing() {
+	root="$1"
+	started="$2"
+	elapsed="$(( $(date +%s) - started ))"
+	printf '%s\t%s\n' "$root" "$elapsed" >> "$timings"
+	echo "run-coverage: TIMING — $root ${elapsed}s"
+}
+
+print_timing_summary() {
+	echo "run-coverage: wall-clock summary"
+	sort -t "$(printf '\t')" -k2,2nr "$timings" | awk -F '\t' '{ printf "  %5ss  %s\n", $2, $1 }'
+	echo "  $(( $(date +%s) - run_started ))s  total"
+	echo "run-coverage: slowest specs/hooks taking at least ${COVERAGE_SLOW_SPEC_THRESHOLD:-3}s (up to ${COVERAGE_SLOW_SPEC_LIMIT:-25} per root)"
+	found=0
+	while IFS="$(printf '\t')" read -r root elapsed; do
+		log="$log_dir/$(log_name "$root")"
+		entries="$(scripts/summarize-ginkgo-waits.sh "${COVERAGE_SLOW_SPEC_THRESHOLD:-3}" "$root" "$log" "${COVERAGE_SLOW_SPEC_LIMIT:-25}")"
+		if [ -n "$entries" ]; then
+			printf '%s\n' "$entries"
+			found=1
+		fi
+	done < "$timings"
+	[ "$found" -eq 1 ] || echo "  none"
+}
+
 # Unit/suite roots: recursive.
 for root in $unit_roots; do
 	base="$(profile_name "$root")"
-	go run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts "$flakes" -v -r "$@" \
-		--cover --covermode=atomic --coverprofile="$base" --output-dir="$out_dir" "$root" || fail=1
+	log="$log_dir/$(log_name "$root")"
+	rotate_log "$log"
+	echo "run-coverage: testing $root (full output: $log)"
+	started="$(date +%s)"
+	# parallel_flags is intentionally word-split: it contains CLI arguments only.
+	# shellcheck disable=SC2086
+	go run github.com/onsi/ginkgo/v2/ginkgo $parallel_flags --keep-separate-coverprofiles --flake-attempts "$flakes" -v -r "$@" \
+		--cover --covermode=atomic --coverprofile="$base" --output-dir="$out_dir" "$root" >"$log" 2>&1 \
+		&& consolidate_root_profiles "$base" \
+		&& echo "run-coverage: PASS — $root" \
+		|| { fail=1; report_failure "$root" "$log"; }
+	record_timing "$root" "$started"
 done
 
 # In-process integration roots: NON-recursive + optional label filter.
 for root in ${COVERAGE_E2E_ROOTS:-}; do
 	base="$(profile_name "$root")"
+	log="$log_dir/$(log_name "$root")"
+	rotate_log "$log"
+	echo "run-coverage: testing $root (full output: $log)"
+	started="$(date +%s)"
 	if [ -n "${COVERAGE_E2E_LABELS:-}" ]; then
-		go run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts "$flakes" -v "$@" \
+		# shellcheck disable=SC2086
+		go run github.com/onsi/ginkgo/v2/ginkgo $parallel_flags --keep-separate-coverprofiles --flake-attempts "$flakes" -v "$@" \
 			--label-filter="$COVERAGE_E2E_LABELS" \
-			--cover --covermode=atomic --coverprofile="$base" --output-dir="$out_dir" "$root" || fail=1
+			--cover --covermode=atomic --coverprofile="$base" --output-dir="$out_dir" "$root" >"$log" 2>&1 \
+			&& consolidate_root_profiles "$base" \
+			&& echo "run-coverage: PASS — $root" \
+			|| { fail=1; report_failure "$root" "$log"; }
 	else
-		go run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts "$flakes" -v "$@" \
-			--cover --covermode=atomic --coverprofile="$base" --output-dir="$out_dir" "$root" || fail=1
+		# shellcheck disable=SC2086
+		go run github.com/onsi/ginkgo/v2/ginkgo $parallel_flags --keep-separate-coverprofiles --flake-attempts "$flakes" -v "$@" \
+			--cover --covermode=atomic --coverprofile="$base" --output-dir="$out_dir" "$root" >"$log" 2>&1 \
+			&& consolidate_root_profiles "$base" \
+			&& echo "run-coverage: PASS — $root" \
+			|| { fail=1; report_failure "$root" "$log"; }
 	fi
+	record_timing "$root" "$started"
 done
 
+print_timing_summary
+
+if [ "$fail" -ne 0 ]; then
+	echo "run-coverage: FAILED — one or more test suites failed; no merged profile was produced." >&2
+	echo "run-coverage: the coverage percentage ratchet was not run." >&2
+	exit "$fail"
+fi
+
 # Collect the per-root profiles by glob (space-safe, no list to track).
 set -- "$out_dir"/cover-*.out
 if [ ! -e "$1" ]; then
@@ -98,4 +244,4 @@ fi
 	' "$@"
 } > "$merged"
 
-exit "$fail"
+echo "run-coverage: all test suites passed; merged profile: $merged"

scripts/run-test-linux-offline.sh

@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: MIT
+set -euo pipefail
+
+if [[ $(uname -s) != Linux ]]; then
+  echo 'kernel-level test egress enforcement is Linux-only' >&2
+  exit 2
+fi
+if [[ $# -lt 2 ]]; then
+  echo "usage: $0 TARGET COMMAND [ARG...]" >&2
+  exit 2
+fi
+
+root=$(cd "$(dirname "$0")/.." && pwd)
+group="localai-test-$$"
+cgroup="/sys/fs/cgroup/$group"
+parent_cgroup="/sys/fs/cgroup$(awk -F: '$1 == "0" {print $3}' /proc/self/cgroup)"
+chain="LAI_TEST_$$"
+
+sudo mkdir "$cgroup"
+cleanup() {
+  echo $$ | sudo tee "$parent_cgroup/cgroup.procs" >/dev/null 2>&1 || true
+  sudo iptables -D OUTPUT -m cgroup --path "$group" -j "$chain" 2>/dev/null || true
+  sudo iptables -F "$chain" 2>/dev/null || true
+  sudo iptables -X "$chain" 2>/dev/null || true
+  sudo ip6tables -D OUTPUT -m cgroup --path "$group" -j "$chain" 2>/dev/null || true
+  sudo ip6tables -F "$chain" 2>/dev/null || true
+  sudo ip6tables -X "$chain" 2>/dev/null || true
+  sudo rmdir "$cgroup" 2>/dev/null || true
+}
+trap cleanup EXIT INT TERM
+
+sudo iptables -N "$chain"
+sudo iptables -A "$chain" -d 127.0.0.0/8 -j ACCEPT
+sudo iptables -A "$chain" -d 10.0.0.0/8 -j ACCEPT
+sudo iptables -A "$chain" -d 172.16.0.0/12 -j ACCEPT
+sudo iptables -A "$chain" -d 192.168.0.0/16 -j ACCEPT
+sudo iptables -A "$chain" -j REJECT
+sudo iptables -I OUTPUT 1 -m cgroup --path "$group" -j "$chain"
+
+sudo ip6tables -N "$chain"
+sudo ip6tables -A "$chain" -d ::1/128 -j ACCEPT
+sudo ip6tables -A "$chain" -j REJECT
+sudo ip6tables -I OUTPUT 1 -m cgroup --path "$group" -j "$chain"
+echo $$ | sudo tee "$cgroup/cgroup.procs" >/dev/null
+
+set +e
+LOCALAI_TEST_KERNEL_ACTIVE=1 "$root/scripts/run-test-offline.sh" "$@"
+command_status=$?
+set -e
+
+# The final rule in each private chain is the external-egress REJECT. Reading
+# its counter makes even an optional probe that swallows its network error a
+# hard, visible test failure. Report aggregate counters only: packet-by-packet
+# kernel logging can itself exhaust CI logs when a client retries aggressively.
+ipv4_blocked=$(sudo iptables -L "$chain" -v -x -n | awk '$3 == "REJECT" { print $1; exit }')
+ipv6_blocked=$(sudo ip6tables -L "$chain" -v -x -n | awk '$3 == "REJECT" { print $1; exit }')
+ipv4_blocked=${ipv4_blocked:-0}
+ipv6_blocked=${ipv6_blocked:-0}
+
+if (( ipv4_blocked > 0 || ipv6_blocked > 0 )); then
+  {
+    echo 'offline-test: external network access was attempted and blocked'
+    echo "offline-test: blocked packets: ipv4=$ipv4_blocked ipv6=$ipv6_blocked"
+    echo 'offline-test: replace real network clients with fixtures or injected test doubles'
+  } >&2
+  exit 1
+fi
+
+exit "$command_status"

scripts/run-test-offline.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: MIT
+set -euo pipefail
+
+if [[ $# -lt 2 ]]; then
+  echo "usage: $0 TARGET COMMAND [ARG...]" >&2
+  exit 2
+fi
+
+target=$1
+shift
+root=$(cd "$(dirname "$0")/.." && pwd)
+
+if [[ ${LOCALAI_TEST_KERNEL_ENFORCE:-0} == 1 && ${LOCALAI_TEST_KERNEL_ACTIVE:-0} != 1 ]]; then
+  exec "$root/scripts/run-test-linux-offline.sh" "$target" "$@"
+fi
+
+exec go run "$root/cmd/test-resources" run "$target" \
+  "$root/test-resources/manifests" "${TEST_RESOURCE_CACHE:-$root/.cache/test-resources}" -- "$@"

scripts/summarize-ginkgo-waits.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env sh
+# summarize-ginkgo-waits.sh THRESHOLD_SECONDS ROOT LOG [LIMIT]
+set -eu
+
+threshold="${1:?missing threshold in seconds}"
+root="${2:?missing test root}"
+log="${3:?missing Ginkgo log}"
+limit="${4:-25}"
+case "$limit" in
+	''|*[!0-9]*) echo "limit must be a non-negative integer" >&2; exit 2 ;;
+esac
+
+# Strip terminal colour sequences, then report slow specs and hooks. This
+# catches sleeps, polling, channel waits, teardown and any other idle time
+# without unsafe attempts to replace Go's process-wide clock primitives.
+sed 's/\[[0-9;]*[[:alpha:]]//g' "$log" | awk -v threshold="$threshold" -v root="$root" '
+	/\[[0-9]+([.][0-9]+)? seconds\]/ {
+		line = $0
+		sub(/^.*\[/, "", line)
+		sub(/ seconds\].*$/, "", line)
+		seconds = line + 0
+		if (seconds < threshold) next
+		description = "(description unavailable)"
+		location = "(location unavailable)"
+		if (getline > 0) description = $0
+		if (getline > 0) location = $0
+		printf "%010.3f\t%-18s\t%s\t%s\n", seconds, root, description, location
+	}
+' | sort -t "$(printf '\t')" -k1,1nr | sed -n "1,${limit}p" | awk -F '\t' '{ printf "  %7.3fs  %s  %s  %s\n", $1 + 0, $2, $3, $4 }'