chore(tests): Avoid network, sleep and more during tests (#11050)

* test: make coverage failures observable

Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: parallelize coverage without remote fixtures

Assisted-by: Codex:gpt-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: add offline resource infrastructure

Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: enforce offline resource replay

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: harden offline resource refresh

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: expose slow coverage waits

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: eliminate avoidable wall-clock waits

Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: remove repeated fixture startup waits

Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: fix offline resource CI portability

Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: cache Go modules before offline tests

Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: drop the static network lint in favour of real isolation

The offline test suite already prevents tests from reaching the network
twice over: run-test-linux-offline.sh puts the test process in a cgroup
and REJECTs egress outside the private ranges, and HardenedTransport
installs testnetwork.LocalGuard to refuse dials that resolve to a public
address. Both fail the test with a precise error at the moment of the
dial.

test-network-lint.sh added neither. Its diff stage defaulted to a HEAD
base, so on a clean checkout it compared the tree against itself and
inspected nothing; the branch's own commits were never examined. It only
produced output when an earlier job step dirtied the tree, and then it
matched a bare https?:// against whatever changed. make react-ui runs
npm install rather than npm ci, so CI rewrote
core/http/react-ui/package-lock.json and the lint reported an npm
registry URL as forbidden test network access:

  +      "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz",

Its fingerprint stage was self-defeating in a quieter way: hashing the
whole tree's network-mechanism inventory meant every rebase onto a master
that touched any _test.go needed a manual baseline bump, so the check
mostly caught its own staleness.

Remove the script, its make target and the two prerequisite edges, along
with the test-network: fixture markers that existed only to suppress it.
The isolation itself is untouched.

Assisted-by: Claude:claude-opus-5 [go vet]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: keep hidden files in the offline test bundle artifact

Cherry-picked from 15a37b0ac on the remote branch. The offline bundle lives
under .cache/, which actions/upload-artifact skips by default, so the Linux
job packed an artifact missing the very file the next step restores.

The other half of 15a37b0ac moved test-network-lint out of the `test` and
`test-coverage` prerequisite lists into a recipe line, so parallel make could
not fingerprint the tree while generated fixtures were still changing. That
is dropped: the preceding commit removes the lint entirely, and the race it
worked around is one more reason a whole-tree fingerprint was the wrong
mechanism.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* refactor: share bounded exponential backoff

Use overflow-safe saturating arithmetic for retry delays across model import polling, downloads, registration, node operations, and model loading. Keep model import status checks responsive initially while capping their interval at 500ms.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: mirror Jetson Python wheels

Keep the CUDA aarch64 wheel subset in GHCR and serve it as a local PEP 503 index during L4T backend builds, preserving last-known-good packages through upstream outages.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* docs(agents): index the Jetson wheels mirror

Mention the GHCR-hosted L4T wheel mirror in the CI caching guide summary so maintainers can find its outage and cache documentation.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: add defensive build network proxy

Record build destinations and byte counts, retry observable idempotent HTTP downloads, and isolate explorer database tests that race under coverage.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(kokoros): implement updated backend trait

Return unimplemented for image upscaling, matching the backend's other unsupported modalities after the protobuf API update.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): clear recovered proxy errors

Do not mark a request failed when a later safe retry succeeds.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: require HTTPS build interception

Inject a short-lived proxy CA into BuildKit and Dockerfile RUN steps, reject plain HTTP and opaque tunnels, and retain method/status/byte telemetry for verified HTTPS traffic.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): preserve system trust in unproxied builds

Mount the generated interception CA at a dedicated secret path and add it to the trust bundle only in proxy-aware dependency stages. This prevents optional secret mounts from masking the system CA bundle in ordinary backend test builds.

Install the requested Go toolchain before starting the proxy and satisfy cleanup error checks found by CI lint.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): persist build proxy trust

Install the generated proxy CA through the system-managed local certificate directory so ca-certificates upgrades retain it. Avoid turning canceled matrix jobs into proxy cleanup failures.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): trust proxy in nested build scripts

Install the build proxy CA before nested source fetches, route the DS4 package setup through the HTTPS mirror helper, and avoid repeated OCI setup in gallery behavior tests.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): use HTTPS apt sources for Bonsai

Rewrite ARM64 package sources before installing GCC and check gallery fixture cleanup errors so the optimized tests satisfy errcheck.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(privacy-filter): trust build proxy CA

Install the mounted build proxy certificate before privacy-filter's make target fetches its HTTPS sources, for both source and prebuilt builder paths.\n\nAssisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: fail on hidden offline egress

Count cgroup-scoped firewall rejects and fail the offline test harness with bounded aggregate diagnostics. Inject the gen-audio GGUF probe so fixture-backed importer tests do not attempt real network access.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): preserve system CA trust

Build a combined runner certificate bundle instead of replacing public roots with the generated proxy CA. Centralize additive container installation in the shared proxy CA helper.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
This commit is contained in:
Richard Palethorpeandlocalai-org-maint-bot authored and GitHub committed 2026-08-19 10:59:31 +02:00
1 parent de3329e332
commit cb3bf7af3f
121 files changed
+4756 -407

No files matched your search

+71 -2
View File
@@ -475,6 +475,62 @@ function runProtogen() {
}
# When JETSON_WHEELS_DIR points at a directory of wheels mirrored from
# pypi.jetson-ai-lab.io (CI bind-mounts the jetson-wheels OCI image there —
# see backend/Dockerfile.python and .github/workflows/jetson-wheels.yml),
# installRequirements serves it on localhost as a PEP 503 index and swaps the
# jetson index host in the requirements files for the local one.
#
# The upstream index has a history of multi-hour 502 outages, and a 502 on
# any project page aborts the whole uv resolution — uv consults every
# configured index for every requirement, so even PyPI-hosted packages die
# with it. The local index instead 404s for anything it doesn't carry, which
# resolvers cleanly follow up on PyPI; only the jetson-built wheels (torch
# and friends) resolve locally. When JETSON_WHEELS_DIR is unset — the
# default, e.g. building on a real Jetson — nothing changes and the upstream
# index is used as written in the requirements files.
JETSON_PYPI_HOST="pypi.jetson-ai-lab.io"
_JETSON_MIRROR_PID=""
_JETSON_MIRROR_URL=""
function _stopJetsonMirror() {
if [ -n "${_JETSON_MIRROR_PID}" ]; then
kill "${_JETSON_MIRROR_PID}" 2>/dev/null || true
_JETSON_MIRROR_PID=""
_JETSON_MIRROR_URL=""
fi
}
function _startJetsonMirror() {
local script_dir port_file port tries
# An empty dir is the JETSON_WHEELS_IMAGE=scratch default in
# Dockerfile.python: no mirror was provided, use upstream as-is.
if [ -z "$(find "${JETSON_WHEELS_DIR}" -name '*.whl' -print -quit 2>/dev/null)" ]; then
echo "jetson wheels dir ${JETSON_WHEELS_DIR} has no wheels, using upstream ${JETSON_PYPI_HOST}"
return 0
fi
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
port_file="$(mktemp)"
rm -f "${port_file}"
python3 "${script_dir}/pypi_mirror_server.py" --root "${JETSON_WHEELS_DIR}" --port-file "${port_file}" &
_JETSON_MIRROR_PID=$!
trap _stopJetsonMirror EXIT
tries=0
until [ -s "${port_file}" ]; do
tries=$((tries + 1))
if [ ${tries} -gt 50 ] || ! kill -0 "${_JETSON_MIRROR_PID}" 2>/dev/null; then
echo "WARNING: local jetson wheel mirror failed to start, using upstream ${JETSON_PYPI_HOST}"
_stopJetsonMirror
return 0
fi
sleep 0.2
done
port="$(cat "${port_file}")"
rm -f "${port_file}"
_JETSON_MIRROR_URL="http://127.0.0.1:${port}"
echo "serving jetson wheels from ${JETSON_WHEELS_DIR} at ${_JETSON_MIRROR_URL}"
}
# installRequirements looks for several requirements files and if they exist runs the install for them in order
#
# - requirements-install.txt
@@ -520,18 +576,31 @@ function installRequirements() {
export C_INCLUDE_PATH="${C_INCLUDE_PATH:-}:$(_portable_dir)/include/python${PYTHON_VERSION}"
fi
if [ -n "${JETSON_WHEELS_DIR:-}" ] && [ -d "${JETSON_WHEELS_DIR}" ]; then
_startJetsonMirror
fi
local installFile
for reqFile in ${requirementFiles[@]}; do
if [ -f "${reqFile}" ]; then
installFile="${reqFile}"
if [ -n "${_JETSON_MIRROR_URL}" ] && grep -q "${JETSON_PYPI_HOST}" "${reqFile}"; then
installFile="$(mktemp)"
sed "s,https://${JETSON_PYPI_HOST},${_JETSON_MIRROR_URL},g" "${reqFile}" > "${installFile}"
echo "rewrote ${JETSON_PYPI_HOST} in ${reqFile} to the local wheel mirror (${installFile})"
fi
echo "starting requirements install for ${reqFile}"
if [ "x${USE_PIP}" == "xtrue" ]; then
pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement "${reqFile}"
pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement "${installFile}"
else
uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement "${reqFile}"
uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement "${installFile}"
fi
echo "finished requirements install for ${reqFile}"
fi
done
_stopJetsonMirror
runProtogen
}
+226
View File
@@ -0,0 +1,226 @@
"""Ephemeral PEP 503 "simple" index over a local directory of wheels.
Serves a directory tree laid out like a package index (e.g.
``<root>/jp6/cu129/torch/torch-2.8.0-cp312-...whl``) as a standards-compliant
"simple" index on localhost, so uv/pip can resolve against it exactly as they
would against the real remote index — same per-project pages, same 404
fall-through to PyPI for projects the mirror does not carry.
This exists because pypi.jetson-ai-lab.io (the only source of CUDA-enabled
aarch64 torch wheels for JetPack) has a history of multi-hour 502 outages,
and an --extra-index-url that errors is fatal to the whole resolution: uv
consults every configured index for every requirement, so one 502 on any
project page kills the install even for projects hosted on PyPI. CI mirrors
the handful of jetson-only wheels into an OCI image, bind-mounts it into the
backend build, and libbackend.sh serves it with this script while rewriting
the index host in the requirements files to 127.0.0.1 (see
installRequirements in libbackend.sh). A 404 from this server is a clean
"not here" that resolvers follow up on PyPI; the upstream 502 never was.
Standard library only — it runs inside every python backend's build
container, before any venv exists.
Usage:
python3 pypi_mirror_server.py --root /jetson-wheels --port-file /tmp/port
Run the tests standalone:
python3 -m unittest pypi_mirror_server_test
"""
import argparse
import hashlib
import html
import os
import re
import sys
import threading
import urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
# Extensions treated as distribution files: a directory containing at least
# one of these is a project page; any other directory is a sub-index listing.
DIST_SUFFIXES = (".whl", ".tar.gz", ".zip")
_hash_cache = {}
_hash_lock = threading.Lock()
def normalize(name):
"""PEP 503 project-name normalization."""
return re.sub(r"[-_.]+", "-", name).lower()
def _file_sha256(path):
"""sha256 of a file, cached on (path, mtime, size) — wheels are large."""
st = os.stat(path)
key = (path, st.st_mtime_ns, st.st_size)
with _hash_lock:
cached = _hash_cache.get(key)
if cached:
return cached
digest = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
digest.update(chunk)
value = digest.hexdigest()
with _hash_lock:
_hash_cache[key] = value
return value
def resolve_path(root, url_path):
"""Map a URL path onto the tree under root, or None.
Path segments match either exactly or via PEP 503 normalization
(resolvers request ``liquid-audio`` even if the directory on disk is
named ``liquid_audio``). Rejects any segment that would escape root.
"""
current = root
for segment in url_path.split("/"):
if segment in ("", "."):
continue
if segment == ".." or "/" in segment or "\\" in segment:
return None
candidate = os.path.join(current, segment)
if not os.path.exists(candidate):
try:
entries = os.listdir(current)
except (NotADirectoryError, FileNotFoundError):
return None
wanted = normalize(segment)
matches = [e for e in entries if normalize(e) == wanted]
if not matches:
return None
candidate = os.path.join(current, matches[0])
current = candidate
return current
class SimpleIndexHandler(BaseHTTPRequestHandler):
root = None
protocol_version = "HTTP/1.1"
def do_GET(self):
self._respond(head_only=False)
def do_HEAD(self):
self._respond(head_only=True)
def _respond(self, head_only):
url_path = urllib.parse.unquote(urllib.parse.urlsplit(self.path).path)
local = resolve_path(self.root, url_path)
if local is None:
self._send_error(404, "not found")
return
if os.path.isfile(local):
self._send_file(local, head_only)
return
# Relative hrefs on index pages resolve against the request URL, so
# directory URLs must end in "/" — redirect like real indexes do.
if not url_path.endswith("/"):
self.send_response(301)
self.send_header("Location", self.path + "/")
self.send_header("Content-Length", "0")
self.end_headers()
return
entries = sorted(os.listdir(local))
files = [e for e in entries if e.endswith(DIST_SUFFIXES)]
if files:
body = self._project_page(local, files)
else:
dirs = [e for e in entries if os.path.isdir(os.path.join(local, e))]
body = self._listing_page(dirs)
self._send_html(body, head_only)
def _project_page(self, project_dir, files):
anchors = []
for name in files:
digest = _file_sha256(os.path.join(project_dir, name))
anchors.append(
'<a href="%s#sha256=%s">%s</a><br/>'
% (urllib.parse.quote(name), digest, html.escape(name))
)
return self._page(anchors)
def _listing_page(self, dirs):
anchors = [
'<a href="%s/">%s</a><br/>'
% (urllib.parse.quote(normalize(d)), html.escape(normalize(d)))
for d in dirs
]
return self._page(anchors)
def _page(self, anchors):
return (
"<!DOCTYPE html><html><head>"
'<meta name="pypi:repository-version" content="1.0">'
"<title>simple index</title></head><body>\n"
+ "\n".join(anchors)
+ "\n</body></html>"
).encode()
def _send_html(self, body, head_only):
self.send_response(200)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
if not head_only:
self.wfile.write(body)
def _send_file(self, path, head_only):
size = os.path.getsize(path)
self.send_response(200)
self.send_header("Content-Type", "application/octet-stream")
self.send_header("Content-Length", str(size))
self.end_headers()
if head_only:
return
with open(path, "rb") as f:
while True:
chunk = f.read(1 << 20)
if not chunk:
break
self.wfile.write(chunk)
def _send_error(self, code, message):
body = message.encode()
self.send_response(code)
self.send_header("Content-Type", "text/plain")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, format, *args):
sys.stderr.write("pypi-mirror: %s\n" % (format % args))
def make_server(root, host="127.0.0.1", port=0):
handler = type("Handler", (SimpleIndexHandler,), {"root": os.path.abspath(root)})
return ThreadingHTTPServer((host, port), handler)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--root", required=True, help="directory tree to serve")
parser.add_argument("--host", default="127.0.0.1")
parser.add_argument("--port", type=int, default=0, help="0 picks a free port")
parser.add_argument(
"--port-file",
help="write the bound port here once listening (readiness signal)",
)
args = parser.parse_args()
server = make_server(args.root, args.host, args.port)
port = server.server_address[1]
if args.port_file:
# Write-then-rename so a reader never sees a partially written port.
tmp = args.port_file + ".tmp"
with open(tmp, "w") as f:
f.write(str(port))
os.replace(tmp, args.port_file)
sys.stderr.write("pypi-mirror: serving %s on %s:%d\n" % (args.root, args.host, port))
server.serve_forever()
if __name__ == "__main__":
main()
@@ -0,0 +1,100 @@
"""Unit tests for the ephemeral PEP 503 index (pypi_mirror_server.py).
Run standalone (Python standard library only, no backend venv needed):
python3 -m unittest pypi_mirror_server_test
"""
import hashlib
import os
import shutil
import tempfile
import threading
import unittest
import urllib.error
import urllib.request
from pypi_mirror_server import make_server, normalize, resolve_path
WHEEL_BYTES = b"not a real wheel, but the server must serve it verbatim"
class TestHelpers(unittest.TestCase):
def test_normalize(self):
self.assertEqual(normalize("Liquid_Audio.Extra"), "liquid-audio-extra")
self.assertEqual(normalize("torch"), "torch")
def test_resolve_rejects_traversal(self):
root = tempfile.mkdtemp()
try:
self.assertIsNone(resolve_path(root, "/../etc/passwd"))
self.assertIsNone(resolve_path(root, "/a/../../etc"))
finally:
shutil.rmtree(root)
def test_resolve_normalized_segment(self):
root = tempfile.mkdtemp()
try:
os.makedirs(os.path.join(root, "jp6", "liquid_audio"))
found = resolve_path(root, "/jp6/liquid-audio/")
self.assertEqual(found, os.path.join(root, "jp6", "liquid_audio"))
finally:
shutil.rmtree(root)
class TestServer(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.root = tempfile.mkdtemp()
project = os.path.join(cls.root, "jp6", "cu129", "torch")
os.makedirs(project)
cls.wheel_name = "torch-2.8.0-cp312-cp312-linux_aarch64.whl"
with open(os.path.join(project, cls.wheel_name), "wb") as f:
f.write(WHEEL_BYTES)
cls.server = make_server(cls.root)
cls.base = "http://127.0.0.1:%d" % cls.server.server_address[1]
cls.thread = threading.Thread(target=cls.server.serve_forever, daemon=True)
cls.thread.start()
@classmethod
def tearDownClass(cls):
cls.server.shutdown()
cls.server.server_close()
shutil.rmtree(cls.root)
def _get(self, path):
with urllib.request.urlopen(self.base + path) as resp:
return resp.status, resp.read()
def test_project_page_lists_wheel_with_hash(self):
status, body = self._get("/jp6/cu129/torch/")
self.assertEqual(status, 200)
digest = hashlib.sha256(WHEEL_BYTES).hexdigest()
self.assertIn(
('<a href="%s#sha256=%s">' % (self.wheel_name, digest)).encode(), body
)
def test_index_listing_names_projects(self):
status, body = self._get("/jp6/cu129/")
self.assertEqual(status, 200)
self.assertIn(b'<a href="torch/">torch</a>', body)
def test_wheel_download_is_verbatim(self):
status, body = self._get("/jp6/cu129/torch/" + self.wheel_name)
self.assertEqual(status, 200)
self.assertEqual(body, WHEEL_BYTES)
def test_unknown_project_is_404(self):
# 404 (not 5xx) matters: resolvers treat it as "not in this index"
# and fall back to PyPI, which is the whole point of the mirror.
with self.assertRaises(urllib.error.HTTPError) as ctx:
self._get("/jp6/cu129/liquid-audio/")
self.assertEqual(ctx.exception.code, 404)
def test_directory_without_slash_redirects(self):
status, _ = self._get("/jp6/cu129/torch")
# urllib follows the 301; landing on the page proves the redirect
self.assertEqual(status, 200)
if __name__ == "__main__":
unittest.main()