shelfmark/tests/e2e/platform

Shelfmark E2E Docker Testing Platform

A hermetic, container-based end-to-end platform that boots the real Shelfmark app against controllable dependencies — a fake Anna's Archive, a Cloudflare gate, a mock FlareSolverr bypasser, a mock Prowlarr + a real qBittorrent, custom DNS servers, HTTP/SOCKS5 proxies, and a Tor profile — and runs a cluster test suite under each config profile.

It exists to make the recurring bug clusters from the issue/PR analysis impossible to reintroduce silently. The biggest one — Tor/Cloudflare/bypasser (37 issues / 67 fix PRs) — had almost no automated coverage; this platform changes that.

  pytest suite (host :8084)
        │
        ▼
  shelfmark (under test) ── egress depends on the active profile:
        ├─ direct ──────────────► mock-aa (.10)                  fake Anna's Archive
        ├─ Cloudflare gate ─────► mock-cf (.11) ─► mock-aa       [full: real Chrome solves it]
        ├─ FlareSolverr ────────► mock-cf (.11) ─► mock-flaresolverr (.12)  [bypasser-external]
        ├─ custom DNS ──────────► coredns (.20) / coredns-blocked (.22)     [dns-manual/blocked]
        ├─ HTTP / SOCKS proxy ──► tinyproxy (.30) / microsocks (.31)        [proxy-http/socks]
        ├─ Tor (transparent) ───► in-image tor.sh                           [tor]
        └─ Prowlarr → client ───► mock-prowlarr (.40) ─► qBittorrent        [full: real download]

  (all on one e2e docker network, 172.30.0.0/24, static IPs for DNS determinism)

Quick start

# one profile
make e2e-platform                       # baseline
make e2e-platform-profile PROFILE=bypasser-external
make e2e-platform-profile PROFILE=client-deluge
make e2e-platform-full                  # heavy: real Chrome + DoH + real qBittorrent

# the whole matrix
make e2e-platform-matrix

# build the heavy image once, then reuse it (matrix does this automatically)
make e2e-platform-build
E2E_NO_BUILD=1 tests/e2e/platform/run-e2e.sh env/dns-doh.env

# debug: leave the stack up after the run
KEEP_UP=1 tests/e2e/platform/run-e2e.sh env/dns-blocked.env

Requirements: Docker + Compose v2, and uv (for the pytest runner). The runner builds the Shelfmark image from the repo Dockerfile, boots the profile's stack, waits for /api/health, runs the suite, and tears down. run-matrix.sh builds the image once and reuses it across profiles (E2E_NO_BUILD=1) so the slow xvfb/chromium layer isn't rebuilt per profile.

How profiles work

Each profile is an env file in env/. It sets:

• COMPOSE_PROFILES — which optional services start (compose profiles:).
• SM_* — the app's config, injected as container env. Shelfmark treats deployment ENV as authoritative (config.get: "Deployment-level ENV values always win"), so a profile fully determines the app's DNS/proxy/bypasser/source configuration with no runtime mutation.
• E2E_PROFILE — handed to pytest so the suite selects applicable tests.

Tests declare applicability with @pytest.mark.profiles(...). A test with no marker is a profile-agnostic invariant and runs under every profile — that is how one cluster test ("source must be reachable") becomes the config matrix.

The matrix (cluster × profile)

Status column: ✅ = run live on Docker and passing. Every profile below was run end-to-end (docker compose up + suite + teardown) and passes.

Profile	Egress / what it proves	Clusters	Regression targets	Status
baseline	Direct to fake AA; search/parse + #1028 clean-failure	2,3,4	#198 #293 #214 #1040 #1028	✅ 9 passed
bypasser-external	External bypasser wired; CF-gated search fails cleanly	1	#284 #202 #410 #369	✅ 5 passed
bypasser-disabled	CF-gated AA + bypasser OFF → no results (control)	1	#202 #410	✅ 4 passed
dns-manual	AA only resolvable via custom DNS (coredns)	config: DNS	#108	✅ 4 passed
dns-blocked	System DNS NXDOMAINs AA; custom DNS resolves it	config: DNS	#1028	✅ 4 passed
dns-doh	System DNS blocks AA; DoH over real HTTPS resolves it	config: DoH	#1028 #108	✅ 3 passed
proxy-http	All egress via tinyproxy, proven by proxy logs	config: proxy	#956	✅ 6 passed
proxy-socks	All egress via SOCKS5 (microsocks), traversal-checked	config: proxy	#956	✅ 5 passed
tor	USING_TOR=true boots clean (restarts=0)	1/6 Tor boot	#1021 #940 #801	✅ 5 passed
client-transmission	Prowlarr → real Transmission webseed download → /books	5 clients	#1022 #634	✅ 4 passed
client-deluge	Prowlarr → real Deluge webseed download → /books	5 clients	#530	✅ 4 passed
full	real Chrome solves Cloudflare + DoH + real qBittorrent download → /books (Moby-Dick)	1,4,5 + DoH	#284 #1030 #386 #1040 #214	✅ 6 passed
(every profile)	boots healthy under PUID/PGID, no perm errors	6 entrypoint	#171 #447 #801	✅

The bypasser is download-time, not search-time. Running the stack revealed that shelfmark fetches AA search/detail with allow_bypasser_fallback=False, so a search behind Cloudflare returns 503 regardless of the bypasser; the bypasser (internal Chrome or external FlareSolverr) only runs during a file download (use_bypasser=True). The bypasser profiles therefore assert a clean CF-gated-search failure, while the full profile exercises the real end-to-end CF solve: AA search/detail are reachable, but the AA slow-download link points at the gate, so downloading Moby-Dick forces the in-image headless Chromium to detect the challenge, solve it (_bypass_method_cdp_solve), and fetch the file — verified live (Challenge detected: cloudflare → Bypass successful → Moby-Dick in /books).

bypasser-external must set SM_USING_EXTERNAL_BYPASSER=true — shelfmark does not derive it from EXT_BYPASSER_URL; without it the app silently uses the in-image Chrome bypasser instead of FlareSolverr.

Coverage of the 7 clusters from the analysis:

1. Bypasser/Tor/Cloudflare → bypasser-external, bypasser-disabled, tor.
2. Search/metadata → baseline (test_cluster_search_aa.py, hermetic via the direct_download source so no external metadata provider is needed).
3. AA parsing/mirrors → baseline parse guards incl. the layout-drift fail-loud test (#878/#879/#880).
4. Permissions/file-move → baseline (test_cluster_download_permissions.py).
5. Torrent/usenet clients → a mock Prowlarr + webseed torrent drives three real torrent clients end to end (full=qBittorrent, client-transmission, client-deluge) — completion detection + file move into /books. One client-agnostic test (test_cluster_clients.py) covers all three.
6. Docker/entrypoint/PUID-PGID → profile-agnostic health + boot-log checks, run under every profile.
7. Audiobook/ABB → parse-contract guards in tests/audiobookbay/test_scraper_contract.py (info-hash normalization #386, magnet fallback, layout drift). These run in normal CI, not the docker matrix, because ABB hardcodes https:// for its fetches (see Roadmap).

Proxy traversal (not just reachability)

Because the app and the mock AA share the e2e network, a regression that ignores the proxy config would still reach AA directly. test_egress_actually_traverses_proxy drives a search and then inspects the proxy container's logs, so the proxy profiles prove the egress went through the proxy — a real guard for #956.

DoH — two layers

• Offline (tests/download/test_doh_resolver_mock.py, normal CI): the real DoHResolver is driven against the mock doh role over localhost HTTP, covering JSON-answer parsing, NXDOMAIN → empty, and caching.
• In-stack (dns-doh profile): the mock doh role serves the DNS JSON API over real HTTPS (self-signed). The system resolver (coredns-blocked) NXDOMAINs aa.mock.test, so the host can only be resolved via DoH; compose extra_hosts redirects the cloudflare-dns.com provider to the in-stack mock and CERTIFICATE_VALIDATION=disabled accepts the self-signed cert. The search reaching AA proves the app's DoH path resolved the name end to end — no app code change.

The full profile — real Chrome + real client (make e2e-platform-full)

The "everything real" heavy profile (test book: Moby-Dick), run nightly / on demand (excluded from the PR matrix). It spins up, with no mock bypasser, and passes live (6 passed):

• Real Chrome solves Cloudflare, end to end (VERIFIED). AA search/detail are reachable (mock-aa), but the AA slow-download link points at the Cloudflare gate (mock-cf), whose challenge page runs JS that issues cf_clearance and reloads. Downloading Moby-Dick forces the in-image headless Chromium (seleniumbase CDP, in the shelfmark image via xvfb+chromium) to load the gate, detect the challenge (Challenge detected: cloudflare), solve it (_bypass_method_cdp_solve), and fetch the cleared "Download now" page → the file lands in /books. That outcome is only reachable if Chrome solved the gate — the literal "spin a Chrome browser" path and the strongest guard for the #1 cluster. Two subtleties this surfaced, now handled by the mock: the cleared page must exceed the bypasser's _LOADING_BODY_LENGTH_MAX (50 chars of innerText) or it loops as "still loading", and the AA detail page must satisfy the brittle original_nodes[-6] parse (#880).
• DoH on at boot.
• Real qBittorrent download. A mock Prowlarr (/api/v1/system/status, /api/v1/indexer, torznab search) returns one release whose .torrent is a tracker-less BEP-19 webseed pointing at mock-aa's HTTP payload. A real qBittorrent completes the download over HTTP (no tracker/peer/seeder), and shelfmark's completion detection + file move lands the book in /books. The webseed torrent is generated by mocks/make_webseed_torrent.py (cross-checked against shelfmark's own bencode/infohash parser in tests/download/test_webseed_torrent_generator.py), and the whole prowlarr→qBittorrent path is configured declaratively via env (env/full.env).

Components

Path	Purpose
mocks/mock_services.py	One Flask app, five roles (origin-aa, cloudflare, flaresolverr, prowlarr, doh) selected by MOCK_ROLE. origin-aa also serves the webseed payload + .torrent.
mocks/make_webseed_torrent.py	Stdlib bencode + BEP-19 webseed .torrent generator for the full real-client download.
qbittorrent/qBittorrent.conf	Real qBittorrent config (auth bypassed for the e2e subnet) for the full profile.
env/full.env	The heavy full profile: real Chrome bypasser + DoH + real qBittorrent.
mocks/fixtures/*.html	AA search/detail HTML in the exact shape the parser expects, plus drift/empty/no-files variants.
docker-compose.e2e.yml	The stack; optional services gated by compose profiles, static IPs for DNS determinism.
dns/Corefile*, dns/mock.test.db	coredns zones — working + ISP-block (NXDOMAIN).
env/*.env	The config profiles (matrix rows).
suite/	The pytest harness + cluster tests.
run-e2e.sh / run-matrix.sh	Boot one profile / loop the matrix.
build-images.sh	Build the heavy image once (make e2e-platform-build); reused via E2E_NO_BUILD=1.

Fault injection

The mock AA reproduces historical bugs deterministically. Injection rides inside the search query as E2EINJECT:<name> (the app builds the AA URL itself and only forwards the user query as q=). Names: no_files, empty, layout_drift, 500. The harness embeds them via PlatformClient.direct_search(..., inject=...).

Gating PRs (block merge on e2e failure)

The .github/workflows/e2e-platform.yml workflow runs on every PR. On a PR that touches relevant code (shelfmark/**, Dockerfile, entrypoint.sh, tor.sh, tests/e2e/platform/**) it runs the fast PR subset and the heavy full profile (real Chrome solving Cloudflare + DoH + real qBittorrent), then a single e2e required job aggregates them: it fails if any e2e job failed, and passes (so it never hangs) when the e2e jobs are skipped on an unrelated PR.

The workflow producing a failing check is not enough on its own — GitHub only blocks merge on checks listed in branch protection. A repo admin must, once:

• UI: Settings → Branches → branch protection rule for main → Require status checks to pass before merging → add e2e required.
• or gh (admin token):

  gh api -X PUT repos/calibrain/shelfmark/branches/main/protection \
    -H "Accept: application/vnd.github+json" --input - <<'JSON'
  { "required_status_checks": { "strict": true, "contexts": ["e2e required"] },
    "enforce_admins": true, "required_pull_request_reviews": null, "restrictions": null }
  JSON
  

After that, any failure in the e2e platform tests (including the full profile) blocks the PR from merging. Requiring just the one e2e required context covers the whole dynamic matrix, so the list never needs updating as profiles change.

Known limitations / follow-ups

• rTorrent. Not in the matrix: its rakshasa-libtorrent has no GetRight/webseed support, so the hermetic webseed torrent (which qBittorrent/Transmission/Deluge all complete) leaves rTorrent stuck at 0%. Supporting it needs a real tracker + seeder (peer download) — a follow-up that the webseed design intentionally avoids.
• Usenet clients (SABnzbd/NZBGet). Not yet covered — completing a usenet download hermetically needs a mock NNTP server serving the yEnc-encoded payload plus an NZB, which is a separate (larger) build than the torrent webseed path.
• Audiobook (cluster 7) in-stack. ABB hardcodes https://, so it's covered offline (tests/audiobookbay/test_scraper_contract.py); an in-stack audiobookbay role needs the same self-signed-HTTPS plumbing the dns-doh profile now uses for DoH.