Cleanuparr/docs/contributing/e2e-testing.md

End-to-End Testing with Keycloak and Playwright

E2E tests use a real Keycloak instance and Playwright browser automation to validate full OIDC round-trips that mocked tests cannot catch.

Test Coverage Layers

Layer	What it catches
Unit tests (OidcAuthServiceTests)	PKCE, URL encoding, token validation logic, expiry handling
Integration tests (OidcAuthControllerTests, AccountControllerOidcTests)	HTTP routing, middleware, cookie/token handling (mocked IdP)
E2E tests	Real browser redirects, actual Keycloak protocol, full OIDC round-trip

Prerequisites

• Docker + Docker Compose
• Node.js 26+
• GitHub Packages credentials (for building the app image)

Running Locally

cd e2e

# Prepare test-data/ and start the stack, waiting for containers to become healthy.
# Use up-clients instead for the download-cleaner and malware-blocker specs.
make up-core

# Install dependencies and browser
make install

# Run a folder's tests (its setup project runs automatically)
npx playwright test --project=core

# Tear down
make down

make up-core starts the app, Keycloak, nginx and the WireMock servers. make up-clients adds opentracker and the five real torrent clients (qBittorrent, Transmission, Deluge, uTorrent, ruTorrent). make up-arr adds real Sonarr and Radarr containers, a fake Torznab indexer and qBittorrent, for the live-arr folder. All three run scripts/setup-test-data.sh first, which writes the qBittorrent config the tests authenticate against and restores the *arr configs.

Live *arr Tests

tests/live-arr/ runs the Seeker against real Sonarr and Radarr. Only the indexer is faked, because no real indexer can be shipped in the test stack. A search is a real SeasonSearch or MoviesSearch command, and the grab lands in a real qBittorrent and in the *arr's own queue.

The containers start with a library already in place. e2e/arr-seed/ holds a committed config.xml and database per *arr, and setup-test-data.sh copies them into test-data/ on every run. CI therefore needs no metadata lookups.

Regenerate the seed with make seed-arr. That is the only step needing internet: Sonarr resolves the series through Skyhook and Radarr resolves the movie through TMDB. Re-run it when the pinned sonarr or radarr image tag changes, because a seed database is only valid for the version that wrote it.

The seed holds one Sonarr series with its first season monitored, and two Radarr movies. The second movie is left unmonitored on purpose, so a spec that wants a single candidate gets one. The specs that need two monitor it themselves and put it back.

The two live folders

tests/live-arr runs against the shipped image. It waits for the real cron schedule, which is the only place that path is exercised.

tests/live-arr-fast runs against an image built from patched sources, started with make up-arr-fast. The patches live in e2e/patches:

Patch	What it removes
0001-allow-triggering-the-seeker	the guard that rejects POST /api/jobs/Seeker/trigger
0002-drop-the-seeker-jitter	the random delay a run waits before searching
0003-shorten-the-search-command-timeout	30 minutes of waiting before a running command times out, cut to 20 seconds, with the monitor poll cut from 60 to 5 seconds

Together they turn a two-to-four minute wait per test into seconds, which is what makes a behaviour matrix affordable. The third patch is what makes seeker-timeout.api.spec.ts possible: it holds the indexer's answer so the arr's command keeps running, then asserts the event settles on TimedOut.

The poll cut is not cosmetic. A spec must let the indexer answer before the arr gives up on it, because an arr that times out on an indexer benches it for about a minute, and the next spec then searches with no active indexer. The 5-second poll settles the event long before that. scripts/with-patches.sh applies the patches, builds, and reverts from a trap, so an aborted build cannot leave the tree dirty.

The split matters: the patched image is not the shipped one, so anything that depends on scheduling or timing has to stay in tests/live-arr. Put behaviour in live-arr-fast and timing in live-arr.

Specs in live-arr-fast must never rely on the schedule. Its setup project parks searchInterval at its 360-minute maximum, and arrangeInstance verifies every override it sends actually persisted, so a silently dropped setting fails loudly instead of making a test pass for the wrong reason.

How It Works

1. Docker Compose starts Keycloak (with a pre-configured realm), the Cleanuparr app, nginx and four WireMock servers standing in for the *arr, download client, notification and blocklist endpoints
2. Playwright globalSetup (tests/global-setup.ts) waits for Keycloak, the app and WireMock to come up
3. Each spec folder is its own Playwright project, paired with a setup:<folder> project (tests/_setup/<folder>.setup.ts) that runs first via dependencies. The setup restarts the app container — whose /config is a tmpfs, so state is wiped — re-creates the admin, and writes fresh tokens to playwright/.auth/admin.json. The folder is the isolation boundary; specs within a folder cooperate
4. tests/oidc/ holds the numbered OIDC UI specs, and its setup project enables OIDC against the Keycloak realm before they run

CI

E2E tests do not run automatically. Trigger them on a pull request by commenting /e2e, which dispatches .github/workflows/e2e.yml through .github/workflows/pr-build.yml. The result is posted back as a PR comment; reports land in the e2e-test-results-core, e2e-test-results-clients and e2e-test-results-live-arr artifacts of that run.

Only the accounts in the allowlist in .github/workflows/pr-build.yml can trigger a run. A command from any other account does nothing and gets no reply. Ask a maintainer to run the suite for you.

The suite runs as four matrix legs split by service dependency: core covers the 13 folders that need only the app and its mocks, clients covers download-cleaner and malware-blocker, which drive the real torrent clients, and live-arr plus live-arr-fast cover the folders that drive real Sonarr and Radarr containers. The live-arr-fast leg builds the app from patched sources; every other leg builds it clean.