test(react-ui): add page render-smoke specs, reset the coverage gate (#10122)

The UI coverage gate was tightened to 0.1pp against a fast-local
measurement (39.86% baseline); CI's slower runners measure ~0.9pp lower,
so tests-ui-e2e failed there. UI e2e coverage is diffusely
non-deterministic and tracks machine speed — a 0.1pp band can't hold
across environments.

Rather than loosen the gate, raise the floor under it: a render-smoke
spec mounts each lazy page (navigate + assert the header renders),
covering a dozen previously-untested pages and lifting coverage from
~39% to ~42.7% locally. Restore the tolerance to 0.8pp and set the
baseline conservatively (40.0), below the slow-CI floor, so the ratchet
holds without flapping.

Document the coverage policy — install the git hooks and don't bypass
them (no --no-verify, no hand-lowering the baseline or widening the
tolerance); raise coverage by adding tests instead; set the UI baseline
below the slow-CI floor — in AGENTS.md, CONTRIBUTING.md and
.agents/building-and-testing.md.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Richard Palethorpe <io@richiejp.com>
This commit is contained in:
Richard Palethorpe authored and GitHub committed 2026-06-01 14:24:36 +02:00
1 parent c01ed631d6
commit 5a0013defe
6 files changed
+75 -20

No files matched your search

+19 -14
View File
@@ -4,28 +4,33 @@
#
# Compares the total line coverage in an nyc coverage-summary.json against a
# committed baseline and fails (exit 1) if it dropped by more than
# UI_COVERAGE_TOLERANCE percentage points (default 0.1). The React UI e2e suite
# UI_COVERAGE_TOLERANCE percentage points (default 0.8). The React UI e2e suite
# drives the real app, so a removed feature or deleted spec shows up as a
# coverage drop here.
#
# The tolerance exists only to absorb the irreducible measurement noise floor,
# NOT to permit regression. UI e2e coverage USED to swing ~1pp run-to-run, which
# forced a loose 0.8pp band — but that swing was a bug, not inherent jitter: a
# spec that navigated to a route and ended on the URL assertion let the target
# component's render race the coverage teardown, so ~400 lines were collected
# only when the render won (see e2e/agents.spec.js → AgentCreate). With that race
# fixed, repeated runs land within ~0.013pp (a handful of lines) of each other,
# so the band is tightened to 0.1pp — enough for the noise floor, tight enough
# that a real ~40-line regression still trips the gate. If a future run wobbles
# more, fix the racing spec (await a rendered element) rather than loosening this.
# Why the band is this wide: UI e2e line coverage is NOT deterministic. Many
# specs assert on state and end while async/lazy render work is still in flight,
# so those lines are collected only when the render beats the coverage teardown
# — and that depends on machine speed/load. The effect is diffuse (spread across
# dozens of specs, no single dominant file) and tracks the runner: a quiet local
# box measures ~0.9pp higher than a slow/loaded CI runner for the SAME tree
# (observed: 39.9% local vs 39.0% CI). The tolerance absorbs that spread; setting
# it tighter (it was briefly 0.1pp, calibrated to a lucky fast-local cluster)
# makes CI flap.
#
# When coverage rises meaningfully, regenerate and commit the baseline with:
# make test-ui-coverage-baseline
# The principled way to tighten this is to remove the variance at the source —
# make each racing spec await a rendered element before ending (e2e/agents.spec.js
# → AgentCreate fixed the single biggest one) — NOT to chase the baseline up to a
# fast-machine high or loosen further. Keep the baseline conservatively at or
# below the slow-runner floor so the band catches real regressions, not jitter.
#
# When coverage rises meaningfully AND reproducibly (check on a slow/CI-like run),
# regenerate and commit the baseline with: make test-ui-coverage-baseline
set -eu
summary="${1:?usage: ui-coverage-check.sh SUMMARY_JSON BASELINE_FILE}"
baseline_file="${2:?usage: ui-coverage-check.sh SUMMARY_JSON BASELINE_FILE}"
tolerance="${UI_COVERAGE_TOLERANCE:-0.1}"
tolerance="${UI_COVERAGE_TOLERANCE:-0.8}"
if [ ! -f "$summary" ]; then
echo "ui-coverage-check: coverage summary not found: $summary" >&2