Mutable tags like @v6 can be re-pointed by whoever controls the action,
so the next run executes code nobody here reviewed, with the job's
token and secrets. Pinning to the full commit sha freezes what runs.
Version tags are kept as comments so renovate or dependabot can still
track updates.
dtolnay/rust-toolchain is pinned to a master commit with the toolchain
moved to an explicit input, as its readme recommends for sha pinning.
Mirror the wpt/e2e-integration pattern: tee the suite output to a log,
strip the ANSI colors, and upload it to the #ci-agent channel
(AGENT_SLACK_CHANNEL_ID) via the CI slack bot, with the final
pass/fail summary as the comment. The post also happens when the
suite fails, without masking its exit code (explicit bash pipefail).
Add a nightly cron trigger (03:37, offset from wpt's 02:21) so the
suite actually runs unattended; workflow_dispatch stays for manual
runs with a model override.
The suite moved to the demo repo (machinery in agent/, targets the
existing public/ sites — lightpanda-io/demo#206). Delete the in-repo
copy and point the agent-deterministic job, agent-regression.yml, and
make test-agent at the demo checkout, passing the binary via LPD_PATH
(the same variable wptrunner uses) — same consumption model as
runner/integration/wpt.
Includes a temporary push trigger on agent-regression.yml to validate
the cross-repo wiring pre-merge; dropped before merging.
Two layers driven by test/agent/run.sh:
- deterministic: replay a golden PandaScript against frozen local HN
fixtures and diff the returned JSON exactly against a golden file.
No API key, no network; gates every PR via e2e-test.yml.
- live (GOOGLE_API_KEY): closed-form Q&A over a local fixture page
(substring match), and a live Hacker News scrape saved with --save,
replayed token-free and validated against a jq shape invariant.
Runs nightly and on demand via agent-regression.yml.
The live output contract lives in cases/hn-live.task (prompt) +
schemas/hn-live.jq (invariant). Model pinned to gemini-3.5-flash in
run.sh; per-task timeouts and a $usage token ceiling guard runaway
loops. make test-agent for local runs.
