Runs on pushes to main and on pull requests. The config starts from
the generated defaults and is scoped to this repo: the few controls
that are off carry a note explaining why and what would let them
come back on.
Reports are uploaded to code scanning, except on PRs from forks
where the token cannot write security events. Each run on main also
publishes the score through OIDC, which feeds the live badge added
to the README next to the existing ones.
Mutable tags like @v6 can be re-pointed by whoever controls the action,
so the next run executes code nobody here reviewed, with the job's
token and secrets. Pinning to the full commit sha freezes what runs.
Version tags are kept as comments so renovate or dependabot can still
track updates.
dtolnay/rust-toolchain is pinned to a master commit with the toolchain
moved to an explicit input, as its readme recommends for sha pinning.
Mirror the wpt/e2e-integration pattern: tee the suite output to a log,
strip the ANSI colors, and upload it to the #ci-agent channel
(AGENT_SLACK_CHANNEL_ID) via the CI slack bot, with the final
pass/fail summary as the comment. The post also happens when the
suite fails, without masking its exit code (explicit bash pipefail).
Add a nightly cron trigger (03:37, offset from wpt's 02:21) so the
suite actually runs unattended; workflow_dispatch stays for manual
runs with a model override.
The suite moved to the demo repo (machinery in agent/, targets the
existing public/ sites — lightpanda-io/demo#206). Delete the in-repo
copy and point the agent-deterministic job, agent-regression.yml, and
make test-agent at the demo checkout, passing the binary via LPD_PATH
(the same variable wptrunner uses) — same consumption model as
runner/integration/wpt.
Includes a temporary push trigger on agent-regression.yml to validate
the cross-repo wiring pre-merge; dropped before merging.
Two layers driven by test/agent/run.sh:
- deterministic: replay a golden PandaScript against frozen local HN
fixtures and diff the returned JSON exactly against a golden file.
No API key, no network; gates every PR via e2e-test.yml.
- live (GOOGLE_API_KEY): closed-form Q&A over a local fixture page
(substring match), and a live Hacker News scrape saved with --save,
replayed token-free and validated against a jq shape invariant.
Runs nightly and on demand via agent-regression.yml.
The live output contract lives in cases/hn-live.task (prompt) +
schemas/hn-live.jq (invariant). Model pinned to gemini-3.5-flash in
run.sh; per-task timeouts and a $usage token ceiling guard runaway
loops. make test-agent for local runs.
Replace `grep '"id":N' | jq -e ...` with `jq -ec 'select(.id == N) | ...'`.
The grep form also matched `"id":10`, `"id":11`, ... and any tool description
containing that substring; numeric `select` is type-correct. `jq -e` still
fails the job when `select` produces no output (exit 4), so the smoke
semantics are preserved.
Also add `jq --version` up front so the job fails fast and loud if the
`ubuntu-latest` image ever stops shipping jq.
Sends initialize + notifications/initialized + tools/list over stdin
and asserts the JSON-RPC responses with jq. Catches regressions in
the agentic surface (./lightpanda mcp) without needing a node client.
Reuses the existing lightpanda-build-release artifact, so the new
job costs about a minute on top of zig-build-release.
cache=true is problematic for a few reasons
1 - The current cache implementation is known to cause timing issues; i.e. it
executes callbacks synchronously.
2 - Unlike something like robots.txt or proxy, cache tests need to be explicitly
tested. The response has to include cache headers and the resource loaded
again.
Don't use pre-built snapshot for wpt action (the snapshot needs the WebDriver
added, which is not normally added). Instead, rely on runtime snapshot creation
(this just adds a bit of startup time and peak memory, neither or which should
be an issue for WPT runs).
Some WPT tests need to interact with the browser in a way that isn't possible
with web apis. Browsers need to expose a way for tests to do this and then use
the testdriver-vendor.js to hook into these special WPT actions.
This commit sets up the infrastructure for supporting this and includes
the delete_all_cookies functionality needed by various cookie tests (e.g.
/cookies/attributes/attributes-ctl.sub.html).
A new compilation flag, `-Dwpt_extensions`, can be specified. When specified
a `window.webdriver` accessor is defined and a `WebDriver` type is exposed.
Note that, while I only implemented delete_all_cookies for now, I've seen other
tests fail because of missing vendor-specific implementation.
