Mirror the wpt/e2e-integration pattern: tee the suite output to a log,
strip the ANSI colors, and upload it to the #ci-agent channel
(AGENT_SLACK_CHANNEL_ID) via the CI slack bot, with the final
pass/fail summary as the comment. The post also happens when the
suite fails, without masking its exit code (explicit bash pipefail).
Add a nightly cron trigger (03:37, offset from wpt's 02:21) so the
suite actually runs unattended; workflow_dispatch stays for manual
runs with a model override.
The suite moved to the demo repo (machinery in agent/, targets the
existing public/ sites — lightpanda-io/demo#206). Delete the in-repo
copy and point the agent-deterministic job, agent-regression.yml, and
make test-agent at the demo checkout, passing the binary via LPD_PATH
(the same variable wptrunner uses) — same consumption model as
runner/integration/wpt.
Includes a temporary push trigger on agent-regression.yml to validate
the cross-repo wiring pre-merge; dropped before merging.
Two layers driven by test/agent/run.sh:
- deterministic: replay a golden PandaScript against frozen local HN
fixtures and diff the returned JSON exactly against a golden file.
No API key, no network; gates every PR via e2e-test.yml.
- live (GOOGLE_API_KEY): closed-form Q&A over a local fixture page
(substring match), and a live Hacker News scrape saved with --save,
replayed token-free and validated against a jq shape invariant.
Runs nightly and on demand via agent-regression.yml.
The live output contract lives in cases/hn-live.task (prompt) +
schemas/hn-live.jq (invariant). Model pinned to gemini-3.5-flash in
run.sh; per-task timeouts and a $usage token ceiling guard runaway
loops. make test-agent for local runs.
Replace `grep '"id":N' | jq -e ...` with `jq -ec 'select(.id == N) | ...'`.
The grep form also matched `"id":10`, `"id":11`, ... and any tool description
containing that substring; numeric `select` is type-correct. `jq -e` still
fails the job when `select` produces no output (exit 4), so the smoke
semantics are preserved.
Also add `jq --version` up front so the job fails fast and loud if the
`ubuntu-latest` image ever stops shipping jq.
Sends initialize + notifications/initialized + tools/list over stdin
and asserts the JSON-RPC responses with jq. Catches regressions in
the agentic surface (./lightpanda mcp) without needing a node client.
Reuses the existing lightpanda-build-release artifact, so the new
job costs about a minute on top of zig-build-release.
cache=true is problematic for a few reasons
1 - The current cache implementation is known to cause timing issues; i.e. it
executes callbacks synchronously.
2 - Unlike something like robots.txt or proxy, cache tests need to be explicitly
tested. The response has to include cache headers and the resource loaded
again.
Don't use pre-built snapshot for wpt action (the snapshot needs the WebDriver
added, which is not normally added). Instead, rely on runtime snapshot creation
(this just adds a bit of startup time and peak memory, neither or which should
be an issue for WPT runs).
Some WPT tests need to interact with the browser in a way that isn't possible
with web apis. Browsers need to expose a way for tests to do this and then use
the testdriver-vendor.js to hook into these special WPT actions.
This commit sets up the infrastructure for supporting this and includes
the delete_all_cookies functionality needed by various cookie tests (e.g.
/cookies/attributes/attributes-ctl.sub.html).
A new compilation flag, `-Dwpt_extensions`, can be specified. When specified
a `window.webdriver` accessor is defined and a `WebDriver` type is exposed.
Note that, while I only implemented delete_all_cookies for now, I've seen other
tests fail because of missing vendor-specific implementation.
