Our redirect handling was "optimized" to re-use the same easy connection, at the
cost of circumventing the entire pipeline. A redirect would not check the new
target's robots.txt, rate limit, CORS, Web Auth, ...
The new code simply restarts the pipeline (with a discriminator tag so that
things like CDP can tell the difference between a redirected request and a new
one).
Add an update-python-package job that creates the matching release on
lightpanda-io/lightpanda-python (same distribution-app pattern as the
docker and homebrew jobs). That release triggers the wheels workflow
there, replacing its daily track-browser polling; its publish to PyPI
still waits for approval on that repo's pypi environment.
The distribution app needs contents write access to lightpanda-python.
In order to support Selenium the way people are used to, it looks like we need
to support both WebDriver classic (WebDriver) and WebDriver BiDi (BiDi). Typical
scripts look like a mix of the two, e.g. using WebDriver to control the browser
and using BiDi to receive notifications. This commit:
1 - adds a --protocol (cdp|webdriver) CLI argument to the `serve` command to
enable one or the other protocol (defaulting to CDP)
2 - adds basic WebDriver endpoint to let a Selenium client connect. This
implementation is hackish and sits on top of our simple Handshake handler.
The handshake handler is well past its original design. Serving /json/version
and /metrics from it was one thing. But Driving the entire browser session? This
will get a follow up PR.
By default, iframes and workers no longer loaded. Use `--load-resources iframe`
and `--load-resources worker` to restore the previous behavior. The disabling
makes resource loading more consistent.
To further make things more consistent, Config seems the following changes:
1. remove `--timeout` from `serve` which does nothing but has printed a
deprecation warning for a long time
2. added .deprecated field to CLI config flags which now logs the specified
deprecated warning when used
3. `--log-filter-scopes` is deprecated in favor of `--log-scopes`
4. `--disable_subframes` is deprecated. Iframe loading is disabled by default,
use `--load-resources iframe` to enable iframe loading
5. `--disable_workers` is deprecated. Worker loading is disabled by default,
use `--load-resources worker` to enable worker loading
6. `--enable_external_stylesheets` is deprecated. Stylesheets remain disabled
by default. Use `--load-resources stylesheet` to enable loading external
stylesheets
CLI log parameters now alter the logger behavior on parse. This helps minimze
the window where default log settings are in-play. It also means things like
this work:
```
./lightpanda --log-level fatal --disable_subframes --log-level warn
```
More seriously, there's now an optional `beforeParse` fired once the mode is
known. This is used by mcp to set the default log level to logfmt. Previously
this was done much later and could easily result in a mix of pretty and logfmt
logs.
Mimic's https://github.com/lightpanda-io/browser/pull/3294
I looked into this a little bit, and the main issue is that we build the binary
in releasefast twice: once for the snapshot and once for the browser itself.
Each build takes ~10 minutes.
The snapshot cache almost always misses (since any change to webapi/*
invalidates it). I don't see a great solution.
CI build, for e2e-test and nightly now build with:
-Dorderfile/lightpanda.ld
This file informs the build on how to organize the code in the binary, grouping
hot code together so that we have to load less of the binary into memory.
lightpanda.ld will drift: we'll refactor our code, add new features, update
dependencies, update Zig, ... So it has to be re-generated. But we can do that
automatically in the CI (say, before the nightly build). That's for a follow up
PR.
This does not currently cover V8. V8 is being build with
`-no-unique-section-names`, so we don't get names that we can correctly
organize. The real win comes from doing this in V8, since a lot of V8 is cold.
This PR can land as-is, a zig-v8-fork PR will remove that flag, and then we can
have a follow up PR with an lightpanda.ld that includes the v8 symbols.
This is opt-in (via the -Dorderfile flag) because it adds ~20 seconds of
linking time.
Some WPT tests just take more memory, e.g. shadow-dom/declarative/gethtml.html.
Also left some placeholder comments for profiling JS (which I used to make sure
this was just v8 memory usage). These placeholders used to exist but didn't
survive the session multi-page refactor (figured I'd fix it next time I needed
it, and now I needed it).
Runs on pushes to main and on pull requests. The config starts from
the generated defaults and is scoped to this repo: the few controls
that are off carry a note explaining why and what would let them
come back on.
Reports are uploaded to code scanning, except on PRs from forks
where the token cannot write security events. Each run on main also
publishes the score through OIDC, which feeds the live badge added
to the README next to the existing ones.
Mutable tags like @v6 can be re-pointed by whoever controls the action,
so the next run executes code nobody here reviewed, with the job's
token and secrets. Pinning to the full commit sha freezes what runs.
Version tags are kept as comments so renovate or dependabot can still
track updates.
dtolnay/rust-toolchain is pinned to a master commit with the toolchain
moved to an explicit input, as its readme recommends for sha pinning.
Mirror the wpt/e2e-integration pattern: tee the suite output to a log,
strip the ANSI colors, and upload it to the #ci-agent channel
(AGENT_SLACK_CHANNEL_ID) via the CI slack bot, with the final
pass/fail summary as the comment. The post also happens when the
suite fails, without masking its exit code (explicit bash pipefail).
Add a nightly cron trigger (03:37, offset from wpt's 02:21) so the
suite actually runs unattended; workflow_dispatch stays for manual
runs with a model override.
The suite moved to the demo repo (machinery in agent/, targets the
existing public/ sites — lightpanda-io/demo#206). Delete the in-repo
copy and point the agent-deterministic job, agent-regression.yml, and
make test-agent at the demo checkout, passing the binary via LPD_PATH
(the same variable wptrunner uses) — same consumption model as
runner/integration/wpt.
Includes a temporary push trigger on agent-regression.yml to validate
the cross-repo wiring pre-merge; dropped before merging.
Two layers driven by test/agent/run.sh:
- deterministic: replay a golden PandaScript against frozen local HN
fixtures and diff the returned JSON exactly against a golden file.
No API key, no network; gates every PR via e2e-test.yml.
- live (GOOGLE_API_KEY): closed-form Q&A over a local fixture page
(substring match), and a live Hacker News scrape saved with --save,
replayed token-free and validated against a jq shape invariant.
Runs nightly and on demand via agent-regression.yml.
The live output contract lives in cases/hn-live.task (prompt) +
schemas/hn-live.jq (invariant). Model pinned to gemini-3.5-flash in
run.sh; per-task timeouts and a $usage token ceiling guard runaway
loops. make test-agent for local runs.
Replace `grep '"id":N' | jq -e ...` with `jq -ec 'select(.id == N) | ...'`.
The grep form also matched `"id":10`, `"id":11`, ... and any tool description
containing that substring; numeric `select` is type-correct. `jq -e` still
fails the job when `select` produces no output (exit 4), so the smoke
semantics are preserved.
Also add `jq --version` up front so the job fails fast and loud if the
`ubuntu-latest` image ever stops shipping jq.
Adrià Arrufat