1 - TEST_VERBOSE is now off by default
2 - There's a afterEach callback that is automatically run after each tests, it:
a - clears the log filters
b - resets the test arena
3 - LogFilter replace with
a - testing.silenceLog(&.{...scopes...}); to silence all logs for the given
scopes.
b - testing.expectLog(&.{...scopes}); to set log expectations, 1 per log.
The goal here isn't so much to expect logs (though, you can do that),
but rather to silence an expected # of logs, without silencing more.
BodyInit.extract returned an empty body for ReadableStream request
bodies, so fetch POST with duplex:half silently sent Content-Length: 0.
Drain closed streams synchronously via collectBodyBytes; reject open,
locked, or errored streams with TypeError instead of sending nothing.
Fixeslightpanda-io/browser#3052
Expand mime type to be aware of more XML types. DOMParser and XHR now use the
existing Parser.parseXML (via a Frame helper), rather than the HTML parser. Both
these APIs predate the addition of parseXML.
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
Builds on top of the recently added support for <link rel=preload...> and
<link rel=modulePreload...> to scan the HTML for script tags to preload. I.e.
adds script preloading without actually having any preload hits.
At least for this first pass, I opted for a simple approach which leverages
are fully buffered HTML body and html5ever's tokenizer to prescan the body and
kickoff any script fetching before starting the complete parse.
There are doubtless cases where this will either decrease performance and/or
increase memory usage. E.g. a site with no script gets its html scanned twice
and loading multiple blocking scripts in parallel obvious uses more memory
than loading them sequentially. But for most sites and I think most use-cases,
the impact should range between neutral to significantly faster loads.
This is something most browsers do.
This fixes a bug with preload script so that they're actually used for
async/defer scripts (previously, only used for blocking scripts).
More importantly, this cleans up the ScriptManager's addFromElement. For example
inline scripts are handled in their own function, which means we aren't weaving
the two modes in a single function. It also allows the inline-script to use
a better-sized arena. The goal for this cleanup is the follow up commit which
will bring a pre-parse step to preload scripts without an explicit <link> hint.
The EventSource API only requires a simple line-based parser. It's pretty
straightforward. It's RC'd by v8 and itself, but also by scheduled tasks (e.g.
to reconnect).
The bigger change is to the HttpClient to support streaming requests. This
changes to things:
1 - A transfer doesn't only deliver() once
2 - A transfer can be in the multi while delivering
Neither of these are surprising, but they both add complexity.
When an iframe has "loading=lazy" it no longer delays the parent's "load" event.
It's still loaded (subject to `--disable-subframes` parameter).
Many sites use loading=lazy for non-critical content (e.g. ads). On Chrome/
Firefox, these are only loaded when they're in the viewport, so sites usually
can't depend/rely on them being loaded.
Frame.loadExternalStylesheet fetches an external <link rel=stylesheet>
synchronously, which registers a blocking request for the frame. While that
blocking request is active the DeferringLayer holds back the completion
callbacks of every other in-flight transfer for the frame, so they don't run
JS on the parser stack. The blocking-<script> path and the worker path flush
those deferred completions once their synchronous fetch returns, but the
external-stylesheet path did not. As a result a <script defer> that finished
loading during the stylesheet's blocking window stayed at complete == false:
the deferred-script queue never drained, so the deferred scripts never ran and
DOMContentLoaded / the load event / readyState -> "complete" never fired —
the document was stuck at readyState "loading" even though every request
completed with HTTP 200.
Flush the frame's deferring layer after the synchronous stylesheet fetch,
mirroring the other two synchronous-request call sites.
Closes#2842
Follow up to https://github.com/lightpanda-io/browser/pull/2789 which adds
better multi-page support for runner. When waiting, callers have the choice
to wait for a specific frame, all current frames, or a given list of frames. The
last one, a given list of frames, is the most flexible, allowing callers to
provide an `until` per frame AND receive a per-frame result.
waitForSelector and waitForScript continue to be per frame (which is now
explicitly given).
Like 2789, while the changes aren't insignificant (Runner is doing important
work), a number of files were touched either purely because of changes in tests
or other superficial changes, e.g. `session.runner` is now infallible.
Replaces the 1 (+1 inflight) page design of session for an unlimited number of
pages. The main goal is to support a more efficient async goto (1). Without this
commit, async goto has two implementation: multiple browser, which is resource
intensive (thread + isolate per page), bolting it onto sub-frames (like iframe
or popups). The issue with the 2nd version (which I originally suggested) is
that most resources are tied to the Page. So even if an async goto "page" (which
would map to a Frame) is released, most things stay in memory, including the
DOM (page.factory) and the V8::Context.
This new approach adds multiple page support to sessions. The advantage is
pretty clear: the existing memory model (page-tied resources) becomes a strength
of the design, rather than a weakness.
This change is not tirivla, but the diff is inflated by 2 large mechanial
changes, so it isn't _that_ big either. That said, I'd divide this into four
parts.
1 - The old concept of _active/_pending is now baked into the Page itself. A
Page has a `replaces: ?*Page = null` and `replacement: ?*Page = null` field.
2 - Because of #1 above, the Session now just has an `pages: ArrayList(*Page)`.
As much as possible, single-page APIs, like `removePage` no longer exists. We're
trying to present a consistent multi-page API. See #3.
3 - References to *Page and *Frame have always been dangerous. For example,
`lightpanda.fetch` has a block to scope `frame`:
```zig
{
const frame = try session.createPage();
// frame isn't safe to use after navigate, it can be swapped out
_ = try frame.navigate(...)
}
```
While we still hand out *Page and *Frame (user-beware), some of the more
important APIs now take a frame_id which Session can resolve. Furthermore,
createPage now returns a PageHandle which is a safe wrapper around a page/frame.
4 - Two large mechanical changes were made:
a - Many tests were superficially changed to account for new naming or use
a new test-helper to preserve the single-page illusion (because 99% of
tests _are_ single-page)
b - Many CDP changes where `bc.session.currentFrame()` -> `bc.mainFrame()`).
This isn't to say CDP changes are meaningless, but it's mostly 1 change
about how the "main" target/frame_id is tracked (by CDP itself, rather
than the Session) that required a number of superficial changes to
accomodate
`Runner` remains largely single-page focused. Runner and some MCP/Agent tools
continue to be tied to the "currentFrame" or "primaryPage". The Runner is
something I want to address in a follow-up PR, but we need to figure out what
it means to "wait" for multiple pages.
As much as possible, Session becomes multi-page native and has no concept of
a special/first/primary/main page. Users of session become responsible for
tracking pages of interest.
This change is not trivial, but the diff is inflated by 2 large mechanical
changes:
1 - Many tests
(1) https://github.com/lightpanda-io/browser/pull/2759
Browser.setDownloadBehavior was a noop, so Lightpanda had no file-download
path. A response with Content-Disposition: attachment is now streamed to disk
under downloadPath, and Page.downloadWillBegin / Browser.downloadProgress are
emitted when eventsEnabled.
Fixes#2701
`--log-filter-scopes` previously took a comma-separated list of scopes to
suppress. Extend it with explicit include/exclude directives applied
left-to-right:
-X filter out scope X
X alias for -X (backward compatible)
+X filter in scope X
all target every scope
This makes "filter everything except one scope" expressible, e.g.
`-all,+cdp` suppresses all logs except `cdp`. Bare scope names keep their
old meaning, so existing invocations like `http,unknown_prop` are
unaffected.
Internally, the suppressed-scope list is replaced by a resolved per-scope
`scope_enabled` boolean array built via `log.resolveFilterScopes()`, so the
hot-path `enabled()` check is a single array index. Filtering remains
Debug-only, as before. `testing.LogFilter` is adapted to the new
representation; all existing call sites are unchanged.
Adds a unit test for resolveFilterScopes (default, backward-compat,
-all,+cdp, ordering) and updates the help text.
Every 300-399 status was routed into the redirect path, where the missing
Location header fails the transfer with error.LocationNotFound — the old
document is destroyed and nothing replaces it. Per the fetch standard's
HTTP-redirect fetch ("If locationURL is null, then return response") and
RFC 9110 §15.4, a 3xx without Location is a normal final response whose
body must be delivered. Guard both the redirect dispatch and the 3xx
skip-body branch on the header's presence so such responses fall through
to the regular completion path.
Closes#2713
We currently skip capturing 401 and 407 bodies. This appears to be an
optimization with the intent that it won't be needed. While that might be true
in some cases (though, not sure when), it isn't always true. A page.navigate
to a 401 should display the content.
(Also, tried to silence meaningless BrokenPipe noise in tests).
Caught in code review: `loadExternalStylesheet` created a fresh
`CSSStyleSheet` and appended to `document.styleSheets` on every call, so
mutating `link.href` on a connected stylesheet element accumulated stale
sheets — the old rules kept cascading because the previous sheet was
never removed.
Cache the sheet on `Link._sheet` (mirroring `Style._sheet`) and reuse it
via `replaceSync` on re-fetch. First load creates + registers as before;
subsequent loads swap content in place, keeping `document.styleSheets`
length stable.
On fetch failure the cached sheet is untouched — matches browser
behavior where a broken href doesn't invalidate the previously loaded
sheet until the link itself is removed.
Refs #2343
Wires up --enable-external-stylesheets / LP.configureLoading.externalStylesheets
from the prior surface-only commit. When the flag is set, parser- and
JS-created <link rel=stylesheet> elements now synchronously fetch and parse
their href, register a CSSStyleSheet on document.styleSheets, and feed
StyleManager so checkVisibility() reflects external rules. Flag stays
default-off — scrapers that don't need accurate visibility pay nothing.
Frame.loadExternalStylesheet mirrors ScriptManager.addFromElement: same
HttpClient.syncRequest path, same arena ownership, same per-frame
notification + cookie wiring. Body is routed through CSSStyleSheet.replaceSync,
which already parses, populates cssRules, and calls sheetModified() — no
StyleManager changes needed. 2 MiB hard cap on a single sheet body, status
non-2xx and oversize both fire `error` on the link.
Link.Build.created is added so static head <link> elements reach
linkAddedCallback at all — void elements never trigger nodeComplete, which
is why static `<link>` had no observable effect before. Mirrors Image.
HttpClient.Request.ResourceType gains a `.stylesheet` variant so CDP Network
events report the right type; cdp.fetch.zig switches updated.
Refs #2343
At some point recently, we started to process scripts that fail to load (e.g.
404). This stops such scripts from [trying] to be evaluated, and executes the
onerror handler in all script loading paths.
Anchor click, form submit, and `location.href = ...` assignments queue a
navigation through `Frame.scheduleNavigation`, which then tears down the
originating page and rebuilds the frame in `Session.processRootQueuedNavigation`
before `Frame.navigate` issues the HTTP request. The originator's URL was
discarded with the old arena, so the request went out without a Referer
header — even though the HTML "navigate" algorithm and Fetch §4.5 require
one. `Frame.headersForRequest` (#1449) handled subresource fetches but was
never called from the navigation path.
Capture the originating frame's URL into a new `referer` field on
`NavigateOpts` at scheduling time, dup'd into the `QueuedNavigation` arena
so it survives the page tear-down. `Frame.navigate` adds it as a
`Referer:` header alongside the existing per-request headers. Iframe
initial navigation (`Frame.zig:1282`) also sets `referer = parent.url`
since the parent frame outlives that direct `navigate` call. CDP
`Page.navigate` (`.reason = .address_bar`) and `Page.reload` continue to
omit Referer — matches Chrome.
Closes#2281
If a POST navigation gets redirected (302/303), the page that actually
loads is fetched with GET — but Frame._navigated_options still carries
the original POST method, body, and Content-Type. Page.reload would
then re-POST the form data to the redirect target, which is both
incorrect and dangerous (re-submission of credentials, charges, etc.).
Reset method/body/header on _navigated_options inside
frameHeaderDoneCallback whenever response.redirectCount() > 0. The full
spec-correct version would distinguish 307/308 (preserve method) from
301/302/303 (convert POST→GET), but resubmitting form data is the more
dangerous failure mode — conservative reset matches Chrome's practical
behavior on reload.
Also collapse the prev_body/prev_header extraction in doReload to a
single tuple-destructured blk: block (no behavior change).
Tests: new cdp.frame: reload after POST→redirect drops the POST drives
POST /redirect_to_echo → 302 → /echo_method, then Page.reload, asserts
the second request is GET. /redirect_to_echo route added to
testing.zig. The existing reload-replays-POST test still passes (no
redirect, POST is still replayed).
Per RFC 7231 §7.1.2, when a 3xx response carries a Location header
without a fragment, the user agent must process the redirect as if
the value inherited the fragment of the request URL. URL.resolve
follows RFC 3986 §5.3 which drops the base fragment, so handleRedirect
now reattaches the original fragment when the resolved target has none.
Closes#2263
doReload built a NavigateOpts with only url + kind=.reload; method/body/header
defaulted to GET/null/null, so any prior POST navigation regressed to a GET
on reload. The HTML reload navigation re-fetches the document that produced
the current entry, and Chrome replays the same HTTP request that loaded the
page (including method, body, and Content-Type) — Lightpanda dropped all
three.
Retain the prior request body and content-type header in Frame.NavigatedOpts
(duped into the frame arena), and have doReload capture them into the CDP
command's arena before replacePage() frees the old frame. The reload's
frame.navigate call carries the replayed method/body/header so the request
the page was loaded with is the request that runs again.
Closes#2258
Follow up to https://github.com/lightpanda-io/browser/pull/2200
This change is actually pretty mundane, but a bunch of files that used to
take a *Session (e.g. every WebAPI releaseRef and deinit) now take a *Page.
This aims to separate the 2 lifetimes currently managed by Session by moving
the "Page" lifetime to a dedicated container: Page. Ultimately, the goal is to
remove the 1-page-per-session limit of the current design. Not to explicitly
support multiple pages per session (though, that's more possible now), but
in order to better emulate Chrome where, during a navigation event, the old and
new page both exist.
This is to pave the way for introducing a new "Page" container, which will take
over the page lifecycle currently burdening Session. The ultimate goal of that
is to allow the Session to have multiple pages (mostly for better transitions
between pages), which is hard to do now since the Session has so much state.
This rename was aggressive, e.g. currentPage() -> currentFrame() so that, when
the new Page container is added, you won't see "currentPage()" and wonder:
"Does 'currentPage' mean the new Page container, or the Frame (which
used to be called Page)".
@import("lightpanda") where needed.
Would also like to do this for String, Page, Session and js which all stand out
as types that are use across the codebase.
I know that a few devs are doing this in new work and I haven't heard anyone
voice an objection.
This commit fixes a few serious issues with the Websocket implementation.
1 - libcurl recursive api calls
Creating a Websocket instance from within a libcurl callback results in libcurl
failing with a RecursiveApiCall error. I fixed this more generally by adding a
`ready_queue` which connections can use when the `HttpClient` is performing
actions. Once `perform` ends, this new `ready_queue` is processed. There might
be a more holistic solution to this (we seem to run into RecursiveApiCall
everywhere), but since HttpClient is going through heavy changes, this seemed
like the smallest possible change to fix it.
2 - "load" blocking
Load and IdleNetwork notifications should not block on Websocket connections. To
solve this, `HttpClient` now ha `http_active` and `ws_active` to replace `active`.
Only `http_active` is used for things like "load" triggering.
3 - The above change made the Runner's job more complicated. It used to be
binary: you either have active connections or not. Now there are different types
of active connections. To keep it simple, and I think probably more correct,
the "done-ness" (based on the `wait` parameter) is now independent of active
(or not) network activity. If the page's `load_state == .complete`, then the
`wait == .done` is considered successful, whether or not we have active
connections.
4 - As a consequence of the above, and seemingly unrelated to all of these
changes, a number of html tests now use the "new" robust async framework. Most
of these tests were using the `testing.onload` (aka `testing.eventually`) which
had somewhat...unclear semantics. These tests passed more of a consequence of
how we processed a page and being very simple (e.g. just needing 1 micro or
macrotask tick). But `eventually` never worked for more complicated cases, and
the previous `testing.async` didn't work well. Now, the test runner waits for
.load (which, as per #3, can fire more aggressively), which caused many
`eventually` tests to fail. Moving these tests to the new `async` is more
robust and works with the new aggressive "load".
Uses libcurl's websocket capabilities to add support for WebSocket.
Depends on https://github.com/lightpanda-io/zig-v8-fork/pull/167
Issue: https://github.com/lightpanda-io/browser/issues/1952
This is a WIP because it currently uses the same connection pool used for all
HTTP requests. It would be pretty easy for a page to starve the pool and block
any progress.
We previously stored the *Transfer inside of the easy's private data. We now
store the *Connection, and a Connection now has a `transport` field which is
a union for `http: *Transfer` or `websocket: *Websocket`.
These new optional parameter run AFTER --wait-until, allowing the (imo) useful
combination of `--wait-until load --wait-script "report.complete === true"`.
However, if `--wait-until` IS NOT specified but `--wait-selector/script` IS,
then there is no default wait and it'll just check the selector/script. If
neither `--wait-selector` or `--wait-script/--wait-script-file` are specified
then `--wait-until` continues to default to `done`.
These waiters were added to the Runner, and the existing Action.waitForSelector
now uses the runner's version. Selector querying has been split into distinct
parse and query functions, so that we can parse once, and query on every tick.
We could potentially optimize --wait-script to compile the script once and call
it on each tick, but we'd have to detect page navigation to recompile the script
in the new context. Something I'd rather optimize separately.
testing.async(...) is pretty lame. It works for simple cases, where the
microtask is very quickly resolved, but otherwise can't block the test from
exiting.
This adds an overload to testing.async and leverages the new Runner
https://github.com/lightpanda-io/browser/pull/1958 to "tick" until completion
(or timeout).
The overloaded version of testing.async() (called without a callback) will
increment a counter which is only decremented with the promise is resolved. The
test runner will now `tick` until the counter == 0.
CDPT used to be a generic so that we could inject Browser, Session, Page and
Client. At some point, it [thankfully] became a generic only to inject Client.
This commit removes the generic and bakes the *Server.Client instance in CDP.
It uses a socketpair for testing.
BrowserContext is still generic, but that's generic for a very different reason
and, while I'd like to remove that generic too, it belongs in a different PR.
This is done for a couple reasons. The first is just to have things a little
more self-contained for eventually supporting more advanced "wait" logic, e.g.
waiting for a selector.
The other is to provide callers with more fine-grained controlled. Specifically
the ability to manually "tick", so that they can [presumably] do something
after every tick. This is needed by the test runner to support more advanced
cases (cases that need to test beyond 'load') and it also improves (and fixes
potential use-after-free, the lp.waitForSelector)
Small tweaks to https://github.com/lightpanda-io/browser/pull/1896
Improve the wait ergonomics with an Option with default parameter. Revert
page pointer logic to original (don't think that change was necessary).
Add `--wait_until` and `--wait_ms` CLI arguments to configure session wait behavior. Updates `Session.wait` to evaluate specific page load states (`load`, `domcontentloaded`, `networkidle`, `fixed`) before completing the wait loop.
Karl Seguin