Frame.loadExternalStylesheet fetches an external <link rel=stylesheet>
synchronously, which registers a blocking request for the frame. While that
blocking request is active the DeferringLayer holds back the completion
callbacks of every other in-flight transfer for the frame, so they don't run
JS on the parser stack. The blocking-<script> path and the worker path flush
those deferred completions once their synchronous fetch returns, but the
external-stylesheet path did not. As a result a <script defer> that finished
loading during the stylesheet's blocking window stayed at complete == false:
the deferred-script queue never drained, so the deferred scripts never ran and
DOMContentLoaded / the load event / readyState -> "complete" never fired —
the document was stuck at readyState "loading" even though every request
completed with HTTP 200.
Flush the frame's deferring layer after the synchronous stylesheet fetch,
mirroring the other two synchronous-request call sites.
Closes#2842
Follow up to https://github.com/lightpanda-io/browser/pull/2789 which adds
better multi-page support for runner. When waiting, callers have the choice
to wait for a specific frame, all current frames, or a given list of frames. The
last one, a given list of frames, is the most flexible, allowing callers to
provide an `until` per frame AND receive a per-frame result.
waitForSelector and waitForScript continue to be per frame (which is now
explicitly given).
Like 2789, while the changes aren't insignificant (Runner is doing important
work), a number of files were touched either purely because of changes in tests
or other superficial changes, e.g. `session.runner` is now infallible.
Replaces the 1 (+1 inflight) page design of session for an unlimited number of
pages. The main goal is to support a more efficient async goto (1). Without this
commit, async goto has two implementation: multiple browser, which is resource
intensive (thread + isolate per page), bolting it onto sub-frames (like iframe
or popups). The issue with the 2nd version (which I originally suggested) is
that most resources are tied to the Page. So even if an async goto "page" (which
would map to a Frame) is released, most things stay in memory, including the
DOM (page.factory) and the V8::Context.
This new approach adds multiple page support to sessions. The advantage is
pretty clear: the existing memory model (page-tied resources) becomes a strength
of the design, rather than a weakness.
This change is not tirivla, but the diff is inflated by 2 large mechanial
changes, so it isn't _that_ big either. That said, I'd divide this into four
parts.
1 - The old concept of _active/_pending is now baked into the Page itself. A
Page has a `replaces: ?*Page = null` and `replacement: ?*Page = null` field.
2 - Because of #1 above, the Session now just has an `pages: ArrayList(*Page)`.
As much as possible, single-page APIs, like `removePage` no longer exists. We're
trying to present a consistent multi-page API. See #3.
3 - References to *Page and *Frame have always been dangerous. For example,
`lightpanda.fetch` has a block to scope `frame`:
```zig
{
const frame = try session.createPage();
// frame isn't safe to use after navigate, it can be swapped out
_ = try frame.navigate(...)
}
```
While we still hand out *Page and *Frame (user-beware), some of the more
important APIs now take a frame_id which Session can resolve. Furthermore,
createPage now returns a PageHandle which is a safe wrapper around a page/frame.
4 - Two large mechanical changes were made:
a - Many tests were superficially changed to account for new naming or use
a new test-helper to preserve the single-page illusion (because 99% of
tests _are_ single-page)
b - Many CDP changes where `bc.session.currentFrame()` -> `bc.mainFrame()`).
This isn't to say CDP changes are meaningless, but it's mostly 1 change
about how the "main" target/frame_id is tracked (by CDP itself, rather
than the Session) that required a number of superficial changes to
accomodate
`Runner` remains largely single-page focused. Runner and some MCP/Agent tools
continue to be tied to the "currentFrame" or "primaryPage". The Runner is
something I want to address in a follow-up PR, but we need to figure out what
it means to "wait" for multiple pages.
As much as possible, Session becomes multi-page native and has no concept of
a special/first/primary/main page. Users of session become responsible for
tracking pages of interest.
This change is not trivial, but the diff is inflated by 2 large mechanical
changes:
1 - Many tests
(1) https://github.com/lightpanda-io/browser/pull/2759
Browser.setDownloadBehavior was a noop, so Lightpanda had no file-download
path. A response with Content-Disposition: attachment is now streamed to disk
under downloadPath, and Page.downloadWillBegin / Browser.downloadProgress are
emitted when eventsEnabled.
Fixes#2701
`--log-filter-scopes` previously took a comma-separated list of scopes to
suppress. Extend it with explicit include/exclude directives applied
left-to-right:
-X filter out scope X
X alias for -X (backward compatible)
+X filter in scope X
all target every scope
This makes "filter everything except one scope" expressible, e.g.
`-all,+cdp` suppresses all logs except `cdp`. Bare scope names keep their
old meaning, so existing invocations like `http,unknown_prop` are
unaffected.
Internally, the suppressed-scope list is replaced by a resolved per-scope
`scope_enabled` boolean array built via `log.resolveFilterScopes()`, so the
hot-path `enabled()` check is a single array index. Filtering remains
Debug-only, as before. `testing.LogFilter` is adapted to the new
representation; all existing call sites are unchanged.
Adds a unit test for resolveFilterScopes (default, backward-compat,
-all,+cdp, ordering) and updates the help text.
Every 300-399 status was routed into the redirect path, where the missing
Location header fails the transfer with error.LocationNotFound — the old
document is destroyed and nothing replaces it. Per the fetch standard's
HTTP-redirect fetch ("If locationURL is null, then return response") and
RFC 9110 §15.4, a 3xx without Location is a normal final response whose
body must be delivered. Guard both the redirect dispatch and the 3xx
skip-body branch on the header's presence so such responses fall through
to the regular completion path.
Closes#2713
We currently skip capturing 401 and 407 bodies. This appears to be an
optimization with the intent that it won't be needed. While that might be true
in some cases (though, not sure when), it isn't always true. A page.navigate
to a 401 should display the content.
(Also, tried to silence meaningless BrokenPipe noise in tests).
Caught in code review: `loadExternalStylesheet` created a fresh
`CSSStyleSheet` and appended to `document.styleSheets` on every call, so
mutating `link.href` on a connected stylesheet element accumulated stale
sheets — the old rules kept cascading because the previous sheet was
never removed.
Cache the sheet on `Link._sheet` (mirroring `Style._sheet`) and reuse it
via `replaceSync` on re-fetch. First load creates + registers as before;
subsequent loads swap content in place, keeping `document.styleSheets`
length stable.
On fetch failure the cached sheet is untouched — matches browser
behavior where a broken href doesn't invalidate the previously loaded
sheet until the link itself is removed.
Refs #2343
Wires up --enable-external-stylesheets / LP.configureLoading.externalStylesheets
from the prior surface-only commit. When the flag is set, parser- and
JS-created <link rel=stylesheet> elements now synchronously fetch and parse
their href, register a CSSStyleSheet on document.styleSheets, and feed
StyleManager so checkVisibility() reflects external rules. Flag stays
default-off — scrapers that don't need accurate visibility pay nothing.
Frame.loadExternalStylesheet mirrors ScriptManager.addFromElement: same
HttpClient.syncRequest path, same arena ownership, same per-frame
notification + cookie wiring. Body is routed through CSSStyleSheet.replaceSync,
which already parses, populates cssRules, and calls sheetModified() — no
StyleManager changes needed. 2 MiB hard cap on a single sheet body, status
non-2xx and oversize both fire `error` on the link.
Link.Build.created is added so static head <link> elements reach
linkAddedCallback at all — void elements never trigger nodeComplete, which
is why static `<link>` had no observable effect before. Mirrors Image.
HttpClient.Request.ResourceType gains a `.stylesheet` variant so CDP Network
events report the right type; cdp.fetch.zig switches updated.
Refs #2343
At some point recently, we started to process scripts that fail to load (e.g.
404). This stops such scripts from [trying] to be evaluated, and executes the
onerror handler in all script loading paths.
Anchor click, form submit, and `location.href = ...` assignments queue a
navigation through `Frame.scheduleNavigation`, which then tears down the
originating page and rebuilds the frame in `Session.processRootQueuedNavigation`
before `Frame.navigate` issues the HTTP request. The originator's URL was
discarded with the old arena, so the request went out without a Referer
header — even though the HTML "navigate" algorithm and Fetch §4.5 require
one. `Frame.headersForRequest` (#1449) handled subresource fetches but was
never called from the navigation path.
Capture the originating frame's URL into a new `referer` field on
`NavigateOpts` at scheduling time, dup'd into the `QueuedNavigation` arena
so it survives the page tear-down. `Frame.navigate` adds it as a
`Referer:` header alongside the existing per-request headers. Iframe
initial navigation (`Frame.zig:1282`) also sets `referer = parent.url`
since the parent frame outlives that direct `navigate` call. CDP
`Page.navigate` (`.reason = .address_bar`) and `Page.reload` continue to
omit Referer — matches Chrome.
Closes#2281
If a POST navigation gets redirected (302/303), the page that actually
loads is fetched with GET — but Frame._navigated_options still carries
the original POST method, body, and Content-Type. Page.reload would
then re-POST the form data to the redirect target, which is both
incorrect and dangerous (re-submission of credentials, charges, etc.).
Reset method/body/header on _navigated_options inside
frameHeaderDoneCallback whenever response.redirectCount() > 0. The full
spec-correct version would distinguish 307/308 (preserve method) from
301/302/303 (convert POST→GET), but resubmitting form data is the more
dangerous failure mode — conservative reset matches Chrome's practical
behavior on reload.
Also collapse the prev_body/prev_header extraction in doReload to a
single tuple-destructured blk: block (no behavior change).
Tests: new cdp.frame: reload after POST→redirect drops the POST drives
POST /redirect_to_echo → 302 → /echo_method, then Page.reload, asserts
the second request is GET. /redirect_to_echo route added to
testing.zig. The existing reload-replays-POST test still passes (no
redirect, POST is still replayed).
Per RFC 7231 §7.1.2, when a 3xx response carries a Location header
without a fragment, the user agent must process the redirect as if
the value inherited the fragment of the request URL. URL.resolve
follows RFC 3986 §5.3 which drops the base fragment, so handleRedirect
now reattaches the original fragment when the resolved target has none.
Closes#2263
doReload built a NavigateOpts with only url + kind=.reload; method/body/header
defaulted to GET/null/null, so any prior POST navigation regressed to a GET
on reload. The HTML reload navigation re-fetches the document that produced
the current entry, and Chrome replays the same HTTP request that loaded the
page (including method, body, and Content-Type) — Lightpanda dropped all
three.
Retain the prior request body and content-type header in Frame.NavigatedOpts
(duped into the frame arena), and have doReload capture them into the CDP
command's arena before replacePage() frees the old frame. The reload's
frame.navigate call carries the replayed method/body/header so the request
the page was loaded with is the request that runs again.
Closes#2258
Follow up to https://github.com/lightpanda-io/browser/pull/2200
This change is actually pretty mundane, but a bunch of files that used to
take a *Session (e.g. every WebAPI releaseRef and deinit) now take a *Page.
This aims to separate the 2 lifetimes currently managed by Session by moving
the "Page" lifetime to a dedicated container: Page. Ultimately, the goal is to
remove the 1-page-per-session limit of the current design. Not to explicitly
support multiple pages per session (though, that's more possible now), but
in order to better emulate Chrome where, during a navigation event, the old and
new page both exist.
This is to pave the way for introducing a new "Page" container, which will take
over the page lifecycle currently burdening Session. The ultimate goal of that
is to allow the Session to have multiple pages (mostly for better transitions
between pages), which is hard to do now since the Session has so much state.
This rename was aggressive, e.g. currentPage() -> currentFrame() so that, when
the new Page container is added, you won't see "currentPage()" and wonder:
"Does 'currentPage' mean the new Page container, or the Frame (which
used to be called Page)".
@import("lightpanda") where needed.
Would also like to do this for String, Page, Session and js which all stand out
as types that are use across the codebase.
I know that a few devs are doing this in new work and I haven't heard anyone
voice an objection.
This commit fixes a few serious issues with the Websocket implementation.
1 - libcurl recursive api calls
Creating a Websocket instance from within a libcurl callback results in libcurl
failing with a RecursiveApiCall error. I fixed this more generally by adding a
`ready_queue` which connections can use when the `HttpClient` is performing
actions. Once `perform` ends, this new `ready_queue` is processed. There might
be a more holistic solution to this (we seem to run into RecursiveApiCall
everywhere), but since HttpClient is going through heavy changes, this seemed
like the smallest possible change to fix it.
2 - "load" blocking
Load and IdleNetwork notifications should not block on Websocket connections. To
solve this, `HttpClient` now ha `http_active` and `ws_active` to replace `active`.
Only `http_active` is used for things like "load" triggering.
3 - The above change made the Runner's job more complicated. It used to be
binary: you either have active connections or not. Now there are different types
of active connections. To keep it simple, and I think probably more correct,
the "done-ness" (based on the `wait` parameter) is now independent of active
(or not) network activity. If the page's `load_state == .complete`, then the
`wait == .done` is considered successful, whether or not we have active
connections.
4 - As a consequence of the above, and seemingly unrelated to all of these
changes, a number of html tests now use the "new" robust async framework. Most
of these tests were using the `testing.onload` (aka `testing.eventually`) which
had somewhat...unclear semantics. These tests passed more of a consequence of
how we processed a page and being very simple (e.g. just needing 1 micro or
macrotask tick). But `eventually` never worked for more complicated cases, and
the previous `testing.async` didn't work well. Now, the test runner waits for
.load (which, as per #3, can fire more aggressively), which caused many
`eventually` tests to fail. Moving these tests to the new `async` is more
robust and works with the new aggressive "load".
Uses libcurl's websocket capabilities to add support for WebSocket.
Depends on https://github.com/lightpanda-io/zig-v8-fork/pull/167
Issue: https://github.com/lightpanda-io/browser/issues/1952
This is a WIP because it currently uses the same connection pool used for all
HTTP requests. It would be pretty easy for a page to starve the pool and block
any progress.
We previously stored the *Transfer inside of the easy's private data. We now
store the *Connection, and a Connection now has a `transport` field which is
a union for `http: *Transfer` or `websocket: *Websocket`.
These new optional parameter run AFTER --wait-until, allowing the (imo) useful
combination of `--wait-until load --wait-script "report.complete === true"`.
However, if `--wait-until` IS NOT specified but `--wait-selector/script` IS,
then there is no default wait and it'll just check the selector/script. If
neither `--wait-selector` or `--wait-script/--wait-script-file` are specified
then `--wait-until` continues to default to `done`.
These waiters were added to the Runner, and the existing Action.waitForSelector
now uses the runner's version. Selector querying has been split into distinct
parse and query functions, so that we can parse once, and query on every tick.
We could potentially optimize --wait-script to compile the script once and call
it on each tick, but we'd have to detect page navigation to recompile the script
in the new context. Something I'd rather optimize separately.
testing.async(...) is pretty lame. It works for simple cases, where the
microtask is very quickly resolved, but otherwise can't block the test from
exiting.
This adds an overload to testing.async and leverages the new Runner
https://github.com/lightpanda-io/browser/pull/1958 to "tick" until completion
(or timeout).
The overloaded version of testing.async() (called without a callback) will
increment a counter which is only decremented with the promise is resolved. The
test runner will now `tick` until the counter == 0.
CDPT used to be a generic so that we could inject Browser, Session, Page and
Client. At some point, it [thankfully] became a generic only to inject Client.
This commit removes the generic and bakes the *Server.Client instance in CDP.
It uses a socketpair for testing.
BrowserContext is still generic, but that's generic for a very different reason
and, while I'd like to remove that generic too, it belongs in a different PR.
This is done for a couple reasons. The first is just to have things a little
more self-contained for eventually supporting more advanced "wait" logic, e.g.
waiting for a selector.
The other is to provide callers with more fine-grained controlled. Specifically
the ability to manually "tick", so that they can [presumably] do something
after every tick. This is needed by the test runner to support more advanced
cases (cases that need to test beyond 'load') and it also improves (and fixes
potential use-after-free, the lp.waitForSelector)
Small tweaks to https://github.com/lightpanda-io/browser/pull/1896
Improve the wait ergonomics with an Option with default parameter. Revert
page pointer logic to original (don't think that change was necessary).
Add `--wait_until` and `--wait_ms` CLI arguments to configure session wait behavior. Updates `Session.wait` to evaluate specific page load states (`load`, `domcontentloaded`, `networkidle`, `fixed`) before completing the wait loop.
Depends on: https://github.com/lightpanda-io/zig-v8-fork/pull/153
In some ways this is an extension of
https://github.com/lightpanda-io/browser/pull/1635 but it has more implications
with respect to correctness.
A js.Context wraps a v8::Context. One of the important thing it adds is the
identity_map so that, given a Zig instance we always return the same v8::Object.
But imagine code running in a frame. This frame has its own Context, and thus
its own identity_map. What happens when that frame does:
```js
window.top.frame_loaded = true;
```
From Zig's point of view, `Window.getTop` will return the correct Zig instance.
It will return the *Window references by the "root" page. When that instance is
passed to the bridge, we'll look for the v8::Object in the Context's
`identity_map` but wont' find it. The mapping exists in the root context
`identity_map`, but not within this frame. So we create a new v8::Object and now
our 1 zig instance has N v8::Objects for every page/frame that tries to access
it.
This breaks cross-frame scripting which should work, at least to some degree,
even when frames are on the same origin.
This commit adds a `js.Origin` which contains the `identity_map`, along with our
other `v8::Global` storage. The `Env` now contains a `*js.Origin` lookup,
mapping an origin string (e.g. lightpanda.io:443) to an *Origin. When a Page's
URL is changed, we call `self.js.setOrigin(new_url)` which will then either get
or create an origin from the Env's origin lookup map.
js.Origin is reference counted so that it remains valid so long as at least 1
frame references them.
There's some special handling for null-origins (i.e. about:blank). At the root,
null origins get a distinct/isolated Origin. For a frame, the parent's origin
is used.
Above, we talked about `identity_map`, but a `js.Context` has 8 other fields
to track v8 values, e.g. `global_objects`, `global_functions`,
`global_values_temp`, etc. These all must be shared by frames on the same
origin. So all of these have also been moved to js.Origin. They've also been
merged so that we now have 3 fields: `identity_map`, `globals` and `temps`.
Finally, when the origin of a context is changed, we set the v8::Context's
SecurityToken (to that origin). This is a key part of how v8 allows cross-
context access.
This was already handled in most cases, but not for a body-less response. It's
safe to call transfer.abort() during a callback, so long as the performing flag
is set to true. This was set during the normal libcurl callbacks, but for a
body-less response, we manually invoke the header_done_callback and were not
setting the performing flag.
Missing:
- [ ] Navigation support within frames (in fact, as-is, any navigation done
inside a frame, will almost certainly break things
- [ ] Correct CDP support. I don't know how frames are supposed to be exposed
to CDP. Normal navigate events? Distinct CDP frame_ids?
- [ ] Cross-origin restrictions. The interaction between frames is supposed to
change depending on whether or not they're on the same origin
- [ ] Potentially handling src-less frames incorrectly. Might not really matter
Adds basic frame support. Initially explored adding a BrowsingContext and
embedding it in Page, with the goal of also having it embedded in a to-be
created Frame. But it turns out that 98% of Page _was_ BrowsingContext and
introducing a BrowsingContext as the primary interaction unit broke pretty much
_every_ single WebAPI. So Page was expanded:
- Added `_parent: ?*Page`, which is `null` for "root" page.
- Added `frame: ?*IFrame`, which is `null` for the "root" page. This is the
HTMLIFrameElement for frame-pages.
- Added a _type: enum{root, frame}, which is currently only used to improve
the logs
- Added a frames: std.ArrayList(*Page). This is a list of frames for the page.
Note that a "frame-page" can itself haven nested frames.
Besides the above, there were 3 "big" changes.
1 - Adding frames (dynamically, parsed) has to create a new page, start
navigation, track it (in the frames list). Part of this was just
piggybacking off of code that handles <script>
2 - The page "load" event blocks on the frame "load" event. This cascades.
when a page triggers it's load, it can do:
```zig
if (self._parent) |p| {
p.iframeLoaded(self);
}
```
Pages need to keep track of how many iframes they're waiting to load. When
all iframes (and all scripts) are loaded, it can then triggers its own load
event.
3 - Our JS execution expects 1 primary entered context (the pages). But we now
have multiple page contexts, and we need to be in the correct one based
on where javascript is being executed. There is no more an default entered
context. Creating a Local.Scope enters the context, and ls.deinit() exits
the context.
Navid EMAD