Saw a non-fatal error on cnn.com related to console. All members are supposed
to be static. Ran it through WPT /console/ and improved a few different thigns:
- added dir and dirxml
- assert condition is optional
CDP driver can send multiple Network.enable which would register the same
listener multiple times. This commit makes it so that only one (the first)
callback registered for a listener+eventtype is used. Subsequent registration
for the same listener+eventtype are ignored. This is safe because all callbacks
are currently static. It's a mistake (enforced by a debug-only assertion) for
code to try to register a different callback for an already registered listener+
eventtype.
This generalizes https://github.com/lightpanda-io/browser/pull/3038
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
CDP clients (Playwright/Puppeteer) never saw same-document navigations —
history.pushState / replaceState changed frame.url but fired no CDP event,
so frame.url() went stale and waitForNavigation missed SPA route changes.
Add a frame_navigated_within_document notification, dispatched from
History.pushState/replaceState, and emit Page.navigatedWithinDocument from
the Page domain with frameId, url, and navigationType. Unlike frameNavigated,
this deliberately does NOT send Runtime.executionContextsCleared or
DOM.documentUpdated — the document and its execution context are unchanged,
and clearing them would invalidate the client's live context ids and break
frame.evaluate(). Adds a cdp.frame test.
Scoped to the History-API path; fragment and Navigation-API paths are left
for follow-ups.
Replaces layering with an inline request pipeline, and transfer queue. This is
meant to simplify the code, reduce footguns, and make future enhancements easier
to implement (e.g. speculative parsing (which requires streaming to fully
leverage)).
Previously, HttpClient implemented deferring as a layer which required special
pumping at various callsites (https://github.com/lightpanda-io/browser/pull/2855,
https://github.com/lightpanda-io/browser/pull/2843, ...). In this new approach,
deferring is built-into the HttpClient/Transfer's flow. Specifically, Transfers
now maintain a queue of events (start, header, data, end, err) which are
dispatched in HttpClient.tick. The result is that JS callbacks are never
executed in the same stack that initiated the I/O, without needing guards or any
external intervention.
tTwo other benefits come from this. The first is that reentrant libcurl is
eliminated. Instead of "libcurl -> callback", it's now "libcurl -> transfer
event queue THEN tick -> callback" (we don't have to wait until the NEXT tick, we
can just do it later in the tick). HttpClient still has to guard against libcurl
reentrancy, but only because of how WebSocket is implemented, and we should be
able to unify WebSockets to use an event queue too in a follow up PR (which will
eliminate a bunch of guard code).
The transfer queue should also be useful to re-implement streaming, since a
data chunk is just an event in the transfer's event queue. For now, I kept it
as a single buffered event to minimize the change. But since speculative parsing
depends on this, and speculative parsing seems to be the next major performance
tweak we can make, we need to re-introduce streaming.
The other change is the removal of all other layers in favor of a pipeline. This
works well with the existing Transfer.park mechanism, where a parked Transfer
can restart the pipeline for a transfer in an arbitrary point (not as fancy as
it sounds given how simple the flow is). The fallout from this is that we're no
longer creating/wrapping contexts and callbacks: whatever the request was
configured with is all we need.
Because of this, HttpClient.Response is removed. There are no intermediary
responses and no changing context, everything is just the Transfer.
A smaller change is the addition of newRequest + transfer.submit(). The one-shot
HttpClient.request and HttpClient.requestT still exist, but this explicit create
+ submit has some advantage. First, callers can use the transfer.arena (e.g.
Frame using the transfer's arena to set the Referrer header). Second, callers
can holds Transfer immediately, rather than waiting for their startCallback to
be fired. An abort on an XMLHttpRequest called before the start of the transfer
no longer silently fails.
Browser.setDownloadBehavior was a noop, so Lightpanda had no file-download
path. A response with Content-Disposition: attachment is now streamed to disk
under downloadPath, and Page.downloadWillBegin / Browser.downloadProgress are
emitted when eventsEnabled.
Fixes#2701
console.log and console.warn were both dispatched as .info, so
Runtime.consoleAPICalled (and Console.messageAdded) reported type
"info" for all three of log/info/warn — clients filtering console
output by severity saw them collapsed into one bucket.
Add log/warning members to ConsoleMessageType (the protocol's wire
values — "warning", not "warn") and map console.log -> log,
console.warn -> warning. info/error/debug/trace already matched.
Closes#2730
A root navigation that fails before any response headers arrive
(connection refused, DNS failure, TLS error) never answered the
Page.navigate command: the success reply is only sent from the
response-header path (frame_navigated), and frameErrorCallback dropped
the stored cdp_id along with the pending Page. The CDP client awaiting
the response was left hanging forever.
Dispatch a frame_navigate_failed notification from frameErrorCallback
when no response headers were processed (_http_status == null guards
against double-answering mid-body failures), and answer the pending
command with {frameId, loaderId, errorText} — matching Chrome, where
errorText is "present if and only if navigation has failed".
Closes#2728
1 - Track owner of a request (for simpler / more accurate abort (TBD))
2 - Create Transfer upfront, make everything work on Transfer (not Request)
This helps remove ambiguity about cleanup and simplifies layers. For example
Robots request is just another normal request, not a special case. This gives
everything a stable address (the *Transfer which can be looked up by id)
During a root navigation, we keep the existing page active until we get
the headers callback from the pending page. Then
Session.commitPendingPage makes the switch.
It delays the deinit of CPD execution context to handle JS execution in
the meantime.
Now session has an array of two pages, _active_idx points to the main
page.
Both active and pending pages share the same frame_id, it must remains
stable. So this PR adds a Request.protect_from_abort to avoid removing
the request form the pending page when deinit the previous active page.
Page.handleJavaScriptDialog previously responded -32000 "No dialog is
showing" regardless of whether a dialog was open, leaving CDP clients
no way to influence the JS-side return value of confirm() / prompt().
PR #2085 wired up the Page.javascriptDialogOpening event but explicitly
deferred the return-value override since true Chrome semantics require
suspending V8 mid-execution.
Add a pre-arm model that fits the auto-dismiss architecture without
runtime suspension: handleJavaScriptDialog stashes {accept, promptText}
on the BrowserContext; when the next JS dialog dispatches the
javascript_dialog_opening notification, the listener pops the stash and
fills it into the dispatch's response output param so Window.confirm /
prompt return the CDP client's choice. Without a pre-arm, headless
auto-dismiss values from PR #2085 are preserved (confirm->false,
prompt->null, alert->void).
Closes#2260
This is to pave the way for introducing a new "Page" container, which will take
over the page lifecycle currently burdening Session. The ultimate goal of that
is to allow the Session to have multiple pages (mostly for better transitions
between pages), which is hard to do now since the Session has so much state.
This rename was aggressive, e.g. currentPage() -> currentFrame() so that, when
the new Page container is added, you won't see "currentPage()" and wonder:
"Does 'currentPage' mean the new Page container, or the Frame (which
used to be called Page)".
Rename page.id -> page._loader_id and propagate the change throughout. This was
my attempt at pretending that page.id (and page._frame_id) weren't CDP-sepcific.
But they are, and it's a lot cleaner to treat them this way. Might seem
unnecessary, but without this, after page -> frame, you'd end up with:
frame.id
frame._frame_id
Which is weird? What is `frame.id` if it isn't the frame id and if that's the
case, what's frame_id? Now it'll be:
frame._loader_id
frame._frame_id
Which removes the ambiguity, makes the CDP code a bit more obvious, and doesn't
try to hide the fact that these are CDP things that, for now at least, pollute
the code a little.
@import("lightpanda") where needed.
Would also like to do this for String, Page, Session and js which all stand out
as types that are use across the codebase.
I know that a few devs are doing this in new work and I haven't heard anyone
voice an objection.
This introduces two slightly related changes.
My understanding is:
- frameId represents the page. Even if the page navigates, it's the same
frameId. We capture this in Page._frame_id. Nothing here changes.
- loaderId is essentially for a specific document of the page. If the page
navigates, it should be a different loaderId. We were using a distinct
loaderId per request. Not sure what problems that caused. But it was wrong.
This was achieved by exposing Page.id to CDP.
- requestId was mostly correct: unique per request. HOWEVER, for the original
document, apparently, requestId == loaderId. This change is particularly
important for various puppeteer and playwrightb behavior. This is a bit
hacked. CDP will look at the resource_type, if it's .document, it'll return
the loaderId, else it returns the requestId it always id.
window.alert(), confirm(), and prompt() now dispatch a
javascript_dialog_opening notification that the CDP layer
forwards as a Page.javascriptDialogOpening event. This enables
Puppeteer's page.on('dialog') to fire when JS dialogs open.
Also adds Page.handleJavaScriptDialog as a CDP method. Dialogs
still auto-dismiss in headless mode (alert is void, confirm
returns false, prompt returns null), so handleJavaScriptDialog
is an acknowledgement rather than a blocking gate.
Changes:
- Notification.zig: add JavascriptDialogOpening event type
- CDP.zig: register listener, forward to page domain
- page.zig: handleJavaScriptDialog handler + event emitter
- Window.zig: alert/confirm/prompt dispatch the notification
Fixes#2082
Ref #2043
These changes all better align with chrome's event ordering/timing.
There are two big changes. The first is that our internal page_navigated event,
which is kind of our heavy hitter, is sent once the header is received as
opposed to (much later) on document load. The main goal of this internal event
is to trigger the "Page.frameNavigated" CDP event which is meant to happen
once the URL is committed, which _is_ on header response.
To accommodate this earlier trigger, new explicit events for DOMContentLoaded
and load have be added.
This drastically changes the flow of events as things go from:
Start Page Navigation
Response Received
Start Frame Navigation
Response Received
End Frame Navigation
End Page Navigation
context clear + reset
DOMContentLoaded
Loaded
TO:
Start Page Navigation
Response Received
End Page Navigation
context clear + reset
Start Frame Navigation
Response Received
End Frame Navigation
DOMContentLoaded
Loaded
So not only does it remove the nesting, but it ensures that the context are
cleared and reset once the main page's navigation is locked in, and before any
frame is created.
This field was recently added and is used to generate correct frameIds in CDP
messages. They remain the same during a navigation event, so calling them
page.id might cause surprises since navigation events create new pages, but
retain the original id. Hence, frame_id is more accurate and hopefully less
surprising.
(This is a small cleanup prior to doing some iframe navigation work).
Our BrowsingContext currently supports 1 target. So we have a per-BC target_id.
Previously, our target had 1 "frame" - our page. So we often treated the
targetId as the frameId. But to work with frames, we need page-specific
frameIds and loaderIds.
This tries to clean up our ids (a little). frameIds are now ids derived from
a new incrementing page.id. This page.id has to be passed around (via http
Requests and through notifications) in order to properly generate messages with
a frameId.
Deciding what should be an lp.assert, vs an std.debug.assert, vs a debug-only
assert is a little arbitrary.
debug-only asserts, guarded with an `if (comptime IS_DEBUG)` obviously avoid the
check in release and thus have a performance advantage. We also use them at
library boundaries. If libcurl says it will always emit a header line with a
trailing \r\n, is that really a check we need to do in production? I don't think
so. First, that code path is checked _a lot_ in debug. Second, it feels a bit
like we're testing libcurl (in production!)..why? A debug-only assertion should
be good enough to catch any changes in libcurl.
Karl Seguin