Currently, the HttpClient owns the inbox and its borrowed by the Link. This is
a bit backwards, but it also means that we can't eagerly create a Link: the
Link needs the inbox, so it needs the HttpClient, which is created by the
Browser (which creates an Isolate).
Remember, the Inbox is one of the few things shared between the main thread
and the worker, so either end can own it and the other can borrow it.
This switches the ownership so that the HttpClient now borrows the Inbox from
the Server's side of the Link (the WebSocket).
The main goal of this change is to prepare for more advanced HTTP WebDriver
flows. The more we can create _without_ a Browser, the fewer edge cases we have
to deal with (Browser because it's expensive and has to be created on the
Worker thread due to how V8::Isolate works).
Give it one pass through some Claude fuzz testing. Add a max message size,
protect against weird interactions during a shutdown and we had some pending
accepts. Put a time limit on blocked writes.
Significant rework of the CDP/BiDi server. There are two main changes:
1 - poll replaced with EPoll/Kqueue (1)
2 - make http serving a first class citizen
The change from poll -> epoll/kqueue isn't performance driven, it's just about
tighter code. Both epoll and kqueue let you associate arbitrary data with a
socket, so we don't need to keep arrays in sync in order to associate a socket
with a CDP by index. They both provide some event/notification mechanism, which
is cleaner than the pipe required by poll.
The poll -> epoll/kqueue change could almost have been mechanical. Making HTTP
a first class citizen is the more significant of the two changes
In `main`, a new connection always spawns a thread and, until does its own
little read loop until the connection is upgraded. This is not efficient, it
uses up a connection slot, and it's inconsistent with the final WebSocket
connection which _is_ polled off the main loop. Using up a slot means that
keepalive isn't possible, else HTTP connections would quickly use up all
available slots/threads.
This commit parses and serves HTTP requests on the main thread (safe
because none of the processing is blocking). The approach is better streamlined
for HTTP requests which never upgrade (/metrics, WebDriver) without causing
any performance overhead for those that do. It simplifies some things (e.g. an
"http" socket or a "websocket" socket is monitored and read in a similar manner
(on the main loop)). It makes other things more complicated; the flow is no
longer accept -> spawn -> upgrade -> websocket loop. It's loop -> accept -> loop
-> process -> (http | ws).
This is built ontop of the BiDi branch because (a) WebDriver is what needs
better HTTP support and (b) some of the more mechanical changes already exist
in that branch (e.g. src/cdp/, src/server.zig -> src/server/*)
(1) kqueue landing in 2 commits from now on this branch.
If you look at https://github.com/lightpanda-io/browser/pull/3293, you'll see
a relatively contained change that has to touch over 20 files. The issue is that
every HttpClient.newRequest needs to provide a lot of data. But `newRequest`
takes a 2nd parameter: the HttpClient.Owner. If we make that Owner a little
smarter, we can start to remove some of the individual fields needed in
newRequest. For example, we can still allow a callsite to pass frame_id but,
by default, we can use the owner's frame_id (which is what we want in most
cases).
The site for cookies were computed from the immediate parent `Frame`, which would allow sending a cookie that's `SameSite=Strict` from 2 levels deep under. Directly from RFC6265bis, this PR essentially implements (except for step 4, we skip host-less ancestors):
Given a Document (document), the following algorithm returns its
"site for cookies":
1. Let top-document be the active document in document's navigable's
top-level traversable.
2. Let top-origin be the origin of top-document's URI if top-
document's sandboxed origin browsing context flag is set, and
top-document's origin otherwise.
3. Let documents be a list consisting of the active documents of
document's inclusive ancestor navigables.
4. For each item in documents:
1. Let origin be the origin of item's URI if item's sandboxed
origin browsing context flag is set, and item's origin
otherwise.
2. If origin is not same-site with top-origin, return an origin
set to an opaque origin.
5. Return top-origin.
Checked the three test files in Firefox and Chrome:
- stepUp/stepDown use the HTML step base (min, else the value attribute)
and snap off-ladder values to the next rung, counting the snap as the
first step as browsers do; clamping lands on the last rung inside
min/max.
- time strings keep a three-digit fraction.
- an empty pattern attribute is a pattern (matches only "").
- tooLong/tooShort only for values last changed by a user edit, so the
text-entry path marks the value and script/attribute values never trip
them; same for textarea.
- showPicker dropped: browsers throw NotAllowedError without a gesture,
a no-op would be a lie.
- month/week assertions skipped where the browser has no such input.
window.stop() is less destructive than other mechanisms we have. For one, it
seems largely isolated to pending or inflight HTTP requests. For anther, it
keeps the page intact.
To achieve this, HttpClient gains an `cancelRequests` which is a gentler version
of `abortOwner`. It cancels inflight/pending HTTP requests, which results in
error callbacks (not shutdown callbacks) firing.
Just like https://github.com/lightpanda-io/browser/pull/3189 I ran into the
problem that I couldn't distinguish between an HTTP request that was canceled
because of user-action (e.g. calling window.stop(), or xhr.abort()) and an HTTP
request that was internally aborted. These now have distinct errors/flows so
that we can present the correct state. Most places that aborted now all
transfer.cancel() which results in a distinct `error.TransferCanceled` (some
places still abort -> `error.Abort`). It should be possible to revisit 3189 now.
The CDP "Page.stopLoading" now hooks into this new behavior. Fixes
https://github.com/lightpanda-io/browser/issues/3351
We currently have 1 note: it prints the server's listening address:port. Note
is a special un-ignorable level. This keeps the "note" level, but logs it under
a new scope: "note", so that it _can_ be silenced with a `--log-filter note`.
Add a new note, on startup, that displays tips. Currently, only displays when
--obey-robots is not enabled:
NOTE note : config tips . . . . . . . . . . . . . . . . . . . [+0ms]
robots = use '--obey-robots' to use a sites robots.txt
meta = use '--log-filter note' to silence this message
A client that disconnects might get treated as a harsher terminate failure (e.g.
watchdog). This doesn't have a huge impact, but it makes the CI flaky and it
produces more logs than is necessary.
In a terminate state, the driver will now check its inbox to see if this is a
client disconnection.
The Fetch.enable call takes `patterns` which can limit the path and type of
request that should be intercepted. This adds the wildcard support.
As before, RI is currently only enabled for Requests, not response. A warning
is printed if RI for responses is requested.
Fixes: https://github.com/lightpanda-io/browser/issues/3349
In order to support Selenium the way people are used to, it looks like we need
to support both WebDriver classic (WebDriver) and WebDriver BiDi (BiDi). Typical
scripts look like a mix of the two, e.g. using WebDriver to control the browser
and using BiDi to receive notifications. This commit:
1 - adds a --protocol (cdp|webdriver) CLI argument to the `serve` command to
enable one or the other protocol (defaulting to CDP)
2 - adds basic WebDriver endpoint to let a Selenium client connect. This
implementation is hackish and sits on top of our simple Handshake handler.
The handshake handler is well past its original design. Serving /json/version
and /metrics from it was one thing. But Driving the entire browser session? This
will get a follow up PR.
1. Abstract "Driver". Non-CDP things that referenced *CDP now reference a Driver
2. Move the NodeRegistry out of CDP. This created an artificial link between
agent / mcp and CDP
3. Add BiDi driver with enough to navigate to a page
Karl Seguin