Give it one pass through some Claude fuzz testing. Add a max message size,
protect against weird interactions during a shutdown and we had some pending
accepts. Put a time limit on blocked writes.
Significant rework of the CDP/BiDi server. There are two main changes:
1 - poll replaced with EPoll/Kqueue (1)
2 - make http serving a first class citizen
The change from poll -> epoll/kqueue isn't performance driven, it's just about
tighter code. Both epoll and kqueue let you associate arbitrary data with a
socket, so we don't need to keep arrays in sync in order to associate a socket
with a CDP by index. They both provide some event/notification mechanism, which
is cleaner than the pipe required by poll.
The poll -> epoll/kqueue change could almost have been mechanical. Making HTTP
a first class citizen is the more significant of the two changes
In `main`, a new connection always spawns a thread and, until does its own
little read loop until the connection is upgraded. This is not efficient, it
uses up a connection slot, and it's inconsistent with the final WebSocket
connection which _is_ polled off the main loop. Using up a slot means that
keepalive isn't possible, else HTTP connections would quickly use up all
available slots/threads.
This commit parses and serves HTTP requests on the main thread (safe
because none of the processing is blocking). The approach is better streamlined
for HTTP requests which never upgrade (/metrics, WebDriver) without causing
any performance overhead for those that do. It simplifies some things (e.g. an
"http" socket or a "websocket" socket is monitored and read in a similar manner
(on the main loop)). It makes other things more complicated; the flow is no
longer accept -> spawn -> upgrade -> websocket loop. It's loop -> accept -> loop
-> process -> (http | ws).
This is built ontop of the BiDi branch because (a) WebDriver is what needs
better HTTP support and (b) some of the more mechanical changes already exist
in that branch (e.g. src/cdp/, src/server.zig -> src/server/*)
(1) kqueue landing in 2 commits from now on this branch.
Adds resource timing, e.g. `performance.getEntriesByType("resource")`.
The `resource-timing` WPT category is currently at 4.6%, and this is a first
step at improving it. It also hopefully fixes https://github.com/lightpanda-io/browser/issues/3359
This is more complicated than I thought because there's a "Timing-Allow-Origin"
header that a server can include which hides some of the data if the request
doesn't come from the listed origin. And that, of course, interacts with
redirects.
(The DOMException change is seemingly random, but it came up in one of the WPT
cases I was looking at).
A delivery count cannot separate a runaway from a busy but healthy page:
airbnb legitimately needs up to ~50 delivery sessions per load, and the
storefront runaway had already exhausted the timer table by ~70.
Duration does: every healthy burst measured went quiet within 4.5s of
its first delivery, while the runaway never did.
Deliveries closer than 2s apart form a burst; a burst that runs past 10s
disconnects the frame's intersection observers. Measuring from the start
of the current burst rather than from the first delivery keeps lazy
loading alive on pages that live for minutes in serve or agent mode.
A chain that re-observes synchronously never leaves the microtask
checkpoint, so a per-burst ceiling of 1024 deliveries stops it from
spinning for the full limit.
Our redirect handling was "optimized" to re-use the same easy connection, at the
cost of circumventing the entire pipeline. A redirect would not check the new
target's robots.txt, rate limit, CORS, Web Auth, ...
The new code simply restarts the pipeline (with a discriminator tag so that
things like CDP can tell the difference between a redirected request and a new
one).
Both globals were missing, so MDN-canonical stream setup code threw
ReferenceError. size() is a method rather than an accessor returning a
function; the streams don't consume it yet.
CURLMOPT_MAXCONNECTS was never set, so libcurl used its default of 4x the
number of easy handles currently attached to the multi. Handles are added
and removed per transfer, so between page loads that default collapses to
roughly zero and every cached connection is evicted: revisiting a host
after browsing elsewhere re-paid connect + TLS every time.
Measured over 25 navigations across 5 sites (5 rounds, ReleaseFast), median
warm navigation drops from 0.73s to 0.42s on news.ycombinator.com and from
0.71s to 0.42s on github.com; total navigation time 11.1s -> 7.3s. Loading
one host repeatedly was already fast and is unchanged; the win is on
cross-site browsing, which is what agents and crawlers actually do.
If you look at https://github.com/lightpanda-io/browser/pull/3293, you'll see
a relatively contained change that has to touch over 20 files. The issue is that
every HttpClient.newRequest needs to provide a lot of data. But `newRequest`
takes a 2nd parameter: the HttpClient.Owner. If we make that Owner a little
smarter, we can start to remove some of the individual fields needed in
newRequest. For example, we can still allow a callsite to pass frame_id but,
by default, we can use the owner's frame_id (which is what we want in most
cases).
Also updates matchCookies/onCookieChanged from the hard-coded "same-site + navigation" to areSameSite(exec.siteForCookies(), host) and is_navigation=false.
The site for cookies were computed from the immediate parent `Frame`, which would allow sending a cookie that's `SameSite=Strict` from 2 levels deep under. Directly from RFC6265bis, this PR essentially implements (except for step 4, we skip host-less ancestors):
Given a Document (document), the following algorithm returns its
"site for cookies":
1. Let top-document be the active document in document's navigable's
top-level traversable.
2. Let top-origin be the origin of top-document's URI if top-
document's sandboxed origin browsing context flag is set, and
top-document's origin otherwise.
3. Let documents be a list consisting of the active documents of
document's inclusive ancestor navigables.
4. For each item in documents:
1. Let origin be the origin of item's URI if item's sandboxed
origin browsing context flag is set, and item's origin
otherwise.
2. If origin is not same-site with top-origin, return an origin
set to an opaque origin.
5. Return top-origin.
Add an update-python-package job that creates the matching release on
lightpanda-io/lightpanda-python (same distribution-app pattern as the
docker and homebrew jobs). That release triggers the wheels workflow
there, replacing its daily track-browser polling; its publish to PyPI
still waits for approval on that repo's pypi environment.
The distribution app needs contents write access to lightpanda-python.
We report every attached element as fully visible, because without layout the
alternative (a faux position vs. the viewport) hides everything past the first
screenful. Infinite scroll turns that into a non-terminating loop: the page
observes a sentinel, we report it intersecting immediately, the callback loads
the next page and observes a fresh sentinel, and so on with nothing to stop it.
Seen on a live storefront: the collection page paginated itself to page 10 over
68 delivery sessions and ~94 requests, exhausted the timer table
(error.TooManyTimeout), and hung until the 30s watchdog killed the navigation.
It reproduced on roughly one run in seven.
Cap the delivery sessions per document and disconnect the frame's intersection
observers past it, mirroring the MutationObserver runaway guard. The limit is
32: measured over a static page, an ad-heavy news hub and a storefront, no
document needed more than 7 sessions, while a runaway climbs without bound.
This bounds the damage rather than fixing the visibility model. Reporting
intersections against a real viewport, and resetting the budget when the page
actually scrolls, are the follow-ups.
Karl Seguin