This piggybacks on the HttpClient rework (1) and makes incoming WebSocket
messages get delivered in a similar fashion. Namely, WebSocket events (connect,
data, close) are queued and only delivered at safe points. The goal is to make
sure JS is never run during a libcurl callback. This has historically caused
issues and currently requires the HttpClient to have various guards (e.g.
performing flag, dirty queue, etc..).
This change removes the last cases where JS could be executed within a libcurl
callback. Consequently, HttpClient is simplified. It no longer has a perfoming
flag nor a dirty queue nor ready queue.
(1) https://github.com/lightpanda-io/browser/pull/2889
Replaces layering with an inline request pipeline, and transfer queue. This is
meant to simplify the code, reduce footguns, and make future enhancements easier
to implement (e.g. speculative parsing (which requires streaming to fully
leverage)).
Previously, HttpClient implemented deferring as a layer which required special
pumping at various callsites (https://github.com/lightpanda-io/browser/pull/2855,
https://github.com/lightpanda-io/browser/pull/2843, ...). In this new approach,
deferring is built-into the HttpClient/Transfer's flow. Specifically, Transfers
now maintain a queue of events (start, header, data, end, err) which are
dispatched in HttpClient.tick. The result is that JS callbacks are never
executed in the same stack that initiated the I/O, without needing guards or any
external intervention.
tTwo other benefits come from this. The first is that reentrant libcurl is
eliminated. Instead of "libcurl -> callback", it's now "libcurl -> transfer
event queue THEN tick -> callback" (we don't have to wait until the NEXT tick, we
can just do it later in the tick). HttpClient still has to guard against libcurl
reentrancy, but only because of how WebSocket is implemented, and we should be
able to unify WebSockets to use an event queue too in a follow up PR (which will
eliminate a bunch of guard code).
The transfer queue should also be useful to re-implement streaming, since a
data chunk is just an event in the transfer's event queue. For now, I kept it
as a single buffered event to minimize the change. But since speculative parsing
depends on this, and speculative parsing seems to be the next major performance
tweak we can make, we need to re-introduce streaming.
The other change is the removal of all other layers in favor of a pipeline. This
works well with the existing Transfer.park mechanism, where a parked Transfer
can restart the pipeline for a transfer in an arbitrary point (not as fancy as
it sounds given how simple the flow is). The fallout from this is that we're no
longer creating/wrapping contexts and callbacks: whatever the request was
configured with is all we need.
Because of this, HttpClient.Response is removed. There are no intermediary
responses and no changing context, everything is just the Transfer.
A smaller change is the addition of newRequest + transfer.submit(). The one-shot
HttpClient.request and HttpClient.requestT still exist, but this explicit create
+ submit has some advantage. First, callers can use the transfer.arena (e.g.
Frame using the transfer's arena to set the Referrer header). Second, callers
can holds Transfer immediately, rather than waiting for their startCallback to
be fired. An abort on an XMLHttpRequest called before the start of the transfer
no longer silently fails.
Idea here is to skip re-parsing that happen for each connection; we already use BoringSSL, so we can take more advantage of it by directly mutating cert store of `SSL_CTX`.
This reverts recent(ish) changes to telemetry which moved it from its own thread
onto the main thread.
The downside is: we have an extra thread.
The upside is largely that Network.zig becomes drastically simpler and more
efficient. There's a bunch of machinery in Network.zig to support arbitrary
workers, of which Telemetry is the only one. There's also a lot of code to
support an optional multi and requests made to is. This is all removed.
Also, fetch, agent and mcp without a cdp server no longer even need to start
the network loop. And, it IS started (e.g. serve/cdp), there's no longer an
arbitrary 250ms wakeup on poll to progress workers. Nor can telemtry block CDP.
Telemetry's implementation itself was changed. The ring buffer was removed in
favor of a double-buffer arraylist. When telemetry is disabled, this saves
64Kb of memory. When it's enabled, it creates more allocator churn, but should
still use less memory in most cases (and never more). Finally, Telemetry is
given its own easy connection rather than using one out of the pool (which
workers would maybe like to use).
Largely aimed at /mimesniff/mime-types/parsing.*.html. Adds ~2000 passing cases.
The blob API is now a bit simpler (hence FormData/REquest/Response/WebSocket)
being touched, but more logic is now in Mime.zig to parse/validate MIME types.
Better escaping support, correct casing and handling edge cases with multiple
charset, or "charset" as a value, .etc...
Headers also hook into this to validate / normalize header name and values.
Network.setExtraHTTPHeaders applied caller headers with a plain
curl_slist append. Requests are seeded with a default `User-Agent:
Lightpanda/1.0`, so a caller-supplied User-Agent produced two
User-Agent entries; libcurl keeps the first and the override was
silently dropped (origins always saw `Lightpanda/1.0`). Other headers
worked because they had no default to collide with.
Add Headers.set, which replaces any existing header with the same
(case-insensitive) name before appending, and use it when applying the
CDP extra headers. This matches Chrome, where Network.setExtraHTTPHeaders
overrides default request headers.
Refs #2704
Previously, the CDP socket was added to the worker's multi and fully owned
by the worker. While this is simple, it introduced some issues:
1 - Cannot detect a disconnected client during JS processing ( for(;;) )
2 - A blocked worker can cause back-pressure that blocks the client. This can
cause a deadlock if the worker is blocked waiting for a CDP message
In addition to these 2 problems, there was 1 other serious CDP-related issue:
arbitrary CDP messages could be processed during JavaScript callback. For
example, a Worker calls importScripts while request interception is enabled,
this requires us to tick the HttpClient waiting for the interception response.
But, a client could sent Target.closeTarget, which we'd process and delete the
frame..all while importScripts is still blocked. Assuming importScripts unblocks
everything is a big UAF since the frame (and its workers) were cleared from
closeTarget.
The CDP socket is now read from the network (main) thread and an OTP-style
mailbox is used. The network thread posts message to the Worker's inbox and
signals it to wakeup. This solves #1 and #2. It doesn't directly solve the
reentrancy issue, but it provides the foundation. Specifically, in introduces
a queue for of CDP message and more control over when/how that queue is
processed. At "safe points" (Runner.tick, HttpClient.tick), any message can
be processed. But, when inside a JavaScript callback, we can process only non-
destructive/mutating message. Specifically, we can process only messages related
to request interception.
This commit fixes a few serious issues with the Websocket implementation.
1 - libcurl recursive api calls
Creating a Websocket instance from within a libcurl callback results in libcurl
failing with a RecursiveApiCall error. I fixed this more generally by adding a
`ready_queue` which connections can use when the `HttpClient` is performing
actions. Once `perform` ends, this new `ready_queue` is processed. There might
be a more holistic solution to this (we seem to run into RecursiveApiCall
everywhere), but since HttpClient is going through heavy changes, this seemed
like the smallest possible change to fix it.
2 - "load" blocking
Load and IdleNetwork notifications should not block on Websocket connections. To
solve this, `HttpClient` now ha `http_active` and `ws_active` to replace `active`.
Only `http_active` is used for things like "load" triggering.
3 - The above change made the Runner's job more complicated. It used to be
binary: you either have active connections or not. Now there are different types
of active connections. To keep it simple, and I think probably more correct,
the "done-ness" (based on the `wait` parameter) is now independent of active
(or not) network activity. If the page's `load_state == .complete`, then the
`wait == .done` is considered successful, whether or not we have active
connections.
4 - As a consequence of the above, and seemingly unrelated to all of these
changes, a number of html tests now use the "new" robust async framework. Most
of these tests were using the `testing.onload` (aka `testing.eventually`) which
had somewhat...unclear semantics. These tests passed more of a consequence of
how we processed a page and being very simple (e.g. just needing 1 micro or
macrotask tick). But `eventually` never worked for more complicated cases, and
the previous `testing.async` didn't work well. Now, the test runner waits for
.load (which, as per #3, can fire more aggressively), which caused many
`eventually` tests to fail. Moving these tests to the new `async` is more
robust and works with the new aggressive "load".
CIDRs prefixed with '-' are treated as allow rules that exempt matching
IPs from blocking. Allow rules take precedence over both
--block_private_networks and custom block CIDRs.
Example: --block_private_networks --block_cidrs -10.0.0.42/32
blocks all private ranges except 10.0.0.42.
Adds 3 new tests for allow-list behavior.
Block outbound HTTP requests to specified IP ranges before TCP handshake
using libcurl CURLOPT_OPENSOCKETFUNCTION callback. Fires after DNS
resolution, reads resolved IP directly from sockaddr, does bitwise CIDR
comparison. Fail-closed: unknown address families are blocked.
--block_private_networks blocks RFC1918, localhost, link-local, ULA.
--block_cidrs blocks additional comma-separated CIDRs.
IPv4-mapped IPv6 (::ffff:x.x.x.x) is unwrapped to prevent bypass.
Implement the XHR timeout property end-to-end: the JS-visible
getter/setter stores the value, send() passes it to the HTTP client,
and curl enforces it via CURLOPT_TIMEOUT_MS. On timeout, a `timeout`
event is dispatched instead of `error`, per the XHR spec.
Uses libcurl's websocket capabilities to add support for WebSocket.
Depends on https://github.com/lightpanda-io/zig-v8-fork/pull/167
Issue: https://github.com/lightpanda-io/browser/issues/1952
This is a WIP because it currently uses the same connection pool used for all
HTTP requests. It would be pretty easy for a page to starve the pool and block
any progress.
We previously stored the *Transfer inside of the easy's private data. We now
store the *Connection, and a Connection now has a `transport` field which is
a union for `http: *Transfer` or `websocket: *Websocket`.
Karl Seguin