Some status-codes should never have a body except for a single trailing blank
line. If we don't handle these, then we end up with a dirty connection in our
connection pool:
1 - read the header, but not the body
2 - put the connection back in the pool
3 - try to read the header, but actually get the body from #1
WPT /fetch/api/basic/response-null-body.any.html exercises this path and is
flaky (because it depends whether the request goes back out on a keep-alive
connection)..but for a given run,you'll almost always get 1-3 failures.
This commit processes the request, but tells libcurl not to re-use the
connection.
Requests issued to an origin while its first connection is still
handshaking each opened their own socket, up to --http-max-host-open,
because curl only learns from ALPN whether the origin multiplexes. With
pipewait they wait for that answer and share one h2 connection.
Fixture: 12 fetch() calls to a fresh cdnjs (h2) origin, release build.
new TCP+TLS connections 6 -> 1
in-page time to last resp ~285 ms -> ~105-135 ms
H1-only origins are unchanged in connection count; their first burst
waits one handshake before fanning out.
Specifically to distinguish cross-site iframe navigation from top-level navigation, this PR reworks how `SameSite=Lax` moved. Since we're not checking if its a navigation alone now, the field for it is also renamed to `lax_allowed`.
Adds resource timing, e.g. `performance.getEntriesByType("resource")`.
The `resource-timing` WPT category is currently at 4.6%, and this is a first
step at improving it. It also hopefully fixes https://github.com/lightpanda-io/browser/issues/3359
This is more complicated than I thought because there's a "Timing-Allow-Origin"
header that a server can include which hides some of the data if the request
doesn't come from the listed origin. And that, of course, interacts with
redirects.
(The DOMException change is seemingly random, but it came up in one of the WPT
cases I was looking at).
CURLMOPT_MAXCONNECTS was never set, so libcurl used its default of 4x the
number of easy handles currently attached to the multi. Handles are added
and removed per transfer, so between page loads that default collapses to
roughly zero and every cached connection is evicted: revisiting a host
after browsing elsewhere re-paid connect + TLS every time.
Measured over 25 navigations across 5 sites (5 rounds, ReleaseFast), median
warm navigation drops from 0.73s to 0.42s on news.ycombinator.com and from
0.71s to 0.42s on github.com; total navigation time 11.1s -> 7.3s. Loading
one host repeatedly was already fast and is unchanged; the win is on
cross-site browsing, which is what agents and crawlers actually do.
window.stop() is less destructive than other mechanisms we have. For one, it
seems largely isolated to pending or inflight HTTP requests. For anther, it
keeps the page intact.
To achieve this, HttpClient gains an `cancelRequests` which is a gentler version
of `abortOwner`. It cancels inflight/pending HTTP requests, which results in
error callbacks (not shutdown callbacks) firing.
Just like https://github.com/lightpanda-io/browser/pull/3189 I ran into the
problem that I couldn't distinguish between an HTTP request that was canceled
because of user-action (e.g. calling window.stop(), or xhr.abort()) and an HTTP
request that was internally aborted. These now have distinct errors/flows so
that we can present the correct state. Most places that aborted now all
transfer.cancel() which results in a distinct `error.TransferCanceled` (some
places still abort -> `error.Abort`). It should be possible to revisit 3189 now.
The CDP "Page.stopLoading" now hooks into this new behavior. Fixes
https://github.com/lightpanda-io/browser/issues/3351
https://github.com/lightpanda-io/browser/issues/3348
When set to 1.1, libcurl is configured to only offer HTTP 1.1. By default, or
when set to "auto", it's up to libcurl to decide how to connect. This maps to
libcurl's CURL_HTTP_VERSION_1_1 and CURL_HTTP_VERSION_NONE.
LP.configureCDP now takes an `httpVersion` field which can be "1.1" or "auto"
to control that specific browser session. Ideally this is called prior to any
navigation.
Network has accumulated a bit of bagged. It knows a lot about certificates, it
knows a lot of the cache. I have plans to expand Network, and wanted to clean it
up.
1 - certificate logic moved to dedicated Certificates.zig
2 - Removed CurlDebugAllocator
- this makes setup easier, to Updater can just init libcurl directly
3 - Change Updater to be a single function
4 - Cache initialization is don in the Cache
5 - ?Cache => Cache{.kind = .noop}
Mime.isHttpToken already exists for the header name, added
Mime.isHttpHeaderValue which was extracted from het/Headers.zig. Now Config (
for validating CLI headers), net/Headers for script headers, and cdp/network.zig
all use the same validation methods.
Adding headers to an HTTP request was a bit awkward due to my desire to avoid
having an intermediate representation (e.g. an ArrayList(Header)). Going
straight to a curl slist avoids double-copying the headers (first to Zig, then
to curl).
But the CORS work (https://github.com/lightpanda-io/browser/pull/3002) showcases
that this micro-optimization simply isn't worth it, since it needs that
intermediate representation anyways.
And, this change isn't just for CORS. Headers have been a silly pain in the past
like unclear ownership, and messy APIs used in _a lot_ of places (WebBotAuth,
WebSocket, Fetch, ...)
This new approach stores headers on the transfer in an ArrayList. The API is:
```
const transfer = try client.newRequest(.{...}, owner);
{
errdefer transfer.deinit();
try transfer.addHeader("Over", "9000", .{});
}
try transfer.submit();
```
This:
1 - Eliminates ambiguity about errdefer cleanup responsibility
2 - Eliminates a bunch of stringZ concat that Frame, Config, CDP were doing
3 - Transfer.arena is now available for headers
1 - TEST_VERBOSE is now off by default
2 - There's a afterEach callback that is automatically run after each tests, it:
a - clears the log filters
b - resets the test arena
3 - LogFilter replace with
a - testing.silenceLog(&.{...scopes...}); to silence all logs for the given
scopes.
b - testing.expectLog(&.{...scopes}); to set log expectations, 1 per log.
The goal here isn't so much to expect logs (though, you can do that),
but rather to silence an expected # of logs, without silencing more.
Give accurate connectionId, connectionReused, initialPriority and securityState
values.
Always set `referrerPolicy` to `unsafe-url` as the most honest answer (we should
implement proper referrer policy!).
For workers, track the underlying frame_id so that it can be used for the
`documentURL` field.
1 - Blocks connection to "bad" ports
2 - new WebSocket('...'); mostly returns the instance, and errors are emitted
in a close event (1)
3 - More validation (no # allowed, ...);
4 - http -> ws, https -> wss (yes, per spec)
There's also some groundwork for better float16 support, but it turns out this
will require a new v8 build to complete. So I left the harmless float16 mapping
in (but, without the matching Float16Array, it doesn't help much).
(1) Some things cause an exception to be raised, but most don't
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
This piggybacks on the HttpClient rework (1) and makes incoming WebSocket
messages get delivered in a similar fashion. Namely, WebSocket events (connect,
data, close) are queued and only delivered at safe points. The goal is to make
sure JS is never run during a libcurl callback. This has historically caused
issues and currently requires the HttpClient to have various guards (e.g.
performing flag, dirty queue, etc..).
This change removes the last cases where JS could be executed within a libcurl
callback. Consequently, HttpClient is simplified. It no longer has a perfoming
flag nor a dirty queue nor ready queue.
(1) https://github.com/lightpanda-io/browser/pull/2889
Replaces layering with an inline request pipeline, and transfer queue. This is
meant to simplify the code, reduce footguns, and make future enhancements easier
to implement (e.g. speculative parsing (which requires streaming to fully
leverage)).
Previously, HttpClient implemented deferring as a layer which required special
pumping at various callsites (https://github.com/lightpanda-io/browser/pull/2855,
https://github.com/lightpanda-io/browser/pull/2843, ...). In this new approach,
deferring is built-into the HttpClient/Transfer's flow. Specifically, Transfers
now maintain a queue of events (start, header, data, end, err) which are
dispatched in HttpClient.tick. The result is that JS callbacks are never
executed in the same stack that initiated the I/O, without needing guards or any
external intervention.
tTwo other benefits come from this. The first is that reentrant libcurl is
eliminated. Instead of "libcurl -> callback", it's now "libcurl -> transfer
event queue THEN tick -> callback" (we don't have to wait until the NEXT tick, we
can just do it later in the tick). HttpClient still has to guard against libcurl
reentrancy, but only because of how WebSocket is implemented, and we should be
able to unify WebSockets to use an event queue too in a follow up PR (which will
eliminate a bunch of guard code).
The transfer queue should also be useful to re-implement streaming, since a
data chunk is just an event in the transfer's event queue. For now, I kept it
as a single buffered event to minimize the change. But since speculative parsing
depends on this, and speculative parsing seems to be the next major performance
tweak we can make, we need to re-introduce streaming.
The other change is the removal of all other layers in favor of a pipeline. This
works well with the existing Transfer.park mechanism, where a parked Transfer
can restart the pipeline for a transfer in an arbitrary point (not as fancy as
it sounds given how simple the flow is). The fallout from this is that we're no
longer creating/wrapping contexts and callbacks: whatever the request was
configured with is all we need.
Because of this, HttpClient.Response is removed. There are no intermediary
responses and no changing context, everything is just the Transfer.
A smaller change is the addition of newRequest + transfer.submit(). The one-shot
HttpClient.request and HttpClient.requestT still exist, but this explicit create
+ submit has some advantage. First, callers can use the transfer.arena (e.g.
Frame using the transfer's arena to set the Referrer header). Second, callers
can holds Transfer immediately, rather than waiting for their startCallback to
be fired. An abort on an XMLHttpRequest called before the start of the transfer
no longer silently fails.
Idea here is to skip re-parsing that happen for each connection; we already use BoringSSL, so we can take more advantage of it by directly mutating cert store of `SSL_CTX`.
This reverts recent(ish) changes to telemetry which moved it from its own thread
onto the main thread.
The downside is: we have an extra thread.
The upside is largely that Network.zig becomes drastically simpler and more
efficient. There's a bunch of machinery in Network.zig to support arbitrary
workers, of which Telemetry is the only one. There's also a lot of code to
support an optional multi and requests made to is. This is all removed.
Also, fetch, agent and mcp without a cdp server no longer even need to start
the network loop. And, it IS started (e.g. serve/cdp), there's no longer an
arbitrary 250ms wakeup on poll to progress workers. Nor can telemtry block CDP.
Telemetry's implementation itself was changed. The ring buffer was removed in
favor of a double-buffer arraylist. When telemetry is disabled, this saves
64Kb of memory. When it's enabled, it creates more allocator churn, but should
still use less memory in most cases (and never more). Finally, Telemetry is
given its own easy connection rather than using one out of the pool (which
workers would maybe like to use).
Largely aimed at /mimesniff/mime-types/parsing.*.html. Adds ~2000 passing cases.
The blob API is now a bit simpler (hence FormData/REquest/Response/WebSocket)
being touched, but more logic is now in Mime.zig to parse/validate MIME types.
Better escaping support, correct casing and handling edge cases with multiple
charset, or "charset" as a value, .etc...
Headers also hook into this to validate / normalize header name and values.
Network.setExtraHTTPHeaders applied caller headers with a plain
curl_slist append. Requests are seeded with a default `User-Agent:
Lightpanda/1.0`, so a caller-supplied User-Agent produced two
User-Agent entries; libcurl keeps the first and the override was
silently dropped (origins always saw `Lightpanda/1.0`). Other headers
worked because they had no default to collide with.
Add Headers.set, which replaces any existing header with the same
(case-insensitive) name before appending, and use it when applying the
CDP extra headers. This matches Chrome, where Network.setExtraHTTPHeaders
overrides default request headers.
Refs #2704
Previously, the CDP socket was added to the worker's multi and fully owned
by the worker. While this is simple, it introduced some issues:
1 - Cannot detect a disconnected client during JS processing ( for(;;) )
2 - A blocked worker can cause back-pressure that blocks the client. This can
cause a deadlock if the worker is blocked waiting for a CDP message
In addition to these 2 problems, there was 1 other serious CDP-related issue:
arbitrary CDP messages could be processed during JavaScript callback. For
example, a Worker calls importScripts while request interception is enabled,
this requires us to tick the HttpClient waiting for the interception response.
But, a client could sent Target.closeTarget, which we'd process and delete the
frame..all while importScripts is still blocked. Assuming importScripts unblocks
everything is a big UAF since the frame (and its workers) were cleared from
closeTarget.
The CDP socket is now read from the network (main) thread and an OTP-style
mailbox is used. The network thread posts message to the Worker's inbox and
signals it to wakeup. This solves #1 and #2. It doesn't directly solve the
reentrancy issue, but it provides the foundation. Specifically, in introduces
a queue for of CDP message and more control over when/how that queue is
processed. At "safe points" (Runner.tick, HttpClient.tick), any message can
be processed. But, when inside a JavaScript callback, we can process only non-
destructive/mutating message. Specifically, we can process only messages related
to request interception.
This commit fixes a few serious issues with the Websocket implementation.
1 - libcurl recursive api calls
Creating a Websocket instance from within a libcurl callback results in libcurl
failing with a RecursiveApiCall error. I fixed this more generally by adding a
`ready_queue` which connections can use when the `HttpClient` is performing
actions. Once `perform` ends, this new `ready_queue` is processed. There might
be a more holistic solution to this (we seem to run into RecursiveApiCall
everywhere), but since HttpClient is going through heavy changes, this seemed
like the smallest possible change to fix it.
2 - "load" blocking
Load and IdleNetwork notifications should not block on Websocket connections. To
solve this, `HttpClient` now ha `http_active` and `ws_active` to replace `active`.
Only `http_active` is used for things like "load" triggering.
3 - The above change made the Runner's job more complicated. It used to be
binary: you either have active connections or not. Now there are different types
of active connections. To keep it simple, and I think probably more correct,
the "done-ness" (based on the `wait` parameter) is now independent of active
(or not) network activity. If the page's `load_state == .complete`, then the
`wait == .done` is considered successful, whether or not we have active
connections.
4 - As a consequence of the above, and seemingly unrelated to all of these
changes, a number of html tests now use the "new" robust async framework. Most
of these tests were using the `testing.onload` (aka `testing.eventually`) which
had somewhat...unclear semantics. These tests passed more of a consequence of
how we processed a page and being very simple (e.g. just needing 1 micro or
macrotask tick). But `eventually` never worked for more complicated cases, and
the previous `testing.async` didn't work well. Now, the test runner waits for
.load (which, as per #3, can fire more aggressively), which caused many
`eventually` tests to fail. Moving these tests to the new `async` is more
robust and works with the new aggressive "load".
CIDRs prefixed with '-' are treated as allow rules that exempt matching
IPs from blocking. Allow rules take precedence over both
--block_private_networks and custom block CIDRs.
Example: --block_private_networks --block_cidrs -10.0.0.42/32
blocks all private ranges except 10.0.0.42.
Adds 3 new tests for allow-list behavior.
Block outbound HTTP requests to specified IP ranges before TCP handshake
using libcurl CURLOPT_OPENSOCKETFUNCTION callback. Fires after DNS
resolution, reads resolved IP directly from sockaddr, does bitwise CIDR
comparison. Fail-closed: unknown address families are blocked.
--block_private_networks blocks RFC1918, localhost, link-local, ULA.
--block_cidrs blocks additional comma-separated CIDRs.
IPv4-mapped IPv6 (::ffff:x.x.x.x) is unwrapped to prevent bypass.
Implement the XHR timeout property end-to-end: the JS-visible
getter/setter stores the value, send() passes it to the HTTP client,
and curl enforces it via CURLOPT_TIMEOUT_MS. On timeout, a `timeout`
event is dispatched instead of `error`, per the XHR spec.
Uses libcurl's websocket capabilities to add support for WebSocket.
Depends on https://github.com/lightpanda-io/zig-v8-fork/pull/167
Issue: https://github.com/lightpanda-io/browser/issues/1952
This is a WIP because it currently uses the same connection pool used for all
HTTP requests. It would be pretty easy for a page to starve the pool and block
any progress.
We previously stored the *Transfer inside of the easy's private data. We now
store the *Connection, and a Connection now has a `transport` field which is
a union for `http: *Transfer` or `websocket: *Websocket`.
Karl Seguin