Some status-codes should never have a body except for a single trailing blank
line. If we don't handle these, then we end up with a dirty connection in our
connection pool:
1 - read the header, but not the body
2 - put the connection back in the pool
3 - try to read the header, but actually get the body from #1
WPT /fetch/api/basic/response-null-body.any.html exercises this path and is
flaky (because it depends whether the request goes back out on a keep-alive
connection)..but for a given run,you'll almost always get 1-3 failures.
This commit processes the request, but tells libcurl not to re-use the
connection.
Requests issued to an origin while its first connection is still
handshaking each opened their own socket, up to --http-max-host-open,
because curl only learns from ALPN whether the origin multiplexes. With
pipewait they wait for that answer and share one h2 connection.
Fixture: 12 fetch() calls to a fresh cdnjs (h2) origin, release build.
new TCP+TLS connections 6 -> 1
in-page time to last resp ~285 ms -> ~105-135 ms
H1-only origins are unchanged in connection count; their first burst
waits one handshake before fanning out.
Significant rework of the CDP/BiDi server. There are two main changes:
1 - poll replaced with EPoll/Kqueue (1)
2 - make http serving a first class citizen
The change from poll -> epoll/kqueue isn't performance driven, it's just about
tighter code. Both epoll and kqueue let you associate arbitrary data with a
socket, so we don't need to keep arrays in sync in order to associate a socket
with a CDP by index. They both provide some event/notification mechanism, which
is cleaner than the pipe required by poll.
The poll -> epoll/kqueue change could almost have been mechanical. Making HTTP
a first class citizen is the more significant of the two changes
In `main`, a new connection always spawns a thread and, until does its own
little read loop until the connection is upgraded. This is not efficient, it
uses up a connection slot, and it's inconsistent with the final WebSocket
connection which _is_ polled off the main loop. Using up a slot means that
keepalive isn't possible, else HTTP connections would quickly use up all
available slots/threads.
This commit parses and serves HTTP requests on the main thread (safe
because none of the processing is blocking). The approach is better streamlined
for HTTP requests which never upgrade (/metrics, WebDriver) without causing
any performance overhead for those that do. It simplifies some things (e.g. an
"http" socket or a "websocket" socket is monitored and read in a similar manner
(on the main loop)). It makes other things more complicated; the flow is no
longer accept -> spawn -> upgrade -> websocket loop. It's loop -> accept -> loop
-> process -> (http | ws).
This is built ontop of the BiDi branch because (a) WebDriver is what needs
better HTTP support and (b) some of the more mechanical changes already exist
in that branch (e.g. src/cdp/, src/server.zig -> src/server/*)
(1) kqueue landing in 2 commits from now on this branch.
Adds resource timing, e.g. `performance.getEntriesByType("resource")`.
The `resource-timing` WPT category is currently at 4.6%, and this is a first
step at improving it. It also hopefully fixes https://github.com/lightpanda-io/browser/issues/3359
This is more complicated than I thought because there's a "Timing-Allow-Origin"
header that a server can include which hides some of the data if the request
doesn't come from the listed origin. And that, of course, interacts with
redirects.
(The DOMException change is seemingly random, but it came up in one of the WPT
cases I was looking at).
CURLMOPT_MAXCONNECTS was never set, so libcurl used its default of 4x the
number of easy handles currently attached to the multi. Handles are added
and removed per transfer, so between page loads that default collapses to
roughly zero and every cached connection is evicted: revisiting a host
after browsing elsewhere re-paid connect + TLS every time.
Measured over 25 navigations across 5 sites (5 rounds, ReleaseFast), median
warm navigation drops from 0.73s to 0.42s on news.ycombinator.com and from
0.71s to 0.42s on github.com; total navigation time 11.1s -> 7.3s. Loading
one host repeatedly was already fast and is unchanged; the win is on
cross-site browsing, which is what agents and crawlers actually do.
https://github.com/lightpanda-io/browser/issues/3348
When set to 1.1, libcurl is configured to only offer HTTP 1.1. By default, or
when set to "auto", it's up to libcurl to decide how to connect. This maps to
libcurl's CURL_HTTP_VERSION_1_1 and CURL_HTTP_VERSION_NONE.
LP.configureCDP now takes an `httpVersion` field which can be "1.1" or "auto"
to control that specific browser session. Ideally this is called prior to any
navigation.
Zig 0.17 removes @cImport in favor of translate-c steps in the build
system. Each library's link function now creates an addTranslateC step
exposed as a module import: curl and isocline translate their headers
straight from the dependency tree, while sqlite3 uses the artifact's
emitted include tree since the amalgamation lives in a sub-dependency.
Terminal.zig and prompt_assist.zig previously instantiated two
independent @cImport namespaces for isocline; they now share one
translated module, so its types are identical across both files.
Give accurate connectionId, connectionReused, initialPriority and securityState
values.
Always set `referrerPolicy` to `unsafe-url` as the most honest answer (we should
implement proper referrer policy!).
For workers, track the underlying frame_id so that it can be used for the
`documentURL` field.
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
Removes lying dead code here and simplifies `curl_easy_setopt` type checking. Also marks passed callbacks as with C calling convention in order to get away from additional function wrapping.
Implements `url_resolve_with_encoding` and `url_resolve_without_encoding` in Rust; that way, we don't pay the cost of extra `Box`es we allocate during resolving.
I threw Claude at WPT's /WebCryptoAPI/ which represents a significant number of
failing cases.
There were a few very low-hanging fruit, like enabling CryptoKey on Worker and
supporting `{name: string}` in addition to just `string` for some functions.
Some of these "fixes" just handle the error case / validation, which tends to
represent a greater number of WPT cases, e.g. some of the importKey algorithms
aren't fully supported, but they still validate the input and return the correct
errors.
To that end, while there should be considerably more passing cases (~42K), some
of these changes merely unlocked more failing cases, so our total case count
should go up.
Claude's summary:
1. digest {name} object fix (16→80)
2. Symmetric importKey/exportKey/generateKey (AES, HMAC; raw + jwk) + CryptoKey
accessors + DataError
3. EC/OKP importKey usage validation (failure-path)
4. PBKDF2 + HKDF deriveBits, then deriveKey
5. ECDH generateKey + import (spki/pkcs8) + deriveBits/deriveKey
6. AES encrypt/decrypt (CBC/CTR/GCM)
This is a bit all over the place.
1 - Replace libidn2 with rust-idna. It looks like there are different idna
profiles, and rust-idna (from the servo project) implements the whatwg
one. libidn2 would be too strict in some cases and not strict enough in
others. (Gemini says I could use libidn2 for this, but what it suggested
didn't work, and I couldn't figure it out myself, and claude insisted it
_did not_ have the correct implementation for what we want).
2 - We previously only ran a URL through idna if it wasn't ascii. Turns out
we also need to run it if there's a "xn--" (aka, an IDNA ACE prefix) in
there. This helps us pass hundreds of WPT cases, and it's pretty cheap.
3 - Implement more of the Area WebAPI. Mostly copied from Anchor.
4 - Add username/password accessor to Anchor/Area
5 - window.open validates the URL (i.e. tries to resolve it and handles the
error)
6 - Invalid idna conversion maps to a TypeError
7 - Cleanup closed popups on the next tick (like destroyed pages), rather than
at an interval or on shutdown. This one seems unrelated, but some of these
tests are opening hundreds (thousands?) of popups and then closing them.
Previously, the CDP socket was added to the worker's multi and fully owned
by the worker. While this is simple, it introduced some issues:
1 - Cannot detect a disconnected client during JS processing ( for(;;) )
2 - A blocked worker can cause back-pressure that blocks the client. This can
cause a deadlock if the worker is blocked waiting for a CDP message
In addition to these 2 problems, there was 1 other serious CDP-related issue:
arbitrary CDP messages could be processed during JavaScript callback. For
example, a Worker calls importScripts while request interception is enabled,
this requires us to tick the HttpClient waiting for the interception response.
But, a client could sent Target.closeTarget, which we'd process and delete the
frame..all while importScripts is still blocked. Assuming importScripts unblocks
everything is a big UAF since the frame (and its workers) were cleared from
closeTarget.
The CDP socket is now read from the network (main) thread and an OTP-style
mailbox is used. The network thread posts message to the Worker's inbox and
signals it to wakeup. This solves #1 and #2. It doesn't directly solve the
reentrancy issue, but it provides the foundation. Specifically, in introduces
a queue for of CDP message and more control over when/how that queue is
processed. At "safe points" (Runner.tick, HttpClient.tick), any message can
be processed. But, when inside a JavaScript callback, we can process only non-
destructive/mutating message. Specifically, we can process only messages related
to request interception.
Links to libidn2 and builds libcurl with it. This makes libcurl work, and by
extension browser, work on international domain names, e.g.
zig build run -- fetch "https://räksmörgås.se/"
With it available, we can use it in our WebAPIs which should also support these
domains, e.g:
testing.expectEqual('xn--rksmrgs-5wao1o.se', new URL('https://räksmörgås.se').hostname);
There is more integration to be done here, but this is a first step.
claude wrote all of the build.zig code.
I don't have a strong opinion about this feature, I just dislike that our WPT
/url/* tests are at 1704 / 9095 and, this is the biggest chunk (although, this
specific commit just does the basic integration and probably won't fix too many
WPT cases directly).
Block outbound HTTP requests to specified IP ranges before TCP handshake
using libcurl CURLOPT_OPENSOCKETFUNCTION callback. Fires after DNS
resolution, reads resolved IP directly from sockaddr, does bitwise CIDR
comparison. Fail-closed: unknown address families are blocked.
--block_private_networks blocks RFC1918, localhost, link-local, ULA.
--block_cidrs blocks additional comma-separated CIDRs.
IPv4-mapped IPv6 (::ffff:x.x.x.x) is unwrapped to prevent bypass.
Uses libcurl's websocket capabilities to add support for WebSocket.
Depends on https://github.com/lightpanda-io/zig-v8-fork/pull/167
Issue: https://github.com/lightpanda-io/browser/issues/1952
This is a WIP because it currently uses the same connection pool used for all
HTTP requests. It would be pretty easy for a page to starve the pool and block
any progress.
We previously stored the *Transfer inside of the easy's private data. We now
store the *Connection, and a Connection now has a `transport` field which is
a union for `http: *Transfer` or `websocket: *Websocket`.
I've been thinking the implementation here is messy (ever since we added support for it) and thought it would be better to separate each algorithm to their respective files in order to maintain in a long run. `digest` is also refactored to prefer libcrypto instead of std.
Karl Seguin