https://github.com/lightpanda-io/browser/pull/3447 made better use of the
GlobalScope to simplify various callsites. This changes HttpClient.Owner to
contain the global_scope, rather than copying a handful of scope fields.
ScriptManager, XMLHttpRequest.zig, Fetch, Workers, etc. all take ownership (aka
dupe) the HTTP response from HTTPClient. They all have a headerCallback that
does something like:
```zig
if (transfer.getContentLength()) |cl| {
try self.body.ensureTotalCapacity(self.arena, cl);
}
```
But in all non-streaming cases (which is most cases), the HttpClient buffers
the response and only calls the headerCallback _after_ the body has been
received. Rather than relying on "Content-Length" header, the body buffer can
be sized to the exact body length. Why does this matter? Because the
Content-Length is the length of the body on the wire, and if the body is
compressed (like almost all .js files are), it will under-report the final
body length AND, because most callers are using an arena, the buffer growth
will retain more memory than it should.
This adds a `transfer.bodyLen()` method. Callers which dupe the body now use
this rather than the Content-Length (Content-Length is still used, e.g. for
XHR progress report).
This is a small step towards WebDriver supports (non-bidi). It allows creating
and deleting a BiDi "Session" (e.g. a worker). It also allows attaching a BiDi
driver to an HTTP-created BiDi session (the typical selenium startup flow).
This change unblocks the most basic setup/teardown of Selenium, so it still
isn't enough to actually use a Selenium script as-is. But it's significant
because it models a worker (thread) that isn't tied to a WebSocket, something we
haven't had before.
A consequence of a pure HTTP Session is that we don't have a clear cleanup
signal. There is no "the socket is disconnected". There's a new HTTP reaper
which kills HTTP Sessions after --http-session-timeout. It's expected that
drivers properly DELETE /session/:id. I imagine we're going to run into
--cdp-max-connections limits and need to tweak this code. BUT, this entire flow
is only enabled with --protocol webdriver, so it won't impact exiting CDP users.
navigator.languages now lists the Accept-Language tags in order, which is
Chrome's contract, instead of a second derivation from the locale tag that
disagreed with the header (--locale de-DE sent de-DE,de,en but reported
["de-DE","de"]). HttpHeaders.AcceptLanguage owns both shapes and is also
the CDP override type.
ICU canonicalizes a BCP 47 tag read from LC_ALL itself, script subtag
included, so the POSIX id conversion is gone; it dropped the script and
turned zh-Hans-TW into Traditional Chinese.
Also: the CDP handler keeps validateUserAgent's verdict instead of scanning
for Mozilla twice, the override is cleared unconditionally on context
teardown instead of through a flag, and the flags are sentinel strings so
Platform passes them to setenv without copying.
navigator.language was hard-coded to en-US and Accept-Language was a
constant, while Intl, toLocaleString and Date followed the host process
environment. On a de_DE host a page saw navigator.language === "en-US"
next to German number formatting, a mismatch fingerprinting scripts look
for, and the same page rendered differently across machines.
Follow Chrome's --lang rule: one configured tag drives navigator.language(s),
the Accept-Language header and ICU's default locale. --locale defaults to
en-US, so Intl is now en-US on every host instead of whatever LANG says.
--timezone sets the IANA zone Date and Intl use; absent, the host zone stays.
Both are applied by writing LC_ALL and TZ before V8 initializes ICU, which
reads them lazily. Platform.init is the first call in App.init, before any
thread exists, so setenv is safe there.
CDP Emulation.setUserAgentOverride.acceptLanguage, which Playwright sends
for its locale option, now overrides the header and navigator.languages
for the browser context's lifetime, mirroring the user agent override, and
applies even when the Mozilla user agent is refused.
Emulation.setLocaleOverride and setTimezoneOverride stay no-ops: changing
ICU's defaults at runtime needs new zig-v8-fork bindings.
The raw URL lives on `pattern.Url` next to the lowercased one, so
`pattern.matches` owns the `.regex` arm like every other kind and the
engine stops special-casing it. `Request.init` does the lowercasing
itself, as `fromHttp` already had to, instead of asking callers for
both spellings.
The regex shape now spells its uncertain marker as `*` and keeps
non-token literals as one marker, so it is read by the same
bounded-token loop as a plain pattern rather than a copy of it. The
quantifier parser keeps only what it uses: whether the atom may be
absent.
`Regex.matches` runs on a stack-first allocator: PCRE2 wants a match
data block and 20KB of backtracking frames per call, which no longer
touches the heap in the common case. A filter holds a pointer to its
regex, keeping `NetworkFilter` at its previous size.
Filter lists carry a few hundred rules written as JavaScript regex
literals (24 in EasyList, 165 in uBO's badware list); they parsed but
were dropped as unsupported. PCRE2 reads that syntax as-is, `\/` and
friends included, its compiled patterns are immutable so the one
blocker shared by every HTTP client thread can run them, and 10.48
ships a build.zig for 0.16, so it is wired like sqlite3.
`Regex.Context` routes every PCRE2 allocation through the blocker's
allocator, which puts the compiled patterns under the test runner's
leak detection, and caps match and depth so a broken pattern costs a
false negative rather than a stalled request. As in uBO, a regex
tests the raw URL with the case-insensitive flag unless `$match-case`.
Regex filters are still never tokenized: they ride the fallback bucket.
Driven by a handful of /fetch/ WPT tests, three changes:
1 - Prevent libcurl from auto-inserting a 'application/x-www-form-urlencoded"
content type for types we really have no content-type for.
2 - Include origin header in all requests that should have it. This is something
CorsGate was doing in most cases, but cors can be disabled, so the logic
is now moved to HttpClient.
3 - Expands on the header guard added in https://github.com/lightpanda-io/browser/pull/3374/
Adds more modes and more header check. Request.init also uses the header
guard now
Headers strip whitespace and guard against invalid characters
Headers iterator sorts and combines PER step, so that mutations are picked up.
Not the most efficient, but this is a short list, and how often are these being
iterated?
XMLHttpRequest: has its own extra header validation
Mime support for multiple Content-Type headers (or a header with multiple values)
last value wins.
Add BufferSource js bridge type that accepts various types -> []const u8 (at the
cost of losing the actual type). Useful in fetch, where various types can be a
body, but we only care about the underlying bytes (e.g. we didn't support A
rrayBufferView before this)
Refactored response body getters so that they all go through the same consume
and resolve logic
* Engine.Request.fromHttp(req, source_url, buffers) now builds the adblock request straight from HttpClient.Request.
* The URL is tokenized once per request (hashed into the Request, shared by all engines); capped at 128 tokens (same as adblock-rust).
* Document hostname longer than 253 bytes now skips adblocking.
Currently, the HttpClient owns the inbox and its borrowed by the Link. This is
a bit backwards, but it also means that we can't eagerly create a Link: the
Link needs the inbox, so it needs the HttpClient, which is created by the
Browser (which creates an Isolate).
Remember, the Inbox is one of the few things shared between the main thread
and the worker, so either end can own it and the other can borrow it.
This switches the ownership so that the HttpClient now borrows the Inbox from
the Server's side of the Link (the WebSocket).
The main goal of this change is to prepare for more advanced HTTP WebDriver
flows. The more we can create _without_ a Browser, the fewer edge cases we have
to deal with (Browser because it's expensive and has to be created on the
Worker thread due to how V8::Isolate works).
Adds resource timing, e.g. `performance.getEntriesByType("resource")`.
The `resource-timing` WPT category is currently at 4.6%, and this is a first
step at improving it. It also hopefully fixes https://github.com/lightpanda-io/browser/issues/3359
This is more complicated than I thought because there's a "Timing-Allow-Origin"
header that a server can include which hides some of the data if the request
doesn't come from the listed origin. And that, of course, interacts with
redirects.
(The DOMException change is seemingly random, but it came up in one of the WPT
cases I was looking at).
Our redirect handling was "optimized" to re-use the same easy connection, at the
cost of circumventing the entire pipeline. A redirect would not check the new
target's robots.txt, rate limit, CORS, Web Auth, ...
The new code simply restarts the pipeline (with a discriminator tag so that
things like CDP can tell the difference between a redirected request and a new
one).
If you look at https://github.com/lightpanda-io/browser/pull/3293, you'll see
a relatively contained change that has to touch over 20 files. The issue is that
every HttpClient.newRequest needs to provide a lot of data. But `newRequest`
takes a 2nd parameter: the HttpClient.Owner. If we make that Owner a little
smarter, we can start to remove some of the individual fields needed in
newRequest. For example, we can still allow a callsite to pass frame_id but,
by default, we can use the owner's frame_id (which is what we want in most
cases).
The site for cookies were computed from the immediate parent `Frame`, which would allow sending a cookie that's `SameSite=Strict` from 2 levels deep under. Directly from RFC6265bis, this PR essentially implements (except for step 4, we skip host-less ancestors):
Given a Document (document), the following algorithm returns its
"site for cookies":
1. Let top-document be the active document in document's navigable's
top-level traversable.
2. Let top-origin be the origin of top-document's URI if top-
document's sandboxed origin browsing context flag is set, and
top-document's origin otherwise.
3. Let documents be a list consisting of the active documents of
document's inclusive ancestor navigables.
4. For each item in documents:
1. Let origin be the origin of item's URI if item's sandboxed
origin browsing context flag is set, and item's origin
otherwise.
2. If origin is not same-site with top-origin, return an origin
set to an opaque origin.
5. Return top-origin.
window.stop() is less destructive than other mechanisms we have. For one, it
seems largely isolated to pending or inflight HTTP requests. For anther, it
keeps the page intact.
To achieve this, HttpClient gains an `cancelRequests` which is a gentler version
of `abortOwner`. It cancels inflight/pending HTTP requests, which results in
error callbacks (not shutdown callbacks) firing.
Just like https://github.com/lightpanda-io/browser/pull/3189 I ran into the
problem that I couldn't distinguish between an HTTP request that was canceled
because of user-action (e.g. calling window.stop(), or xhr.abort()) and an HTTP
request that was internally aborted. These now have distinct errors/flows so
that we can present the correct state. Most places that aborted now all
transfer.cancel() which results in a distinct `error.TransferCanceled` (some
places still abort -> `error.Abort`). It should be possible to revisit 3189 now.
The CDP "Page.stopLoading" now hooks into this new behavior. Fixes
https://github.com/lightpanda-io/browser/issues/3351
Muki Kiboigo