Requests issued to an origin while its first connection is still
handshaking each opened their own socket, up to --http-max-host-open,
because curl only learns from ALPN whether the origin multiplexes. With
pipewait they wait for that answer and share one h2 connection.
Fixture: 12 fetch() calls to a fresh cdnjs (h2) origin, release build.
new TCP+TLS connections 6 -> 1
in-page time to last resp ~285 ms -> ~105-135 ms
H1-only origins are unchanged in connection count; their first burst
waits one handshake before fanning out.
https://github.com/lightpanda-io/browser/pull/3447 made better use of the
GlobalScope to simplify various callsites. This changes HttpClient.Owner to
contain the global_scope, rather than copying a handful of scope fields.
The RobotStore is shared by all Browsers. While every browser has a single
flight to prevent duplicate requests to the same robots.txt, that's limited to
that specific browser. So, 2 browsers can ask for the same robots.txt and then
put try to store the result. The RobotStore _is_ thread safe, but it's a simple
last-one-wins which overrites the previous record, without freeing either the
key or value.
This replaces the last-write-wins with a first-write-wins, avoiding the leak.
ScriptManager, XMLHttpRequest.zig, Fetch, Workers, etc. all take ownership (aka
dupe) the HTTP response from HTTPClient. They all have a headerCallback that
does something like:
```zig
if (transfer.getContentLength()) |cl| {
try self.body.ensureTotalCapacity(self.arena, cl);
}
```
But in all non-streaming cases (which is most cases), the HttpClient buffers
the response and only calls the headerCallback _after_ the body has been
received. Rather than relying on "Content-Length" header, the body buffer can
be sized to the exact body length. Why does this matter? Because the
Content-Length is the length of the body on the wire, and if the body is
compressed (like almost all .js files are), it will under-report the final
body length AND, because most callers are using an arena, the buffer growth
will retain more memory than it should.
This adds a `transfer.bodyLen()` method. Callers which dupe the body now use
this rather than the Content-Length (Content-Length is still used, e.g. for
XHR progress report).
This is a small step towards WebDriver supports (non-bidi). It allows creating
and deleting a BiDi "Session" (e.g. a worker). It also allows attaching a BiDi
driver to an HTTP-created BiDi session (the typical selenium startup flow).
This change unblocks the most basic setup/teardown of Selenium, so it still
isn't enough to actually use a Selenium script as-is. But it's significant
because it models a worker (thread) that isn't tied to a WebSocket, something we
haven't had before.
A consequence of a pure HTTP Session is that we don't have a clear cleanup
signal. There is no "the socket is disconnected". There's a new HTTP reaper
which kills HTTP Sessions after --http-session-timeout. It's expected that
drivers properly DELETE /session/:id. I imagine we're going to run into
--cdp-max-connections limits and need to tweak this code. BUT, this entire flow
is only enabled with --protocol webdriver, so it won't impact exiting CDP users.
An alternation or a repeat keeps only whether its text may start and
end with a token character, and read that off one marker. An optional
non-token stretch there (`\/?x`) was taken as a definite non-token,
so `\/ads(\/?x|\/y)` was filed under "ads" while `/adsx` carries no
such token. What follows the stretch answers now.
navigator.languages now lists the Accept-Language tags in order, which is
Chrome's contract, instead of a second derivation from the locale tag that
disagreed with the header (--locale de-DE sent de-DE,de,en but reported
["de-DE","de"]). HttpHeaders.AcceptLanguage owns both shapes and is also
the CDP override type.
ICU canonicalizes a BCP 47 tag read from LC_ALL itself, script subtag
included, so the POSIX id conversion is gone; it dropped the script and
turned zh-Hans-TW into Traditional Chinese.
Also: the CDP handler keeps validateUserAgent's verdict instead of scanning
for Mozilla twice, the override is cleared unconditionally on context
teardown instead of through a flag, and the flags are sentinel strings so
Platform passes them to setenv without copying.
navigator.language was hard-coded to en-US and Accept-Language was a
constant, while Intl, toLocaleString and Date followed the host process
environment. On a de_DE host a page saw navigator.language === "en-US"
next to German number formatting, a mismatch fingerprinting scripts look
for, and the same page rendered differently across machines.
Follow Chrome's --lang rule: one configured tag drives navigator.language(s),
the Accept-Language header and ICU's default locale. --locale defaults to
en-US, so Intl is now en-US on every host instead of whatever LANG says.
--timezone sets the IANA zone Date and Intl use; absent, the host zone stays.
Both are applied by writing LC_ALL and TZ before V8 initializes ICU, which
reads them lazily. Platform.init is the first call in App.init, before any
thread exists, so setenv is safe there.
CDP Emulation.setUserAgentOverride.acceptLanguage, which Playwright sends
for its locale option, now overrides the header and navigator.languages
for the browser context's lifetime, mirroring the user agent override, and
applies even when the Mozilla user agent is refused.
Emulation.setLocaleOverride and setTimezoneOverride stay no-ops: changing
ICU's defaults at runtime needs new zig-v8-fork bindings.
The raw URL lives on `pattern.Url` next to the lowercased one, so
`pattern.matches` owns the `.regex` arm like every other kind and the
engine stops special-casing it. `Request.init` does the lowercasing
itself, as `fromHttp` already had to, instead of asking callers for
both spellings.
The regex shape now spells its uncertain marker as `*` and keeps
non-token literals as one marker, so it is read by the same
bounded-token loop as a plain pattern rather than a copy of it. The
quantifier parser keeps only what it uses: whether the atom may be
absent.
`Regex.matches` runs on a stack-first allocator: PCRE2 wants a match
data block and 20KB of backtracking frames per call, which no longer
touches the heap in the common case. A filter holds a pointer to its
regex, keeping `NetworkFilter` at its previous size.
A regex filter rode the fallback bucket, which every request pays for.
uBO reads a token out of one by flattening the pattern into a string
where literal characters stay and anything else becomes a marker that
says only whether a token character may be there, then taking the
alphanumeric runs bounded on both sides by something that is surely
not one; `tokenizableStrFromRegex`, ported here as `RegexShape`.
One departure: a positive lookaround becomes a marker rather than
being inlined, since a token must never come from text the regex does
not consume. Anything the flattening does not follow yields no token
at all, which is never wrong.
On the 79 regex rules the parser accepts across EasyList, EasyPrivacy
and uBO's lists, 72 now land in a bucket.
Filter lists carry a few hundred rules written as JavaScript regex
literals (24 in EasyList, 165 in uBO's badware list); they parsed but
were dropped as unsupported. PCRE2 reads that syntax as-is, `\/` and
friends included, its compiled patterns are immutable so the one
blocker shared by every HTTP client thread can run them, and 10.48
ships a build.zig for 0.16, so it is wired like sqlite3.
`Regex.Context` routes every PCRE2 allocation through the blocker's
allocator, which puts the compiled patterns under the test runner's
leak detection, and caps match and depth so a broken pattern costs a
false negative rather than a stalled request. As in uBO, a regex
tests the raw URL with the case-insensitive flag unless `$match-case`.
Regex filters are still never tokenized: they ride the fallback bucket.
Driven by a handful of /fetch/ WPT tests, three changes:
1 - Prevent libcurl from auto-inserting a 'application/x-www-form-urlencoded"
content type for types we really have no content-type for.
2 - Include origin header in all requests that should have it. This is something
CorsGate was doing in most cases, but cors can be disabled, so the logic
is now moved to HttpClient.
3 - Expands on the header guard added in https://github.com/lightpanda-io/browser/pull/3374/
Adds more modes and more header check. Request.init also uses the header
guard now
Headers strip whitespace and guard against invalid characters
Headers iterator sorts and combines PER step, so that mutations are picked up.
Not the most efficient, but this is a short list, and how often are these being
iterated?
XMLHttpRequest: has its own extra header validation
Mime support for multiple Content-Type headers (or a header with multiple values)
last value wins.
Add BufferSource js bridge type that accepts various types -> []const u8 (at the
cost of losing the actual type). Useful in fetch, where various types can be a
body, but we only care about the underlying bytes (e.g. we didn't support A
rrayBufferView before this)
Refactored response body getters so that they all go through the same consume
and resolve logic
Eager tokenization is ~7% of a ~2.1 µs match. A lazy fill saved nothing because every request reaches the first engine. Streaming from scratch per engine cost +10% capped and +16% uncapped, and stayed +10% even with exception gating. The 128 cap changed 1 verdict in 242,908. So we went with hybrid approach: no token is lost, and 97% of URLs still pay one tokenization.
* `@@…$important` -> `error.InvalidOption`
* `||host`, `||host^`, `||host|` (and bare `host|`/`|host` lines) all read as "hostname or subdomain" and land in the trie when option-free.
* Wildcard trimming now follows uBO's pointless-wildcard rules.
* Engine.Request.fromHttp(req, source_url, buffers) now builds the adblock request straight from HttpClient.Request.
* The URL is tokenized once per request (hashed into the Request, shared by all engines); capped at 128 tokens (same as adblock-rust).
* Document hostname longer than 253 bytes now skips adblocking.
Muki Kiboigo