Some status-codes should never have a body except for a single trailing blank
line. If we don't handle these, then we end up with a dirty connection in our
connection pool:
1 - read the header, but not the body
2 - put the connection back in the pool
3 - try to read the header, but actually get the body from #1
WPT /fetch/api/basic/response-null-body.any.html exercises this path and is
flaky (because it depends whether the request goes back out on a keep-alive
connection)..but for a given run,you'll almost always get 1-3 failures.
This commit processes the request, but tells libcurl not to re-use the
connection.
1 - Centralized cache-awareness into Cache and pulled header details out of
SqliteCache and HttpClient
2 - Added support for expires header
3 - Support caching more status types (but not all, since HttpClient would need
to be aware of what caching a 3xx/206 means)
4 - Revalidate cares about "not specified" vs "no-store" vs "stale"
(e.g. expires=0 means "stale", not fallthrough the last-modified logic)
Currently, our waitForImport blocks the caller, but continues to process any
already-queued requests. This can result in new JavaScript running while v8
is linking modules and that JavaScript can itself import a module that is
part of the still-being-linked graph.
waitForImport now works like a syncRequest. While HttpClient will continue to
make progress on all transfers, all other transfers will gate behind the waiting
one (using the same infrastructure that exists for syncRequest).
This crash was seen on an unknown srape URL.
This feature is significant because it adds support for processing an HTTP
request via the worker. It requires parking the connection and then having the
worker notify the loop when the response is ready. A lot of this was already
in-place (e.g. worker -> loop notification) but not quite do this extent.
A multipart form POST followed by a 302 changed to GET and lost its body,
but retained Content-Type: multipart/form-data. Servers could then try to
parse an absent multipart body and return 400. This was reproduced on a
local redirect server and a storefront localization flow.
Delete Fetch's request-body header names when rewriting to GET. Preserve
method and body on 307/308, rewrite only POST on 301/302, and preserve GET
and HEAD on 303 rather than rewriting every request indiscriminately.
Test method/header transitions and header handling through the existing
CDP fulfilled-redirect path.
Compiled patterns are useful anywhere someone else writes the pattern:
the adblock lists today, agent tool arguments next. The wrapper moves
out of the adblock directory and gains an options struct (case, UTF-8
subjects) and a compile diagnostic the caller can log or show. The App
owns the one context every consumer compiles through, the blocker
included.
https://github.com/lightpanda-io/browser/pull/3447 made better use of the
GlobalScope to simplify various callsites. This changes HttpClient.Owner to
contain the global_scope, rather than copying a handful of scope fields.
ScriptManager, XMLHttpRequest.zig, Fetch, Workers, etc. all take ownership (aka
dupe) the HTTP response from HTTPClient. They all have a headerCallback that
does something like:
```zig
if (transfer.getContentLength()) |cl| {
try self.body.ensureTotalCapacity(self.arena, cl);
}
```
But in all non-streaming cases (which is most cases), the HttpClient buffers
the response and only calls the headerCallback _after_ the body has been
received. Rather than relying on "Content-Length" header, the body buffer can
be sized to the exact body length. Why does this matter? Because the
Content-Length is the length of the body on the wire, and if the body is
compressed (like almost all .js files are), it will under-report the final
body length AND, because most callers are using an arena, the buffer growth
will retain more memory than it should.
This adds a `transfer.bodyLen()` method. Callers which dupe the body now use
this rather than the Content-Length (Content-Length is still used, e.g. for
XHR progress report).
This is a small step towards WebDriver supports (non-bidi). It allows creating
and deleting a BiDi "Session" (e.g. a worker). It also allows attaching a BiDi
driver to an HTTP-created BiDi session (the typical selenium startup flow).
This change unblocks the most basic setup/teardown of Selenium, so it still
isn't enough to actually use a Selenium script as-is. But it's significant
because it models a worker (thread) that isn't tied to a WebSocket, something we
haven't had before.
A consequence of a pure HTTP Session is that we don't have a clear cleanup
signal. There is no "the socket is disconnected". There's a new HTTP reaper
which kills HTTP Sessions after --http-session-timeout. It's expected that
drivers properly DELETE /session/:id. I imagine we're going to run into
--cdp-max-connections limits and need to tweak this code. BUT, this entire flow
is only enabled with --protocol webdriver, so it won't impact exiting CDP users.
Specifically to distinguish cross-site iframe navigation from top-level navigation, this PR reworks how `SameSite=Lax` moved. Since we're not checking if its a navigation alone now, the field for it is also renamed to `lax_allowed`.
navigator.languages now lists the Accept-Language tags in order, which is
Chrome's contract, instead of a second derivation from the locale tag that
disagreed with the header (--locale de-DE sent de-DE,de,en but reported
["de-DE","de"]). HttpHeaders.AcceptLanguage owns both shapes and is also
the CDP override type.
ICU canonicalizes a BCP 47 tag read from LC_ALL itself, script subtag
included, so the POSIX id conversion is gone; it dropped the script and
turned zh-Hans-TW into Traditional Chinese.
Also: the CDP handler keeps validateUserAgent's verdict instead of scanning
for Mozilla twice, the override is cleared unconditionally on context
teardown instead of through a flag, and the flags are sentinel strings so
Platform passes them to setenv without copying.
navigator.language was hard-coded to en-US and Accept-Language was a
constant, while Intl, toLocaleString and Date followed the host process
environment. On a de_DE host a page saw navigator.language === "en-US"
next to German number formatting, a mismatch fingerprinting scripts look
for, and the same page rendered differently across machines.
Follow Chrome's --lang rule: one configured tag drives navigator.language(s),
the Accept-Language header and ICU's default locale. --locale defaults to
en-US, so Intl is now en-US on every host instead of whatever LANG says.
--timezone sets the IANA zone Date and Intl use; absent, the host zone stays.
Both are applied by writing LC_ALL and TZ before V8 initializes ICU, which
reads them lazily. Platform.init is the first call in App.init, before any
thread exists, so setenv is safe there.
CDP Emulation.setUserAgentOverride.acceptLanguage, which Playwright sends
for its locale option, now overrides the header and navigator.languages
for the browser context's lifetime, mirroring the user agent override, and
applies even when the Mozilla user agent is refused.
Emulation.setLocaleOverride and setTimezoneOverride stay no-ops: changing
ICU's defaults at runtime needs new zig-v8-fork bindings.
The raw URL lives on `pattern.Url` next to the lowercased one, so
`pattern.matches` owns the `.regex` arm like every other kind and the
engine stops special-casing it. `Request.init` does the lowercasing
itself, as `fromHttp` already had to, instead of asking callers for
both spellings.
The regex shape now spells its uncertain marker as `*` and keeps
non-token literals as one marker, so it is read by the same
bounded-token loop as a plain pattern rather than a copy of it. The
quantifier parser keeps only what it uses: whether the atom may be
absent.
`Regex.matches` runs on a stack-first allocator: PCRE2 wants a match
data block and 20KB of backtracking frames per call, which no longer
touches the heap in the common case. A filter holds a pointer to its
regex, keeping `NetworkFilter` at its previous size.
Filter lists carry a few hundred rules written as JavaScript regex
literals (24 in EasyList, 165 in uBO's badware list); they parsed but
were dropped as unsupported. PCRE2 reads that syntax as-is, `\/` and
friends included, its compiled patterns are immutable so the one
blocker shared by every HTTP client thread can run them, and 10.48
ships a build.zig for 0.16, so it is wired like sqlite3.
`Regex.Context` routes every PCRE2 allocation through the blocker's
allocator, which puts the compiled patterns under the test runner's
leak detection, and caps match and depth so a broken pattern costs a
false negative rather than a stalled request. As in uBO, a regex
tests the raw URL with the case-insensitive flag unless `$match-case`.
Regex filters are still never tokenized: they ride the fallback bucket.
Driven by a handful of /fetch/ WPT tests, three changes:
1 - Prevent libcurl from auto-inserting a 'application/x-www-form-urlencoded"
content type for types we really have no content-type for.
2 - Include origin header in all requests that should have it. This is something
CorsGate was doing in most cases, but cors can be disabled, so the logic
is now moved to HttpClient.
3 - Expands on the header guard added in https://github.com/lightpanda-io/browser/pull/3374/
Adds more modes and more header check. Request.init also uses the header
guard now
Headers strip whitespace and guard against invalid characters
Headers iterator sorts and combines PER step, so that mutations are picked up.
Not the most efficient, but this is a short list, and how often are these being
iterated?
XMLHttpRequest: has its own extra header validation
Mime support for multiple Content-Type headers (or a header with multiple values)
last value wins.
Add BufferSource js bridge type that accepts various types -> []const u8 (at the
cost of losing the actual type). Useful in fetch, where various types can be a
body, but we only care about the underlying bytes (e.g. we didn't support A
rrayBufferView before this)
Refactored response body getters so that they all go through the same consume
and resolve logic
* Engine.Request.fromHttp(req, source_url, buffers) now builds the adblock request straight from HttpClient.Request.
* The URL is tokenized once per request (hashed into the Request, shared by all engines); capped at 128 tokens (same as adblock-rust).
* Document hostname longer than 253 bytes now skips adblocking.
Muki Kiboigo