Network has accumulated a bit of bagged. It knows a lot about certificates, it
knows a lot of the cache. I have plans to expand Network, and wanted to clean it
up.
1 - certificate logic moved to dedicated Certificates.zig
2 - Removed CurlDebugAllocator
- this makes setup easier, to Updater can just init libcurl directly
3 - Change Updater to be a single function
4 - Cache initialization is don in the Cache
5 - ?Cache => Cache{.kind = .noop}
Add an optional per-host rate limit. This currently only applies to the top-
level navigation. This makes it simpler to implement and simpler to reason
about. The full load of a page is only ever delayed at its head, not
sporadically through the page loading.
The use-case where a RateLimiter is most useful is when the browser is crawling
multiple pages of the same site, and in that case, the top-level rate limit
still applies some degree of limit to any linked resources (e.g. a JS on a
different host).
`--http-nav-delay` is the delay, in milliseconds, to apply to top level
navigates per host. Currently defaults to 0 (disabled).
`--http-nav-burst` is the burst allowed per host. Defaults to 1 (has no impact
when `--http-nav-delay` is disabled).
1-
Per spec, dictionary objects should be loaded in lexicographical order (a). So
the following always pushes `endings` before `type`
```js
new Blob([], {
get type() { order.push('type'); return ''; },
get endings() { order.push('endings'); return 'transparent'; },
});
```
This requirement is now enforced by a comptime check. So, a lot of files were
updated to satisfy this requirement.
2-
Blob parts now work with the @@iterator protocol. This is done more generically
(via `value.iterator())` but is currently only used by Blob (and File). Other
types that need this (e.g. URLSearchParams) can be done in a follow up.
3-
Add support for Blob.textStream
4-
Reject XHR/Fetch requests on blob URLs if the method isn't GET
5-
Strip #hash from blob URL when doing lookup
All of this fixes ~175 FileAPI WPT cases, though it's possible #1 (the
dictionary ordering) helps with a few others.
(a) https://webidl.spec.whatwg.org/#js-dictionary
Builds ontop of https://github.com/lightpanda-io/browser/pull/3200 to centralize
header enforcement and standardize merge vs overwrite header logic.
The API is still a `setHeader` and `appendHeader`, with a source, but set/append
both are thin wrappers around private `putHeader`. putHeader blocks overwriting
restricted headers (user-agent). The `source` acts as a priority (ordered enum)
which further restricts the header AND depending on whether set or append were
called, controls if the value is overwritten or appended to.
3200 had an always-append which can cause problems, e.g. a script setting
Accept-Language: fr would have results in the value being appended to the
default, e.g.: 'en-US,en;q=0.9, fr'.
On a redirect, recalculate the Referer header based on the new target.
Also, allow SVG anchors to be clicked (doesn't seem related, but it came up
in referrer-policy WPT tests).
Extracted from https://github.com/lightpanda-io/browser/pull/3122. Sends RI for
redirect. Also, on a continueRequest which does redirect, restores the original
headers (continueRequest's headers are only valid for a single request).
To make this work in all drivers, CDP now decouples the transfer_id from the
intercept_id. Each unique request gets a distinct intercept_id which is managed
in CDP (with a intercept_id -> transfer_id mapping).
When we know the precise final length, prefer ensureTotalCapacityPrecise over
ensureTotalCapacity. The latter goes through `growCapacity` which will allocate
~1.5x padding.
We already had special handling for BoringSSL's RecvError on improperly closed
TLS connection. This moves the handling up, so that redirect handling is
covered by it too.
Adding headers to an HTTP request was a bit awkward due to my desire to avoid
having an intermediate representation (e.g. an ArrayList(Header)). Going
straight to a curl slist avoids double-copying the headers (first to Zig, then
to curl).
But the CORS work (https://github.com/lightpanda-io/browser/pull/3002) showcases
that this micro-optimization simply isn't worth it, since it needs that
intermediate representation anyways.
And, this change isn't just for CORS. Headers have been a silly pain in the past
like unclear ownership, and messy APIs used in _a lot_ of places (WebBotAuth,
WebSocket, Fetch, ...)
This new approach stores headers on the transfer in an ArrayList. The API is:
```
const transfer = try client.newRequest(.{...}, owner);
{
errdefer transfer.deinit();
try transfer.addHeader("Over", "9000", .{});
}
try transfer.submit();
```
This:
1 - Eliminates ambiguity about errdefer cleanup responsibility
2 - Eliminates a bunch of stringZ concat that Frame, Config, CDP were doing
3 - Transfer.arena is now available for headers
1 - Add a metric to track the number of inlight arenas from the pool
2 - Script now use 2 arenas:
- An initial (small) one for the script
- A sized one for the body
Should result in less pressure on our limited .large arenas
3 - DOMPoint and DOMPointRO are now arena free (they live on the slab only)
4 - TextDecoder no longer accumulate garbage in its arena
5 - Response object is much better at picking its arena size, rather than just
using a .large
Introduces a ArenaAllocator wrappre (lp.Arena). This is groundwork for better
memory tracking and reporting memory usage to v8. This is almost purely a
mechanical change to lay a foundation for a follow up PR that will address
https://github.com/lightpanda-io/browser/issues/3027
Some code became a bit leaner: a pooled arena can release itself (it has a
reference to the ArenaPool).
Some code became uglier: The Frame has a `_local_arena: *lp.Arena` and a
`local_arena: Allocator` (same with call_arena, and same with a few other types)
so that consumers aren't impacted (they continue to use `frame.local_arena`).
Zig 0.16 changed clocks/timestamps. Our migration to Zig 0.16 took the path of
least resistance. For example, we kept 'monotonic' and 'real' even though Zig
0.16 renamed them to 'boot' and 'clock'
This commit tries to standardize all timestamp usage to: lp.datetime.timestamp
and lp.datetime.milliTimestamp using the new zig 0.16 names ('boot' and 'clock')
Give accurate connectionId, connectionReused, initialPriority and securityState
values.
Always set `referrerPolicy` to `unsafe-url` as the most honest answer (we should
implement proper referrer policy!).
For workers, track the underlying frame_id so that it can be used for the
`documentURL` field.
Reimplement request blocking on the non-deprecated urlPatterns shape:
each pattern carries an explicit block/allow flag (first match wins),
UrlBlocklist owns the compiled patterns plus their block flags, and the
legacy setBlockedUrls path stays for back-compat. Tests updated.
Currently, Runner assumes that HttpClient.tick did something. But it's possible
that HttpClient had nothing to do, and thus didn't poll. In that case, Runner
would return .{.ok = 0} and Runner's caller would also certainly call
Runner.tick again, resulting in a spin-loop.
The reason Runner allows this to happen is because it can still have macrotasks
to run. So now, when HttpClient.tick has done nothing, Runner will return its
ms_to_next_task, rather than 0.
On sites where all i/o is completed, and only macrotasks are waiting, this
significantly reduces CPU usage.
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
Karl Seguin