Our `Arena` from the `ArenaPool` now tracks if it's already been released. On
a subsequent release, it panics then and there. Without this, the code will
almost certainly panic anyways, but it will panic in a seemingly unrelated
place. Hopefully this makes identifying future cases of this easier (since we'll
get the stack trace of the re-releaser).
Fix 3 separate memory issues, all edge cases.
1 - an XHR error handler that re-open/sends would incorrectly null the new
transfer (maybe this isn't that odd, maybe it's a common retry-on-error).
2 - On a curl_easy_pause error (from WebSocket.zig) we now unqueue the just-
queued message, because the error will errdefer the message arena to be
cleaned up.
3 - ScriptManager now cleans up after itself on a failure prior to submit()
being called.
HttpClient guarantees that only one of done/error/shtudown callback will be
called. But that guarantee had a hole when a done/error callback would trigger
JS that caused an HttpClient.abort (e.g. by scheduling a navigation).
So far, this isn't a problem. fetch/XHR guard against this themselves, and
ScriptManager doesn't care of it happens.
But https://github.com/lightpanda-io/browser/pull/3230 doesn't guard against it
and would crash if it happens: its doneCallback frees `self` but can re-enter
shutdown which needs `self`. While 3230 could fix this issue, I'd prefer to
bake this contract into the HttpClient. Transfers now have an
`_outcome_delivered: bool` which guarantees that only one of done/error/shutdown
is called.
Network is less cohesive than https://github.com/lightpanda-io/browser/pull/3242
would indicate. It has two distinct and _completely_ separate responsibilities.
1 - It acts as the base for each HttpClient, providing a shared connection pool
(for http and ws) and access to the process wide Cache, RobotStore,
WebBothAuth, certificates, ...
2 - It accepts, polls and reads from CDP connection
There is zero relationship between these, and it's a particularly bad place for
this duality to exist because both parts are, in their own way, the main
multi-threaded junction in the system.
This commit is purely mechanical it:
1 - Keeps network as the base for each HttpClient.
2 - Extract the CDP interaction into the existing Server.zig
3 - Gives mcp's HttpServer its own accept loop
Network has accumulated a bit of bagged. It knows a lot about certificates, it
knows a lot of the cache. I have plans to expand Network, and wanted to clean it
up.
1 - certificate logic moved to dedicated Certificates.zig
2 - Removed CurlDebugAllocator
- this makes setup easier, to Updater can just init libcurl directly
3 - Change Updater to be a single function
4 - Cache initialization is don in the Cache
5 - ?Cache => Cache{.kind = .noop}
Network has accumulated a bit of bagged. It knows a lot about certificates, it
knows a lot of the cache. I have plans to expand Network, and wanted to clean it
up.
1 - certificate logic moved to dedicated Certificates.zig
2 - Removed CurlDebugAllocator
- this makes setup easier, to Updater can just init libcurl directly
3 - Change Updater to be a single function
4 - Cache initialization is don in the Cache
5 - ?Cache => Cache{.kind = .noop}
Add an optional per-host rate limit. This currently only applies to the top-
level navigation. This makes it simpler to implement and simpler to reason
about. The full load of a page is only ever delayed at its head, not
sporadically through the page loading.
The use-case where a RateLimiter is most useful is when the browser is crawling
multiple pages of the same site, and in that case, the top-level rate limit
still applies some degree of limit to any linked resources (e.g. a JS on a
different host).
`--http-nav-delay` is the delay, in milliseconds, to apply to top level
navigates per host. Currently defaults to 0 (disabled).
`--http-nav-burst` is the burst allowed per host. Defaults to 1 (has no impact
when `--http-nav-delay` is disabled).
1-
Per spec, dictionary objects should be loaded in lexicographical order (a). So
the following always pushes `endings` before `type`
```js
new Blob([], {
get type() { order.push('type'); return ''; },
get endings() { order.push('endings'); return 'transparent'; },
});
```
This requirement is now enforced by a comptime check. So, a lot of files were
updated to satisfy this requirement.
2-
Blob parts now work with the @@iterator protocol. This is done more generically
(via `value.iterator())` but is currently only used by Blob (and File). Other
types that need this (e.g. URLSearchParams) can be done in a follow up.
3-
Add support for Blob.textStream
4-
Reject XHR/Fetch requests on blob URLs if the method isn't GET
5-
Strip #hash from blob URL when doing lookup
All of this fixes ~175 FileAPI WPT cases, though it's possible #1 (the
dictionary ordering) helps with a few others.
(a) https://webidl.spec.whatwg.org/#js-dictionary
Mime.isHttpToken already exists for the header name, added
Mime.isHttpHeaderValue which was extracted from het/Headers.zig. Now Config (
for validating CLI headers), net/Headers for script headers, and cdp/network.zig
all use the same validation methods.
Builds ontop of https://github.com/lightpanda-io/browser/pull/3200 to centralize
header enforcement and standardize merge vs overwrite header logic.
The API is still a `setHeader` and `appendHeader`, with a source, but set/append
both are thin wrappers around private `putHeader`. putHeader blocks overwriting
restricted headers (user-agent). The `source` acts as a priority (ordered enum)
which further restricts the header AND depending on whether set or append were
called, controls if the value is overwritten or appended to.
3200 had an always-append which can cause problems, e.g. a script setting
Accept-Language: fr would have results in the value being appended to the
default, e.g.: 'en-US,en;q=0.9, fr'.
--http-cache-entry-limit is currently implemented as a hard-limit via a check
on every put. For a large --http-cache-entry-limit, this can result in a non-
trivial delay on every http get.
This commit calls cache.maintenance() on Browser.deinit and on startup.
maintenance:
1 - purges stale entries
2 - enforces the limit
With concurrent requests, the number of entries can easily exceed the limit by
hundreds of entries (but this is temporary).
muki