The RobotStore is shared by all Browsers. While every browser has a single
flight to prevent duplicate requests to the same robots.txt, that's limited to
that specific browser. So, 2 browsers can ask for the same robots.txt and then
put try to store the result. The RobotStore _is_ thread safe, but it's a simple
last-one-wins which overrites the previous record, without freeing either the
key or value.
This replaces the last-write-wins with a first-write-wins, avoiding the leak.
ScriptManager, XMLHttpRequest.zig, Fetch, Workers, etc. all take ownership (aka
dupe) the HTTP response from HTTPClient. They all have a headerCallback that
does something like:
```zig
if (transfer.getContentLength()) |cl| {
try self.body.ensureTotalCapacity(self.arena, cl);
}
```
But in all non-streaming cases (which is most cases), the HttpClient buffers
the response and only calls the headerCallback _after_ the body has been
received. Rather than relying on "Content-Length" header, the body buffer can
be sized to the exact body length. Why does this matter? Because the
Content-Length is the length of the body on the wire, and if the body is
compressed (like almost all .js files are), it will under-report the final
body length AND, because most callers are using an arena, the buffer growth
will retain more memory than it should.
This adds a `transfer.bodyLen()` method. Callers which dupe the body now use
this rather than the Content-Length (Content-Length is still used, e.g. for
XHR progress report).
If you look at https://github.com/lightpanda-io/browser/pull/3293, you'll see
a relatively contained change that has to touch over 20 files. The issue is that
every HttpClient.newRequest needs to provide a lot of data. But `newRequest`
takes a 2nd parameter: the HttpClient.Owner. If we make that Owner a little
smarter, we can start to remove some of the individual fields needed in
newRequest. For example, we can still allow a callsite to pass frame_id but,
by default, we can use the owner's frame_id (which is what we want in most
cases).
The site for cookies were computed from the immediate parent `Frame`, which would allow sending a cookie that's `SameSite=Strict` from 2 levels deep under. Directly from RFC6265bis, this PR essentially implements (except for step 4, we skip host-less ancestors):
Given a Document (document), the following algorithm returns its
"site for cookies":
1. Let top-document be the active document in document's navigable's
top-level traversable.
2. Let top-origin be the origin of top-document's URI if top-
document's sandboxed origin browsing context flag is set, and
top-document's origin otherwise.
3. Let documents be a list consisting of the active documents of
document's inclusive ancestor navigables.
4. For each item in documents:
1. Let origin be the origin of item's URI if item's sandboxed
origin browsing context flag is set, and item's origin
otherwise.
2. If origin is not same-site with top-origin, return an origin
set to an opaque origin.
5. Return top-origin.
When we know the precise final length, prefer ensureTotalCapacityPrecise over
ensureTotalCapacity. The latter goes through `growCapacity` which will allocate
~1.5x padding.
Introduces a ArenaAllocator wrappre (lp.Arena). This is groundwork for better
memory tracking and reporting memory usage to v8. This is almost purely a
mechanical change to lay a foundation for a follow up PR that will address
https://github.com/lightpanda-io/browser/issues/3027
Some code became a bit leaner: a pooled arena can release itself (it has a
reference to the ArenaPool).
Some code became uglier: The Frame has a `_local_arena: *lp.Arena` and a
`local_arena: Allocator` (same with call_arena, and same with a few other types)
so that consumers aren't impacted (they continue to use `frame.local_arena`).
Give accurate connectionId, connectionReused, initialPriority and securityState
values.
Always set `referrerPolicy` to `unsafe-url` as the most honest answer (we should
implement proper referrer policy!).
For workers, track the underlying frame_id so that it can be used for the
`documentURL` field.
Commit eliminates a class of UAF on transfer abort due to the abort potentially
JS callbacks within a libcurl callback.
It also introduces a graveyard queue to the HttpClient so that, on deinit,
Transfers become detached but their memory is still valid. In debug, the
transfer is poisoned so that any uaf fails loudly. In release, the graveyard
is simply cleaned up.
Replaces layering with an inline request pipeline, and transfer queue. This is
meant to simplify the code, reduce footguns, and make future enhancements easier
to implement (e.g. speculative parsing (which requires streaming to fully
leverage)).
Previously, HttpClient implemented deferring as a layer which required special
pumping at various callsites (https://github.com/lightpanda-io/browser/pull/2855,
https://github.com/lightpanda-io/browser/pull/2843, ...). In this new approach,
deferring is built-into the HttpClient/Transfer's flow. Specifically, Transfers
now maintain a queue of events (start, header, data, end, err) which are
dispatched in HttpClient.tick. The result is that JS callbacks are never
executed in the same stack that initiated the I/O, without needing guards or any
external intervention.
tTwo other benefits come from this. The first is that reentrant libcurl is
eliminated. Instead of "libcurl -> callback", it's now "libcurl -> transfer
event queue THEN tick -> callback" (we don't have to wait until the NEXT tick, we
can just do it later in the tick). HttpClient still has to guard against libcurl
reentrancy, but only because of how WebSocket is implemented, and we should be
able to unify WebSockets to use an event queue too in a follow up PR (which will
eliminate a bunch of guard code).
The transfer queue should also be useful to re-implement streaming, since a
data chunk is just an event in the transfer's event queue. For now, I kept it
as a single buffered event to minimize the change. But since speculative parsing
depends on this, and speculative parsing seems to be the next major performance
tweak we can make, we need to re-introduce streaming.
The other change is the removal of all other layers in favor of a pipeline. This
works well with the existing Transfer.park mechanism, where a parked Transfer
can restart the pipeline for a transfer in an arbitrary point (not as fancy as
it sounds given how simple the flow is). The fallout from this is that we're no
longer creating/wrapping contexts and callbacks: whatever the request was
configured with is all we need.
Because of this, HttpClient.Response is removed. There are no intermediary
responses and no changing context, everything is just the Transfer.
A smaller change is the addition of newRequest + transfer.submit(). The one-shot
HttpClient.request and HttpClient.requestT still exist, but this explicit create
+ submit has some advantage. First, callers can use the transfer.arena (e.g.
Frame using the transfer's arena to set the Referrer header). Second, callers
can holds Transfer immediately, rather than waiting for their startCallback to
be fired. An abort on an XMLHttpRequest called before the start of the transfer
no longer silently fails.
Karl Seguin