Fixes https://github.com/lightpanda-io/browser/issues/3141
We now capture request body provided they are <= 64KB (arbitrary limit, we can
revisit, but this doesn't seem to be an overly common request and this pins
the bodies in memory until frame end, so I wanted to take it slowly).
Adds Network.getRequestPostData and the `postData` and the `postDataEntries`
fields to the Request object.
Extracted from https://github.com/lightpanda-io/browser/pull/3122. Sends RI for
redirect. Also, on a continueRequest which does redirect, restores the original
headers (continueRequest's headers are only valid for a single request).
To make this work in all drivers, CDP now decouples the transfer_id from the
intercept_id. Each unique request gets a distinct intercept_id which is managed
in CDP (with a intercept_id -> transfer_id mapping).
1. Every `var caught: js.TryCatch.Caught = undefined;` is now defaulted to .{}.
This eliminates a possible dirty read. Function tryCall (and its variants)
were guarding against this, but newInstance wasn't.
2. Env.terminate() now immediately sets `terminate_requested` which means
`isExecutionTerminating` is removed and only `terminatePending` exists, and
it only needs to check `terminate_requested`.
Review flag: the dcl gating in actions.waitForSelector also changed the CDP
extension, which kept a flat 5s default. It now uses the same rule as the
tool layer (5s at load, 15s before), so a mid-navigation call keeps the wall
budget it had when the pre-wait ran to load. Explicit timeouts unchanged.
As-is, this PR (3122) includes 2 distinct fixes:
1 - a Target.attachToTarget should create a distinct session_id
2 - http redirects with request interception (RI) should:
a - not preserve the original RI headers, and
b - should send their own RI request to the driver
This commit removes #2 so that it can be done in its own follow up. #2 is
relatively big. The commit also removes the integration test and CI changes.
These will be ported to the demo project to leverage the existing CI flow.
3122 breaks non-playwright drivers on both features. #2, will be addressed
in the follow up PR. For #1, when attachToTarget is called when bc.session_id
is null, then the existing behavior is maintained. This happens for example,
with chromedp which doesn't autoAttach.
When we know the precise final length, prefer ensureTotalCapacityPrecise over
ensureTotalCapacity. The latter goes through `growCapacity` which will allocate
~1.5x padding.
Only notify v8 of memory pressure when (a) there's memory to claim and (b)
there are dead context.
Also clean up code that relied on undefined behavior which might have left
local scopes un-freed and caused a v8 leak.
Adding headers to an HTTP request was a bit awkward due to my desire to avoid
having an intermediate representation (e.g. an ArrayList(Header)). Going
straight to a curl slist avoids double-copying the headers (first to Zig, then
to curl).
But the CORS work (https://github.com/lightpanda-io/browser/pull/3002) showcases
that this micro-optimization simply isn't worth it, since it needs that
intermediate representation anyways.
And, this change isn't just for CORS. Headers have been a silly pain in the past
like unclear ownership, and messy APIs used in _a lot_ of places (WebBotAuth,
WebSocket, Fetch, ...)
This new approach stores headers on the transfer in an ArrayList. The API is:
```
const transfer = try client.newRequest(.{...}, owner);
{
errdefer transfer.deinit();
try transfer.addHeader("Over", "9000", .{});
}
try transfer.submit();
```
This:
1 - Eliminates ambiguity about errdefer cleanup responsibility
2 - Eliminates a bunch of stringZ concat that Frame, Config, CDP were doing
3 - Transfer.arena is now available for headers
Alignment allowing (1) this reduces Node size by 8 bytes. It also aligns with
previous cdata changes with the end-goal to have the entire Node chain adopt
bare tags.
(1) string.String is 16-byte aligned, so Text doesn't shrink with this change.
Hopefully this can be addressed separately.
1 - Add a metric to track the number of inlight arenas from the pool
2 - Script now use 2 arenas:
- An initial (small) one for the script
- A sized one for the body
Should result in less pressure on our limited .large arenas
3 - DOMPoint and DOMPointRO are now arena free (they live on the slab only)
4 - TextDecoder no longer accumulate garbage in its arena
5 - Response object is much better at picking its arena size, rather than just
using a .large
1 - TEST_VERBOSE is now off by default
2 - There's a afterEach callback that is automatically run after each tests, it:
a - clears the log filters
b - resets the test arena
3 - LogFilter replace with
a - testing.silenceLog(&.{...scopes...}); to silence all logs for the given
scopes.
b - testing.expectLog(&.{...scopes}); to set log expectations, 1 per log.
The goal here isn't so much to expect logs (though, you can do that),
but rather to silence an expected # of logs, without silencing more.
Introduces a ArenaAllocator wrappre (lp.Arena). This is groundwork for better
memory tracking and reporting memory usage to v8. This is almost purely a
mechanical change to lay a foundation for a follow up PR that will address
https://github.com/lightpanda-io/browser/issues/3027
Some code became a bit leaner: a pooled arena can release itself (it has a
reference to the ArenaPool).
Some code became uglier: The Frame has a `_local_arena: *lp.Arena` and a
`local_arena: Allocator` (same with call_arena, and same with a few other types)
so that consumers aren't impacted (they continue to use `frame.local_arena`).
As part of the ongoing effort to remove the `_proto: *Parent` field,
TaggedOpaque.fromJS is now offset based. Technically this should perform better
since we're no longer walking through pointers. But the real goal is that we
can now start removing _proto fields.
But, for fromJS to be offset-based, we had to make sure every prototype chain
was a contiguous allocation. The Factory did this, but not everything used the
factory. Now, everything does. This change alone, without the fromJS change,
made every WebAPI allocation more consistent.
Ultimately, our goal is to remove the `_proto: *Field`. That would save 24
bytes per Text node (or Element, ...). Large websites have 20K+ nodes, so we're
talking 300-500KB savings.
ef756b0c96 was the first phase of this. This is
the second, and it's targetting at CData for a simple reason: CData currently
has an optimization that makes removal of `_proto: *Field` difficult: it
directly embeds its empty types (most notably, Text). Zig's @offsetOf doesn't
work on tagged unions, so we'd have to jump through hoops to figure it out (it
is doable though). Also, CData has a "incorrect" flattened CDATASection
specifically so that Text doesn't inherit a _type union..so that's another
problem.
The solution is to move to bare unions, which allows us to use our Factory
to control the layout of the CData and thus have predictable offsets.
CData is now an inconsistent with the rest, but, at the very least, all Node
types will move in this direction.
closeTarget above implies that the state this assertion guards against IS valid
AND we just had a crash report confirming. That's enough for me to remove it.
Zig 0.16 changed clocks/timestamps. Our migration to Zig 0.16 took the path of
least resistance. For example, we kept 'monotonic' and 'real' even though Zig
0.16 renamed them to 'boot' and 'clock'
This commit tries to standardize all timestamp usage to: lp.datetime.timestamp
and lp.datetime.milliTimestamp using the new zig 0.16 names ('boot' and 'clock')
Give accurate connectionId, connectionReused, initialPriority and securityState
values.
Always set `referrerPolicy` to `unsafe-url` as the most honest answer (we should
implement proper referrer policy!).
For workers, track the underlying frame_id so that it can be used for the
`documentURL` field.
Keep worker requests visible on the page CDP session when their WorkerGlobalScope frame id is absent from the document frame tree. Add a regression fixture covering the worker script and a fetch from inside the worker.
CDP driver can send multiple Network.enable which would register the same
listener multiple times. This commit makes it so that only one (the first)
callback registered for a listener+eventtype is used. Subsequent registration
for the same listener+eventtype are ignored. This is safe because all callbacks
are currently static. It's a mistake (enforced by a debug-only assertion) for
code to try to register a different callback for an already registered listener+
eventtype.
This generalizes https://github.com/lightpanda-io/browser/pull/3038
Reimplement request blocking on the non-deprecated urlPatterns shape:
each pattern carries an explicit block/allow flag (first match wins),
UrlBlocklist owns the compiled patterns plus their block flags, and the
legacy setBlockedUrls path stays for back-compat. Tests updated.
Include parentId on child frame navigation events and recursively serialize child frames in Page.getFrameTree using their live frame and loader IDs.
Fixes#3014
Currently, Runner assumes that HttpClient.tick did something. But it's possible
that HttpClient had nothing to do, and thus didn't poll. In that case, Runner
would return .{.ok = 0} and Runner's caller would also certainly call
Runner.tick again, resulting in a spin-loop.
The reason Runner allows this to happen is because it can still have macrotasks
to run. So now, when HttpClient.tick has done nothing, Runner will return its
ms_to_next_task, rather than 0.
On sites where all i/o is completed, and only macrotasks are waiting, this
significantly reduces CPU usage.
Scott Taylor