Introduces a ArenaAllocator wrappre (lp.Arena). This is groundwork for better
memory tracking and reporting memory usage to v8. This is almost purely a
mechanical change to lay a foundation for a follow up PR that will address
https://github.com/lightpanda-io/browser/issues/3027
Some code became a bit leaner: a pooled arena can release itself (it has a
reference to the ArenaPool).
Some code became uglier: The Frame has a `_local_arena: *lp.Arena` and a
`local_arena: Allocator` (same with call_arena, and same with a few other types)
so that consumers aren't impacted (they continue to use `frame.local_arena`).
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
Adds a /metrics endpoint in serve mode. Can be disabled via --disable-metrics.
In fetch, dumps the metrics if --metrics is specified (defaults to false).
None of the metrics being collected are on a hot path, so they're just always
collected using atomic operations (i.e. no `if (!enabled) return;`). Because
the operations are cheap and infrequent enough not to matter.
The one place I want to add more metrics is in HttpClient (bytes, status,
counts, ...) but after the pending HttpClient-related PR is merged.
./lightpanda fetch --log-level fatal --metrics "https://lightpanda.io/"
```
build_info{version="1.0.0-dev.7837+70493ce35"} 1
cdp_connections_total 0
cdp_connection_limit_total 0
cdp_active_connections 0
cdp_commands_total 0
cdp_unknown_commands_total 0
js_heap_limits_total 0
script_errors_total 0
arena_hit_total{size="tiny"} 908
arena_hit_total{size="small"} 35
arena_hit_total{size="medium"} 0
arena_hit_total{size="large"} 27
arena_miss_total{size="tiny"} 193
arena_miss_total{size="small"} 31
arena_miss_total{size="medium"} 5
arena_miss_total{size="large"} 23
navigate_total{type="page"} 1
navigate_total{type="iframe"} 0
navigate_total{type="popup"} 0
js_heap_size_bytes_bucket{le="4194304"} 0
js_heap_size_bytes_bucket{le="8388608"} 0
js_heap_size_bytes_bucket{le="16777216"} 1
js_heap_size_bytes_bucket{le="33554432"} 1
js_heap_size_bytes_bucket{le="67108864"} 1
js_heap_size_bytes_bucket{le="134217728"} 1
js_heap_size_bytes_bucket{le="268435456"} 1
js_heap_size_bytes_bucket{le="536870912"} 1
js_heap_size_bytes_bucket{le="+Inf"} 1
js_heap_size_bytes_sum 11223040
js_heap_size_bytes_count 1
```
Move away from hard-coded 512KB (WS) and 4K (http) limits. Introduces two new
serve-specific command line arguments:
--cdp-max-message-size <INT>
Maximum allowed incoming websocket message size.
Defaults to 1048576 (1MB)
--cdp-max-http-message-size <INT>
Maximum allowed HTTP request size
Defaults to 4096 (maximum allowed: 16383)
--cdp-max-message-size has been bumped from 512KB to 1MB default.
Meant to provide a more robust solution than https://github.com/lightpanda-io/browser/pull/2717
This largely reverts 92607ad765 (captured in PR:
https://github.com/lightpanda-io/browser/pull/2398).
https://github.com/lightpanda-io/browser/pull/2495 introduces protection against
execution arbitrary CDP command during JavaScript callbacks. Claude initially
made the case for keeping the existing code as a safety net, but sycophanted
when I pushed by.
My reason for removing it is that it isn't a low-maintenance guard. It's a flag
that serves a real purpose (ensuring 1 JS script is finished before executing
another one), that has been expended to solve these issues. It needs to be set
(and reverted) at every callsite that makes a blocking call, and it needs to be
checked (recursively across all frames) in any place that can teardown the page/
frame.
Claude called the allowlist "load-bearing in a non-obvious way", but I think
it's purpose built specifically for this case. Extended the comment atop
`allowDuringSyncWait` so that future-selves remember this.
Previously, the CDP socket was added to the worker's multi and fully owned
by the worker. While this is simple, it introduced some issues:
1 - Cannot detect a disconnected client during JS processing ( for(;;) )
2 - A blocked worker can cause back-pressure that blocks the client. This can
cause a deadlock if the worker is blocked waiting for a CDP message
In addition to these 2 problems, there was 1 other serious CDP-related issue:
arbitrary CDP messages could be processed during JavaScript callback. For
example, a Worker calls importScripts while request interception is enabled,
this requires us to tick the HttpClient waiting for the interception response.
But, a client could sent Target.closeTarget, which we'd process and delete the
frame..all while importScripts is still blocked. Assuming importScripts unblocks
everything is a big UAF since the frame (and its workers) were cleared from
closeTarget.
The CDP socket is now read from the network (main) thread and an OTP-style
mailbox is used. The network thread posts message to the Worker's inbox and
signals it to wakeup. This solves #1 and #2. It doesn't directly solve the
reentrancy issue, but it provides the foundation. Specifically, in introduces
a queue for of CDP message and more control over when/how that queue is
processed. At "safe points" (Runner.tick, HttpClient.tick), any message can
be processed. But, when inside a JavaScript callback, we can process only non-
destructive/mutating message. Specifically, we can process only messages related
to request interception.
network/WsConnection.zig was poorly named. It didn't represent a generic WS
connection, but rather a CDP-specific connection. This splits the generic WS
logic into network/WS.zig and the CDP-specific details in cdp/Connection.zig.
Some of the connection management in the Server has also been simplified.
Halil Durak