1 - TEST_VERBOSE is now off by default
2 - There's a afterEach callback that is automatically run after each tests, it:
a - clears the log filters
b - resets the test arena
3 - LogFilter replace with
a - testing.silenceLog(&.{...scopes...}); to silence all logs for the given
scopes.
b - testing.expectLog(&.{...scopes}); to set log expectations, 1 per log.
The goal here isn't so much to expect logs (though, you can do that),
but rather to silence an expected # of logs, without silencing more.
1 - On linux, set TCP_USER_TIMEOUT to 10 seconds. This closes the gap where
TCP_KEEPALIVE doesn't trigger if there are un-ack'd writes
2 - shutdown the socket on cases where we know the peer is gone. This would
unblock a worker blocked on a send()
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
Adds a /metrics endpoint in serve mode. Can be disabled via --disable-metrics.
In fetch, dumps the metrics if --metrics is specified (defaults to false).
None of the metrics being collected are on a hot path, so they're just always
collected using atomic operations (i.e. no `if (!enabled) return;`). Because
the operations are cheap and infrequent enough not to matter.
The one place I want to add more metrics is in HttpClient (bytes, status,
counts, ...) but after the pending HttpClient-related PR is merged.
./lightpanda fetch --log-level fatal --metrics "https://lightpanda.io/"
```
build_info{version="1.0.0-dev.7837+70493ce35"} 1
cdp_connections_total 0
cdp_connection_limit_total 0
cdp_active_connections 0
cdp_commands_total 0
cdp_unknown_commands_total 0
js_heap_limits_total 0
script_errors_total 0
arena_hit_total{size="tiny"} 908
arena_hit_total{size="small"} 35
arena_hit_total{size="medium"} 0
arena_hit_total{size="large"} 27
arena_miss_total{size="tiny"} 193
arena_miss_total{size="small"} 31
arena_miss_total{size="medium"} 5
arena_miss_total{size="large"} 23
navigate_total{type="page"} 1
navigate_total{type="iframe"} 0
navigate_total{type="popup"} 0
js_heap_size_bytes_bucket{le="4194304"} 0
js_heap_size_bytes_bucket{le="8388608"} 0
js_heap_size_bytes_bucket{le="16777216"} 1
js_heap_size_bytes_bucket{le="33554432"} 1
js_heap_size_bytes_bucket{le="67108864"} 1
js_heap_size_bytes_bucket{le="134217728"} 1
js_heap_size_bytes_bucket{le="268435456"} 1
js_heap_size_bytes_bucket{le="536870912"} 1
js_heap_size_bytes_bucket{le="+Inf"} 1
js_heap_size_bytes_sum 11223040
js_heap_size_bytes_count 1
```
Move away from hard-coded 512KB (WS) and 4K (http) limits. Introduces two new
serve-specific command line arguments:
--cdp-max-message-size <INT>
Maximum allowed incoming websocket message size.
Defaults to 1048576 (1MB)
--cdp-max-http-message-size <INT>
Maximum allowed HTTP request size
Defaults to 4096 (maximum allowed: 16383)
--cdp-max-message-size has been bumped from 512KB to 1MB default.
Meant to provide a more robust solution than https://github.com/lightpanda-io/browser/pull/2717
`note` is the highest log level (e.g. cannot be turned off), and thus will
always be displayed. The purpose is to make sure that people who use --port 0
can see the listening port WITHOUT giving up log-level filtering (e.g. having
to use --log-level info JUST to see the listening port).
I'm personally a fan of using .note only when (port == 0), but we can tweak it
later if people complain about it showing up despite using a warn/error/fatal
log level.
This limit isn't about correctness,it's just about making sure a test doesn't
block forever. 2 seconds does seem like plenty of time, but I'm thinking it's a
slow/busy CI. If it still happens after this bump, could mean there's an actual
issue.
LogFilter isn't thread safe, so setting it in a test where the log filter is
read from another thread trigger's TSAN. LogFilter.deinit now waits until
the server has no active threads.
Previously, the CDP socket was added to the worker's multi and fully owned
by the worker. While this is simple, it introduced some issues:
1 - Cannot detect a disconnected client during JS processing ( for(;;) )
2 - A blocked worker can cause back-pressure that blocks the client. This can
cause a deadlock if the worker is blocked waiting for a CDP message
In addition to these 2 problems, there was 1 other serious CDP-related issue:
arbitrary CDP messages could be processed during JavaScript callback. For
example, a Worker calls importScripts while request interception is enabled,
this requires us to tick the HttpClient waiting for the interception response.
But, a client could sent Target.closeTarget, which we'd process and delete the
frame..all while importScripts is still blocked. Assuming importScripts unblocks
everything is a big UAF since the frame (and its workers) were cleared from
closeTarget.
The CDP socket is now read from the network (main) thread and an OTP-style
mailbox is used. The network thread posts message to the Worker's inbox and
signals it to wakeup. This solves #1 and #2. It doesn't directly solve the
reentrancy issue, but it provides the foundation. Specifically, in introduces
a queue for of CDP message and more control over when/how that queue is
processed. At "safe points" (Runner.tick, HttpClient.tick), any message can
be processed. But, when inside a JavaScript callback, we can process only non-
destructive/mutating message. Specifically, we can process only messages related
to request interception.
network/WsConnection.zig was poorly named. It didn't represent a generic WS
connection, but rather a CDP-specific connection. This splits the generic WS
logic into network/WS.zig and the CDP-specific details in cdp/Connection.zig.
Some of the connection management in the Server has also been simplified.
buildJSONVersionResponse read the port from app.config.port(), which
is the configured value (still 0 with --port 0). Pass the OS-assigned
port from the bound address instead, so webSocketDebuggerUrl matches
the actual listener.
The integration test previously asserted a literal ws://127.0.0.1:9222/
that happened to match the test config's default port but not the
harness's hardcoded bind address (9583) — drop the URL assertion; the
buildJSONVersionResponse unit test already covers URL formatting.
This change causes lightpanda to display the actual port number (instead of 0)
when binding a dynamic port (--port 0), which makes automating based on
scraping lightpanda output simple.
@import("lightpanda") where needed.
Would also like to do this for String, Page, Session and js which all stand out
as types that are use across the codebase.
I know that a few devs are doing this in new work and I haven't heard anyone
voice an objection.
Replace dynamic build version string with stable Lightpanda/1.0
in the Browser and User-Agent fields of the /json/version response.
The dev version (1.0.0-dev.5492+...) is not useful for CDP clients.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The integration test at "server: get /json/version" was hardcoding
the old response with Content-Length: 48. Updated to verify the
enriched fields structurally since the version string varies at
build time.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add Browser, Protocol-Version, and User-Agent fields to the
/json/version CDP endpoint response. Previously it only returned
webSocketDebuggerUrl, while Chrome and other CDP browsers return
7+ fields that automation tools use for capability detection.
Also add /json/list and /json endpoints that return an empty JSON
array, matching the standard CDP endpoint layout that tools like
Puppeteer and chromedp expect.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fixed CDP inactivity timeout by resetting it when the browser is busy (loading or executing macrotasks).
- Removed the placeholder screenshot tool.
- Refactored MCP tool schemas to constants to avoid duplication.
CDPT used to be a generic so that we could inject Browser, Session, Page and
Client. At some point, it [thankfully] became a generic only to inject Client.
This commit removes the generic and bakes the *Server.Client instance in CDP.
It uses a socketpair for testing.
BrowserContext is still generic, but that's generic for a very different reason
and, while I'd like to remove that generic too, it belongs in a different PR.
This is done for a couple reasons. The first is just to have things a little
more self-contained for eventually supporting more advanced "wait" logic, e.g.
waiting for a selector.
The other is to provide callers with more fine-grained controlled. Specifically
the ability to manually "tick", so that they can [presumably] do something
after every tick. This is needed by the test runner to support more advanced
cases (cases that need to test beyond 'load') and it also improves (and fixes
potential use-after-free, the lp.waitForSelector)
Allows overwriting the --host for the json/version payload. When --host is set
to 0.0.0.0, we want to provide a mechanism to specify the specific address to
connect to in /json/version (or anywhere else that we "advertise" the address).
Inspired by https://github.com/lightpanda-io/browser/pull/1923 but rather than
defaulting to 127.0.0.1 (which seems just as unsafe), adds the explicit config
option.
The CDP timeout handler in httpLoop had two compounding bugs:
1. Unit mismatch: timestamp(.monotonic) returns seconds, but
ms_remaining is in milliseconds. The comparison and subtraction
mixed units.
2. Double-counting: In the .done branch, elapsed was computed as
absolute time since last_message, but last_message was never
updated in this branch. Each iteration subtracted the growing
total elapsed seconds from an already-decremented ms_remaining.
During complex page loads, Session._wait() returns .done rapidly
(due to JS macrotask execution, background tasks, or errors). Each
rapid .done return subtracted the growing elapsed (seconds) from
ms_remaining (milliseconds), draining it to zero in ~2 seconds
instead of the configured 10-second timeout.
Fix: use milliTimestamp() for consistent units, update last_message
in the .done branch for incremental elapsed tracking, and use >= for
correct boundary comparison.
Fixes#1849
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Shutdown on MacOS doesn't work properly. The process appears to shutdown, but
will continue to run in the background. It can run infinitely if it's stuck in
a JS loop. To help it along, Client.stop now force-terminates the isolate.
I also don't think shutdown is working as intended on Linux, but the problem
seems less serious there. On Linux, it appears to properly kill the process
(which is the important thing), but I don't think it necessarily does a clean
shutdown.
Some CDP clients (e.g. playwright-go) request /json/version/ with a
trailing
slash. Added handling for this variant to match the exact same behavior
as /json/version
Karl Seguin