This feature is significant because it adds support for processing an HTTP
request via the worker. It requires parking the connection and then having the
worker notify the loop when the response is ready. A lot of this was already
in-place (e.g. worker -> loop notification) but not quite do this extent.
tighten socket ownership (on error paths)
allow reaper to be disabled
Handle window where link is being destroyed, worker is still alive, and client
attempts to re-link.
This is a small step towards WebDriver supports (non-bidi). It allows creating
and deleting a BiDi "Session" (e.g. a worker). It also allows attaching a BiDi
driver to an HTTP-created BiDi session (the typical selenium startup flow).
This change unblocks the most basic setup/teardown of Selenium, so it still
isn't enough to actually use a Selenium script as-is. But it's significant
because it models a worker (thread) that isn't tied to a WebSocket, something we
haven't had before.
A consequence of a pure HTTP Session is that we don't have a clear cleanup
signal. There is no "the socket is disconnected". There's a new HTTP reaper
which kills HTTP Sessions after --http-session-timeout. It's expected that
drivers properly DELETE /session/:id. I imagine we're going to run into
--cdp-max-connections limits and need to tweak this code. BUT, this entire flow
is only enabled with --protocol webdriver, so it won't impact exiting CDP users.
macOS delivers loopback traffic asynchronously, so LoopTest.accept could
call Server.accept before the handshake ACK landed (NotAccepted) and the
tests could read before the request bytes landed (WouldBlock, RST, and a
double disconnect that cast a -1 socket to usize in KQueue.socketEvent).
Poll for readiness first. Also accept any non-zero SO_KEEPALIVE: BSD
getsockopt returns the option bit, not 1.
Currently, the HttpClient owns the inbox and its borrowed by the Link. This is
a bit backwards, but it also means that we can't eagerly create a Link: the
Link needs the inbox, so it needs the HttpClient, which is created by the
Browser (which creates an Isolate).
Remember, the Inbox is one of the few things shared between the main thread
and the worker, so either end can own it and the other can borrow it.
This switches the ownership so that the HttpClient now borrows the Inbox from
the Server's side of the Link (the WebSocket).
The main goal of this change is to prepare for more advanced HTTP WebDriver
flows. The more we can create _without_ a Browser, the fewer edge cases we have
to deal with (Browser because it's expensive and has to be created on the
Worker thread due to how V8::Isolate works).
Give it one pass through some Claude fuzz testing. Add a max message size,
protect against weird interactions during a shutdown and we had some pending
accepts. Put a time limit on blocked writes.
Significant rework of the CDP/BiDi server. There are two main changes:
1 - poll replaced with EPoll/Kqueue (1)
2 - make http serving a first class citizen
The change from poll -> epoll/kqueue isn't performance driven, it's just about
tighter code. Both epoll and kqueue let you associate arbitrary data with a
socket, so we don't need to keep arrays in sync in order to associate a socket
with a CDP by index. They both provide some event/notification mechanism, which
is cleaner than the pipe required by poll.
The poll -> epoll/kqueue change could almost have been mechanical. Making HTTP
a first class citizen is the more significant of the two changes
In `main`, a new connection always spawns a thread and, until does its own
little read loop until the connection is upgraded. This is not efficient, it
uses up a connection slot, and it's inconsistent with the final WebSocket
connection which _is_ polled off the main loop. Using up a slot means that
keepalive isn't possible, else HTTP connections would quickly use up all
available slots/threads.
This commit parses and serves HTTP requests on the main thread (safe
because none of the processing is blocking). The approach is better streamlined
for HTTP requests which never upgrade (/metrics, WebDriver) without causing
any performance overhead for those that do. It simplifies some things (e.g. an
"http" socket or a "websocket" socket is monitored and read in a similar manner
(on the main loop)). It makes other things more complicated; the flow is no
longer accept -> spawn -> upgrade -> websocket loop. It's loop -> accept -> loop
-> process -> (http | ws).
This is built ontop of the BiDi branch because (a) WebDriver is what needs
better HTTP support and (b) some of the more mechanical changes already exist
in that branch (e.g. src/cdp/, src/server.zig -> src/server/*)
(1) kqueue landing in 2 commits from now on this branch.
We currently have 1 note: it prints the server's listening address:port. Note
is a special un-ignorable level. This keeps the "note" level, but logs it under
a new scope: "note", so that it _can_ be silenced with a `--log-filter note`.
Add a new note, on startup, that displays tips. Currently, only displays when
--obey-robots is not enabled:
NOTE note : config tips . . . . . . . . . . . . . . . . . . . [+0ms]
robots = use '--obey-robots' to use a sites robots.txt
meta = use '--log-filter note' to silence this message
In order to support Selenium the way people are used to, it looks like we need
to support both WebDriver classic (WebDriver) and WebDriver BiDi (BiDi). Typical
scripts look like a mix of the two, e.g. using WebDriver to control the browser
and using BiDi to receive notifications. This commit:
1 - adds a --protocol (cdp|webdriver) CLI argument to the `serve` command to
enable one or the other protocol (defaulting to CDP)
2 - adds basic WebDriver endpoint to let a Selenium client connect. This
implementation is hackish and sits on top of our simple Handshake handler.
The handshake handler is well past its original design. Serving /json/version
and /metrics from it was one thing. But Driving the entire browser session? This
will get a follow up PR.
1. Abstract "Driver". Non-CDP things that referenced *CDP now reference a Driver
2. Move the NodeRegistry out of CDP. This created an artificial link between
agent / mcp and CDP
3. Add BiDi driver with enough to navigate to a page
Karl Seguin