The http max default was 4K with a 16KB hard limit. The default limit is now 1MB
with an initial default of 4K. This is to accommodate larger WebDriver payloads.
This feature is significant because it adds support for processing an HTTP
request via the worker. It requires parking the connection and then having the
worker notify the loop when the response is ready. A lot of this was already
in-place (e.g. worker -> loop notification) but not quite do this extent.
tighten socket ownership (on error paths)
allow reaper to be disabled
Handle window where link is being destroyed, worker is still alive, and client
attempts to re-link.
This is a small step towards WebDriver supports (non-bidi). It allows creating
and deleting a BiDi "Session" (e.g. a worker). It also allows attaching a BiDi
driver to an HTTP-created BiDi session (the typical selenium startup flow).
This change unblocks the most basic setup/teardown of Selenium, so it still
isn't enough to actually use a Selenium script as-is. But it's significant
because it models a worker (thread) that isn't tied to a WebSocket, something we
haven't had before.
A consequence of a pure HTTP Session is that we don't have a clear cleanup
signal. There is no "the socket is disconnected". There's a new HTTP reaper
which kills HTTP Sessions after --http-session-timeout. It's expected that
drivers properly DELETE /session/:id. I imagine we're going to run into
--cdp-max-connections limits and need to tweak this code. BUT, this entire flow
is only enabled with --protocol webdriver, so it won't impact exiting CDP users.
Give it one pass through some Claude fuzz testing. Add a max message size,
protect against weird interactions during a shutdown and we had some pending
accepts. Put a time limit on blocked writes.
Significant rework of the CDP/BiDi server. There are two main changes:
1 - poll replaced with EPoll/Kqueue (1)
2 - make http serving a first class citizen
The change from poll -> epoll/kqueue isn't performance driven, it's just about
tighter code. Both epoll and kqueue let you associate arbitrary data with a
socket, so we don't need to keep arrays in sync in order to associate a socket
with a CDP by index. They both provide some event/notification mechanism, which
is cleaner than the pipe required by poll.
The poll -> epoll/kqueue change could almost have been mechanical. Making HTTP
a first class citizen is the more significant of the two changes
In `main`, a new connection always spawns a thread and, until does its own
little read loop until the connection is upgraded. This is not efficient, it
uses up a connection slot, and it's inconsistent with the final WebSocket
connection which _is_ polled off the main loop. Using up a slot means that
keepalive isn't possible, else HTTP connections would quickly use up all
available slots/threads.
This commit parses and serves HTTP requests on the main thread (safe
because none of the processing is blocking). The approach is better streamlined
for HTTP requests which never upgrade (/metrics, WebDriver) without causing
any performance overhead for those that do. It simplifies some things (e.g. an
"http" socket or a "websocket" socket is monitored and read in a similar manner
(on the main loop)). It makes other things more complicated; the flow is no
longer accept -> spawn -> upgrade -> websocket loop. It's loop -> accept -> loop
-> process -> (http | ws).
This is built ontop of the BiDi branch because (a) WebDriver is what needs
better HTTP support and (b) some of the more mechanical changes already exist
in that branch (e.g. src/cdp/, src/server.zig -> src/server/*)
(1) kqueue landing in 2 commits from now on this branch.
Karl Seguin