Add an optional per-host rate limit. This currently only applies to the top-
level navigation. This makes it simpler to implement and simpler to reason
about. The full load of a page is only ever delayed at its head, not
sporadically through the page loading.
The use-case where a RateLimiter is most useful is when the browser is crawling
multiple pages of the same site, and in that case, the top-level rate limit
still applies some degree of limit to any linked resources (e.g. a JS on a
different host).
`--http-nav-delay` is the delay, in milliseconds, to apply to top level
navigates per host. Currently defaults to 0 (disabled).
`--http-nav-burst` is the burst allowed per host. Defaults to 1 (has no impact
when `--http-nav-delay` is disabled).
Mime.isHttpToken already exists for the header name, added
Mime.isHttpHeaderValue which was extracted from het/Headers.zig. Now Config (
for validating CLI headers), net/Headers for script headers, and cdp/network.zig
all use the same validation methods.
Builds ontop of https://github.com/lightpanda-io/browser/pull/3200 to centralize
header enforcement and standardize merge vs overwrite header logic.
The API is still a `setHeader` and `appendHeader`, with a source, but set/append
both are thin wrappers around private `putHeader`. putHeader blocks overwriting
restricted headers (user-agent). The `source` acts as a priority (ordered enum)
which further restricts the header AND depending on whether set or append were
called, controls if the value is overwritten or appended to.
3200 had an always-append which can cause problems, e.g. a script setting
Accept-Language: fr would have results in the value being appended to the
default, e.g.: 'en-US,en;q=0.9, fr'.
buildJSONVersionResponse() previously formatted whatever Config.advertiseHost()
returned straight into webSocketDebuggerUrl. When serve was started with
--host 0.0.0.0 (or ::) without an explicit --advertise-host, advertiseHost()
returned the literal wildcard, so CDP discovery handed remote clients a
URL like ws://0.0.0.0:9222/ that can never be dialed. This was the root
cause reported in issue #1922: Playwright MCP and chromedp fail with
ECONNREFUSED when connecting through the official Docker image, which
sets --host 0.0.0.0 unconditionally.
AdvertiseHost now substitutes 127.0.0.1 for wildcard binds so the
discovery URL stays connectable from the same host (covers the official
Docker image via its published port, plus WSL's localhost bridge).
Users who serve from a dedicated interface can still pin the URL with
--advertise-host.
buildJSONVersionResponse now emits a one-time log.note when the bind
host is a wildcard so operators know to set --advertise-host for
multi-host setups. Tests cover the wildcard, the explicit-override and
the concrete-host cases.
Adding headers to an HTTP request was a bit awkward due to my desire to avoid
having an intermediate representation (e.g. an ArrayList(Header)). Going
straight to a curl slist avoids double-copying the headers (first to Zig, then
to curl).
But the CORS work (https://github.com/lightpanda-io/browser/pull/3002) showcases
that this micro-optimization simply isn't worth it, since it needs that
intermediate representation anyways.
And, this change isn't just for CORS. Headers have been a silly pain in the past
like unclear ownership, and messy APIs used in _a lot_ of places (WebBotAuth,
WebSocket, Fetch, ...)
This new approach stores headers on the transfer in an ArrayList. The API is:
```
const transfer = try client.newRequest(.{...}, owner);
{
errdefer transfer.deinit();
try transfer.addHeader("Over", "9000", .{});
}
try transfer.submit();
```
This:
1 - Eliminates ambiguity about errdefer cleanup responsibility
2 - Eliminates a bunch of stringZ concat that Frame, Config, CDP were doing
3 - Transfer.arena is now available for headers
Remove the `enable_telemetry` option from `lp_options` and `Config`. Telemetry opt-out is now managed solely via the `LIGHTPANDA_DISABLE_TELEMETRY` environment variable.
A C ABI (include/lightpanda.h + src/c_api.zig) over the browser tool
surface: lp_init/lp_shutdown, lp_fetch, sessions with lp_call/pump/
cancel, lp_tools_json. Built as a shared library only; the version
script keeps everything but lp_* internal so the bundled OpenSSL/curl/
sqlite cannot collide with a host's own.
Embedders get a hidden 'embed' Config mode (not parseable from the
CLI) with telemetry defaulting off; crash reports honor the same
opt-out. ToolSession in lightpanda.zig owns the browser/session/
registry lifecycle the C API drives.
Build: C deps are always PIC (like boringssl's force_pic) and the
pinned zig-v8-fork always builds V8 library-safe, so 'zig build
shared-lib' needs no flags. It only refuses -Dprebuilt_v8_path:
today's published archives are exe-only (local-exec TLS, malloc
shim); the guard goes away once a fork release ships library-safe
archives. make lib-shared / lib-test / lib-shared-example drive it.
std.Io.Threaded.init_single_threaded sets .allocator = .failing, which
spawnPosix uses to build the child's argv/env, so
std.process.spawn(lp.io, ...) always returns OutOfMemory and printPaged
silently fell back to plain output. Spawn the pager through a local
Threaded instance with a real allocator and the real environ (the
single-threaded one is empty, breaking PATH lookup of the less
fallback).
1 - On linux, set TCP_USER_TIMEOUT to 10 seconds. This closes the gap where
TCP_KEEPALIVE doesn't trigger if there are un-ack'd writes
2 - shutdown the socket on cases where we know the peer is gone. This would
unblock a worker blocked on a send()
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
std.start.maybeIgnoreSigpipe() already installs a noop SIGPIPE handler
at startup (std.options.keep_sigpipe defaults to false), so a write to
a dead pager returns error.BrokenPipe — which the existing catch
handles — instead of killing the process. The manual SIG_IGN was also
the disposition std deliberately avoids, since ignored dispositions
survive exec and would leak to child processes.
Adrià Arrufat