Adding headers to an HTTP request was a bit awkward due to my desire to avoid
having an intermediate representation (e.g. an ArrayList(Header)). Going
straight to a curl slist avoids double-copying the headers (first to Zig, then
to curl).
But the CORS work (https://github.com/lightpanda-io/browser/pull/3002) showcases
that this micro-optimization simply isn't worth it, since it needs that
intermediate representation anyways.
And, this change isn't just for CORS. Headers have been a silly pain in the past
like unclear ownership, and messy APIs used in _a lot_ of places (WebBotAuth,
WebSocket, Fetch, ...)
This new approach stores headers on the transfer in an ArrayList. The API is:
```
const transfer = try client.newRequest(.{...}, owner);
{
errdefer transfer.deinit();
try transfer.addHeader("Over", "9000", .{});
}
try transfer.submit();
```
This:
1 - Eliminates ambiguity about errdefer cleanup responsibility
2 - Eliminates a bunch of stringZ concat that Frame, Config, CDP were doing
3 - Transfer.arena is now available for headers
Remove the `enable_telemetry` option from `lp_options` and `Config`. Telemetry opt-out is now managed solely via the `LIGHTPANDA_DISABLE_TELEMETRY` environment variable.
A C ABI (include/lightpanda.h + src/c_api.zig) over the browser tool
surface: lp_init/lp_shutdown, lp_fetch, sessions with lp_call/pump/
cancel, lp_tools_json. Built as a shared library only; the version
script keeps everything but lp_* internal so the bundled OpenSSL/curl/
sqlite cannot collide with a host's own.
Embedders get a hidden 'embed' Config mode (not parseable from the
CLI) with telemetry defaulting off; crash reports honor the same
opt-out. ToolSession in lightpanda.zig owns the browser/session/
registry lifecycle the C API drives.
Build: C deps are always PIC (like boringssl's force_pic) and the
pinned zig-v8-fork always builds V8 library-safe, so 'zig build
shared-lib' needs no flags. It only refuses -Dprebuilt_v8_path:
today's published archives are exe-only (local-exec TLS, malloc
shim); the guard goes away once a fork release ships library-safe
archives. make lib-shared / lib-test / lib-shared-example drive it.
std.Io.Threaded.init_single_threaded sets .allocator = .failing, which
spawnPosix uses to build the child's argv/env, so
std.process.spawn(lp.io, ...) always returns OutOfMemory and printPaged
silently fell back to plain output. Spawn the pager through a local
Threaded instance with a real allocator and the real environ (the
single-threaded one is empty, breaking PATH lookup of the less
fallback).
1 - On linux, set TCP_USER_TIMEOUT to 10 seconds. This closes the gap where
TCP_KEEPALIVE doesn't trigger if there are un-ack'd writes
2 - shutdown the socket on cases where we know the peer is gone. This would
unblock a worker blocked on a send()
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
std.start.maybeIgnoreSigpipe() already installs a noop SIGPIPE handler
at startup (std.options.keep_sigpipe defaults to false), so a write to
a dead pager returns error.BrokenPipe — which the existing catch
handles — instead of killing the process. The manual SIG_IGN was also
the disposition std deliberately avoids, since ignored dispositions
survive exec and would leak to child processes.
Explicitly requested help (help, help <cmd>, --help) now goes to stdout
instead of stderr, and is piped through $PAGER (fallback: less -FIRX)
when stdout is an interactive terminal, so long help like 'help agent'
(~225 lines) no longer clobbers the screen. Piped or redirected output
stays plain, and error-path usage remains unpaged on stderr.
Adds a /metrics endpoint in serve mode. Can be disabled via --disable-metrics.
In fetch, dumps the metrics if --metrics is specified (defaults to false).
None of the metrics being collected are on a hot path, so they're just always
collected using atomic operations (i.e. no `if (!enabled) return;`). Because
the operations are cheap and infrequent enough not to matter.
The one place I want to add more metrics is in HttpClient (bytes, status,
counts, ...) but after the pending HttpClient-related PR is merged.
./lightpanda fetch --log-level fatal --metrics "https://lightpanda.io/"
```
build_info{version="1.0.0-dev.7837+70493ce35"} 1
cdp_connections_total 0
cdp_connection_limit_total 0
cdp_active_connections 0
cdp_commands_total 0
cdp_unknown_commands_total 0
js_heap_limits_total 0
script_errors_total 0
arena_hit_total{size="tiny"} 908
arena_hit_total{size="small"} 35
arena_hit_total{size="medium"} 0
arena_hit_total{size="large"} 27
arena_miss_total{size="tiny"} 193
arena_miss_total{size="small"} 31
arena_miss_total{size="medium"} 5
arena_miss_total{size="large"} 23
navigate_total{type="page"} 1
navigate_total{type="iframe"} 0
navigate_total{type="popup"} 0
js_heap_size_bytes_bucket{le="4194304"} 0
js_heap_size_bytes_bucket{le="8388608"} 0
js_heap_size_bytes_bucket{le="16777216"} 1
js_heap_size_bytes_bucket{le="33554432"} 1
js_heap_size_bytes_bucket{le="67108864"} 1
js_heap_size_bytes_bucket{le="134217728"} 1
js_heap_size_bytes_bucket{le="268435456"} 1
js_heap_size_bytes_bucket{le="536870912"} 1
js_heap_size_bytes_bucket{le="+Inf"} 1
js_heap_size_bytes_sum 11223040
js_heap_size_bytes_count 1
```
`lightpanda agent <script>` replays a saved script token-free (no LLM),
but running it under `agent` reads wrong. Add `run` as a dedicated,
script-only command: `lightpanda run <script>`.
`run` exposes only a required [SCRIPT] positional plus the common
options; its LLM-session flags (--task/--save/--list-models/REPL) are
absent, so they're rejected at parse time. `parseArgs` normalizes a
`run` invocation to an `.agent` command, reusing the existing token-free
replay path unchanged, so nothing downstream needs a `.run` case.
`agent <script>` keeps working. Telemetry records `run` as its own mode
code (R) via a new Config.command field, so its adoption is measurable
separately from agent script replay (AR).
Introduces an HTTP transport option to serve multipleagents from a
single process. Each connection is routed to its own isolated
browsing session using the `Mcp-Session-Id` header.
Also adds new session management tools (`session_new`, `session_list`,
`session_close`) and refactors the MCP server to support multiple V8
isolates with parking.
Adds a new command line argument `--watchdog-ms` which, when set, will terminate
any JS that appears to be hung.
When configured, a new thread is started. Workers heartbeat this thread to
signal activity (e.g. not stuck in a JS loop). However, workers also block for
their own reason (e.g. network polling), so they can signal the watchdog that
they are "entering a wait" and, when complete, that they are "existing a wait".
During such waits, the watchdog will not signal the isolate to terminate.
Obviously, it's important for workers to signal aliveness and whenever they plan
on doing a non-JS wait. So you could say we introduce safe points where the
watchdog (for that browser) is disabled. We could do the opposite: enable the
watchdog whenever we enter JS ("hey, I'm about to execute JS, monitor me). But
there are _a lot_ more place where this happens.
The main addition in this commit is that we hook into the Isolate's
AddNearHeapLimitCallback callback and try to force the isolate to shutdown
rather than letting v8 hit an OOM which would take down the entire process.
In support of this, we now support a `--v8-max-heap-mb` command line option to
set an explicit heap limit. As a simple way to test this feature, load a
relatively heavy JS page with `--v8-max-heap-mb 1`.
There's also a `--v8-flags-unsafe` which is a mechanism to pass arbitrary
flags to v8 via its `SetFlagsFromString`. The parameter is called `unsafe`
because some [of the many] configurable flags could conflict with how the
snapshot is built and result in crashes. The snapshot creator also gains a
`--v8-flags-unsafe` flag, so advance users COULD create their snapshot and
run lightpanda with the same set of flags.
max http connections 10 -> 40
max http connetions per host 4 -> 6
These are just the defaults and can still be adjusted by the command line
arguments. 6 appears to be both Chrome and FireFox's default per host (which
is probably the more important of the two settings).
The limits are really use-case specific. A use case that it multi-threading
different domains can benefit from a conservative max-host with a very large
max conn. A use case that is multi-threading the same host will need to decide
if it's safe to raise max-host.
`--log-filter-scopes` previously took a comma-separated list of scopes to
suppress. Extend it with explicit include/exclude directives applied
left-to-right:
-X filter out scope X
X alias for -X (backward compatible)
+X filter in scope X
all target every scope
This makes "filter everything except one scope" expressible, e.g.
`-all,+cdp` suppresses all logs except `cdp`. Bare scope names keep their
old meaning, so existing invocations like `http,unknown_prop` are
unaffected.
Internally, the suppressed-scope list is replaced by a resolved per-scope
`scope_enabled` boolean array built via `log.resolveFilterScopes()`, so the
hot-path `enabled()` check is a single array index. Filtering remains
Debug-only, as before. `testing.LogFilter` is adapted to the new
representation; all existing call sites are unchanged.
Adds a unit test for resolveFilterScopes (default, backward-compat,
-all,+cdp, ordering) and updates the help text.
Move away from hard-coded 512KB (WS) and 4K (http) limits. Introduces two new
serve-specific command line arguments:
--cdp-max-message-size <INT>
Maximum allowed incoming websocket message size.
Defaults to 1048576 (1MB)
--cdp-max-http-message-size <INT>
Maximum allowed HTTP request size
Defaults to 4096 (maximum allowed: 16383)
--cdp-max-message-size has been bumped from 512KB to 1MB default.
Meant to provide a more robust solution than https://github.com/lightpanda-io/browser/pull/2717
Moves `tagNames` and `tagHint` from `SlashCommand` to `Config` for reuse.
Adds `tagJsonArray` to dynamically generate JSON enum schemas, and
uses it in `tools.zig` for the `WaitUntil` enum.
Adrià Arrufat