By default, iframes and workers no longer loaded. Use `--load-resources iframe`
and `--load-resources worker` to restore the previous behavior. The disabling
makes resource loading more consistent.
To further make things more consistent, Config seems the following changes:
1. remove `--timeout` from `serve` which does nothing but has printed a
deprecation warning for a long time
2. added .deprecated field to CLI config flags which now logs the specified
deprecated warning when used
3. `--log-filter-scopes` is deprecated in favor of `--log-scopes`
4. `--disable_subframes` is deprecated. Iframe loading is disabled by default,
use `--load-resources iframe` to enable iframe loading
5. `--disable_workers` is deprecated. Worker loading is disabled by default,
use `--load-resources worker` to enable worker loading
6. `--enable_external_stylesheets` is deprecated. Stylesheets remain disabled
by default. Use `--load-resources stylesheet` to enable loading external
stylesheets
CLI log parameters now alter the logger behavior on parse. This helps minimze
the window where default log settings are in-play. It also means things like
this work:
```
./lightpanda --log-level fatal --disable_subframes --log-level warn
```
More seriously, there's now an optional `beforeParse` fired once the mode is
known. This is used by mcp to set the default log level to logfmt. Previously
this was done much later and could easily result in a mix of pretty and logfmt
logs.
Review feedback: markdown caps its own output through Opts.max_bytes while
the html dump left it to each caller, so the html tool and fetch each
carried a copy of the LimitedWriter wrapping. root and deep now do it,
with a no-cap fast path so innerHTML/outerHTML are unaffected.
A wait failure on one URL aborted the whole multi-URL fetch with no
output, the wait_ms budget was computed once before the loop so N pages
could take N times the budget, navigation failures exited 0 with an empty
dump, and HTTP 4xx/5xx were indistinguishable from success by exit code.
Each page now has its own error slot: wait, navigation and dump failures
are recorded there instead of returned, every page is still written (the
JSON envelope carries the name under "error"), then the first failure is
returned so the process exits 1. The remaining budget is recomputed per
page. --fail-on-http-error turns a status >= 400 into exit 22, curl's
code for the same condition; the dump is written first either way.
--strip-mode was silently ignored for --dump markdown, and fetch had no
way to scope a dump to one element or cap its size although the markdown
and html tools have both.
--selector dumps the first matching element in any dump mode, --max-bytes
caps html and markdown with the tools' truncation marker, and markdown
now honors strip ui (images; scripts, styles and hidden elements are
never rendered, so the other groups don't apply).
MCP mode unconditionally set log.opts.format = .logfmt after the
--log_format flag had already been applied, so `--log_format pretty`
was silently ignored. It also ran after the "starting server" line,
which came out in the pre-override format in debug builds.
Move it into Config.logFormat() as a mode-aware default, mirroring
what logLevel() does for agent mode: null -> logfmt for mcp, and an
explicit flag wins.
Network is less cohesive than https://github.com/lightpanda-io/browser/pull/3242
would indicate. It has two distinct and _completely_ separate responsibilities.
1 - It acts as the base for each HttpClient, providing a shared connection pool
(for http and ws) and access to the process wide Cache, RobotStore,
WebBothAuth, certificates, ...
2 - It accepts, polls and reads from CDP connection
There is zero relationship between these, and it's a particularly bad place for
this duality to exist because both parts are, in their own way, the main
multi-threaded junction in the system.
This commit is purely mechanical it:
1 - Keeps network as the base for each HttpClient.
2 - Extract the CDP interaction into the existing Server.zig
3 - Gives mcp's HttpServer its own accept loop
fetchThread and mcpThread caught their errors, logged fatal, and
returned void; the main thread joined and exited 0. A fetch whose
--wait-selector timed out (or any lp.fetch/browser-init error) printed
$level=fatal and still reported success to the shell. Same for mcp
stdio errors, and for three fatal paths in the mcp HTTP branch that ran
on the main thread and returned void.
Threads now hand the error back through an out-pointer (the agentThread
failed/cancelled pattern) and the dispatch returns it after join; the
mcp main-thread paths return their errors directly.
Clean mcp stdin EOF is unaffected: processRequests returns normally on
EOF (router.zig takeDelimiter orelse break), so only real failures
reach the exit code.
Out of scope: a failed navigation in fetch still exits 0 by design —
it renders the synthetic "Navigation failed" error page and dumps it;
--json carries the status for detection.
Make sure jemallocator is disabled in Release builds (we only use it in Debug
builds to collect metrics. Supposedly something changed in our 0.16 build that
causes it to be initialized even though it isn't used. Not sure, for this, I'm
just doing what Claude tell me, but disabling something in Release that I know
we aren't using sounds ok).
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
Explicitly requested help (help, help <cmd>, --help) now goes to stdout
instead of stderr, and is piped through $PAGER (fallback: less -FIRX)
when stdout is an interactive terminal, so long help like 'help agent'
(~225 lines) no longer clobbers the screen. Piped or redirected output
stays plain, and error-path usage remains unpaged on stderr.
Adds a /metrics endpoint in serve mode. Can be disabled via --disable-metrics.
In fetch, dumps the metrics if --metrics is specified (defaults to false).
None of the metrics being collected are on a hot path, so they're just always
collected using atomic operations (i.e. no `if (!enabled) return;`). Because
the operations are cheap and infrequent enough not to matter.
The one place I want to add more metrics is in HttpClient (bytes, status,
counts, ...) but after the pending HttpClient-related PR is merged.
./lightpanda fetch --log-level fatal --metrics "https://lightpanda.io/"
```
build_info{version="1.0.0-dev.7837+70493ce35"} 1
cdp_connections_total 0
cdp_connection_limit_total 0
cdp_active_connections 0
cdp_commands_total 0
cdp_unknown_commands_total 0
js_heap_limits_total 0
script_errors_total 0
arena_hit_total{size="tiny"} 908
arena_hit_total{size="small"} 35
arena_hit_total{size="medium"} 0
arena_hit_total{size="large"} 27
arena_miss_total{size="tiny"} 193
arena_miss_total{size="small"} 31
arena_miss_total{size="medium"} 5
arena_miss_total{size="large"} 23
navigate_total{type="page"} 1
navigate_total{type="iframe"} 0
navigate_total{type="popup"} 0
js_heap_size_bytes_bucket{le="4194304"} 0
js_heap_size_bytes_bucket{le="8388608"} 0
js_heap_size_bytes_bucket{le="16777216"} 1
js_heap_size_bytes_bucket{le="33554432"} 1
js_heap_size_bytes_bucket{le="67108864"} 1
js_heap_size_bytes_bucket{le="134217728"} 1
js_heap_size_bytes_bucket{le="268435456"} 1
js_heap_size_bytes_bucket{le="536870912"} 1
js_heap_size_bytes_bucket{le="+Inf"} 1
js_heap_size_bytes_sum 11223040
js_heap_size_bytes_count 1
```
Introduces an HTTP transport option to serve multipleagents from a
single process. Each connection is routed to its own isolated
browsing session using the `Mcp-Session-Id` header.
Also adds new session management tools (`session_new`, `session_list`,
`session_close`) and refactors the MCP server to support multiple V8
isolates with parking.
std.crypto.random's default backend mmaps a thread-local 528-byte state
page on first use and never unmaps it — there is no thread-exit hook.
With one detached thread per CDP connection (Server.handleConnection),
that leaks one resident page per connection (uuidv4 in
Page.getOrCreateOrigin touches it), ~4KB/conn of unbounded RSS growth.
Route every std.crypto.random call to the getrandom syscall instead.
.crypto_always_getrandom = true,
Lightpanda installs no SIGSEGV handler, so a segfault (or the abort() in
the panic path) falls through to the kernel and writes a core dump. When
many instances run under a shared core_pattern crash reporter -- e.g. a
containerized crawl fleet -- those cores become pure storage/alert noise,
and a browser core can capture the contents of arbitrary pages.
Crashes are already reported via telemetry, so this adds an opt-in
LIGHTPANDA_DISABLE_CORE_DUMP env var (mirroring LIGHTPANDA_DISABLE_TELEMETRY)
that zeroes the soft RLIMIT_CORE at startup. Default behavior is unchanged.
This reverts recent(ish) changes to telemetry which moved it from its own thread
onto the main thread.
The downside is: we have an extra thread.
The upside is largely that Network.zig becomes drastically simpler and more
efficient. There's a bunch of machinery in Network.zig to support arbitrary
workers, of which Telemetry is the only one. There's also a lot of code to
support an optional multi and requests made to is. This is all removed.
Also, fetch, agent and mcp without a cdp server no longer even need to start
the network loop. And, it IS started (e.g. serve/cdp), there's no longer an
arbitrary 250ms wakeup on poll to progress workers. Nor can telemtry block CDP.
Telemetry's implementation itself was changed. The ring buffer was removed in
favor of a double-buffer arraylist. When telemetry is disabled, this saves
64Kb of memory. When it's enabled, it creates more allocator churn, but should
still use less memory in most cases (and never more). Finally, Telemetry is
given its own easy connection rather than using one out of the pool (which
workers would maybe like to use).
`--log-filter-scopes` previously took a comma-separated list of scopes to
suppress. Extend it with explicit include/exclude directives applied
left-to-right:
-X filter out scope X
X alias for -X (backward compatible)
+X filter in scope X
all target every scope
This makes "filter everything except one scope" expressible, e.g.
`-all,+cdp` suppresses all logs except `cdp`. Bare scope names keep their
old meaning, so existing invocations like `http,unknown_prop` are
unaffected.
Internally, the suppressed-scope list is replaced by a resolved per-scope
`scope_enabled` boolean array built via `log.resolveFilterScopes()`, so the
hot-path `enabled()` check is a single array index. Filtering remains
Debug-only, as before. `testing.LogFilter` is adapted to the new
representation; all existing call sites are unchanged.
Adds a unit test for resolveFilterScopes (default, backward-compat,
-all,+cdp, ordering) and updates the help text.
Introduce an `Llm` struct to bundle the provider and API key together.
Add `UserError` to identify errors that have already printed human-
readable messages, preventing duplicate logging on exit.
The `fetch` command is very practical to render pages without needing to
have a long running browser instance.
It is however masking all details on the fetch, most importantly the HTTP status code.
This is a big caveat when leveraging `lightpanda fetch` in a pipeline.
This introduces a `--json` option to provide a structured output that
contains:
* url
* HTTP status code
* response headers
* rendered content as controlled by the `--dump` option
The proposal is to always output the same JSON format even when not
using `--dump` with an option.
* Prefer `--inject-*` prefix.
* Support injecting multiple scripts (also allows using both variants together).
* Instead of executing scripts in JS context, actually insert them to `<head>` for correct dump output.
Karl Seguin