The implicit default was done (full quiescence), capped at --wait-ms.
Pages with constant background activity (ad timers, live sockets) never
go quiescent, so every default fetch of such a page silently rode the
full 5s cap; the timeout is non-fatal (Runner.waitForAll discards
WaitResult.timeout), so the default was effectively "wait 5s, then dump
whatever is there". load matches the agent/MCP default_nav_wait and its
rationale.
Measured on an ad-heavy news hub (ReleaseFast, interleaved runs):
default now ~3.2s vs 5.8-5.9s with done, with the load-time dump
containing slightly more of the extracted content, not less (the extra
seconds of ad-script execution mutate the page, they do not add to it).
A quiet page dumps byte-identical output under both.
Explicit --wait-until done|networkidle and --wait-selector/--wait-script
are unchanged; the fetch() doc comment now describes when error.Timeout
can actually be raised (selector/script phases only).
1 - Add a metric to track the number of inlight arenas from the pool
2 - Script now use 2 arenas:
- An initial (small) one for the script
- A sized one for the body
Should result in less pressure on our limited .large arenas
3 - DOMPoint and DOMPointRO are now arena free (they live on the slab only)
4 - TextDecoder no longer accumulate garbage in its arena
5 - Response object is much better at picking its arena size, rather than just
using a .large
Introduces a ArenaAllocator wrappre (lp.Arena). This is groundwork for better
memory tracking and reporting memory usage to v8. This is almost purely a
mechanical change to lay a foundation for a follow up PR that will address
https://github.com/lightpanda-io/browser/issues/3027
Some code became a bit leaner: a pooled arena can release itself (it has a
reference to the ArenaPool).
Some code became uglier: The Frame has a `_local_arena: *lp.Arena` and a
`local_arena: Allocator` (same with call_arena, and same with a few other types)
so that consumers aren't impacted (they continue to use `frame.local_arena`).
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
Adds a /metrics endpoint in serve mode. Can be disabled via --disable-metrics.
In fetch, dumps the metrics if --metrics is specified (defaults to false).
None of the metrics being collected are on a hot path, so they're just always
collected using atomic operations (i.e. no `if (!enabled) return;`). Because
the operations are cheap and infrequent enough not to matter.
The one place I want to add more metrics is in HttpClient (bytes, status,
counts, ...) but after the pending HttpClient-related PR is merged.
./lightpanda fetch --log-level fatal --metrics "https://lightpanda.io/"
```
build_info{version="1.0.0-dev.7837+70493ce35"} 1
cdp_connections_total 0
cdp_connection_limit_total 0
cdp_active_connections 0
cdp_commands_total 0
cdp_unknown_commands_total 0
js_heap_limits_total 0
script_errors_total 0
arena_hit_total{size="tiny"} 908
arena_hit_total{size="small"} 35
arena_hit_total{size="medium"} 0
arena_hit_total{size="large"} 27
arena_miss_total{size="tiny"} 193
arena_miss_total{size="small"} 31
arena_miss_total{size="medium"} 5
arena_miss_total{size="large"} 23
navigate_total{type="page"} 1
navigate_total{type="iframe"} 0
navigate_total{type="popup"} 0
js_heap_size_bytes_bucket{le="4194304"} 0
js_heap_size_bytes_bucket{le="8388608"} 0
js_heap_size_bytes_bucket{le="16777216"} 1
js_heap_size_bytes_bucket{le="33554432"} 1
js_heap_size_bytes_bucket{le="67108864"} 1
js_heap_size_bytes_bucket{le="134217728"} 1
js_heap_size_bytes_bucket{le="268435456"} 1
js_heap_size_bytes_bucket{le="536870912"} 1
js_heap_size_bytes_bucket{le="+Inf"} 1
js_heap_size_bytes_sum 11223040
js_heap_size_bytes_count 1
```
The /save script documentation lived as a hand-maintained string in
Agent.zig whose primitives table drifted whenever a tool changed. Move
it to src/script/skill.zig and render the reference (signatures, option
lists, enums, defaults, per-parameter descriptions) from Schema.all()
at first use, keeping the curated per-tool notes behind an exhaustive
switch on Tool so a new or renamed tool is a compile error until its
doc entry exists.
The rendered skill is shared by three consumers:
- the /save and revision system prompts (built lazily in Agent.zig)
- a new mcp://skill/pandascript resource
- `zig build skills`, which writes zig-out/skills/<name>/SKILL.md with
Claude Code frontmatter via a registry-based generator exe, so
future Lightpanda skills are one registry entry each
Schema.FieldEntry now retains per-parameter schema descriptions, which
previously existed only in the raw JSON.
Replaces layering with an inline request pipeline, and transfer queue. This is
meant to simplify the code, reduce footguns, and make future enhancements easier
to implement (e.g. speculative parsing (which requires streaming to fully
leverage)).
Previously, HttpClient implemented deferring as a layer which required special
pumping at various callsites (https://github.com/lightpanda-io/browser/pull/2855,
https://github.com/lightpanda-io/browser/pull/2843, ...). In this new approach,
deferring is built-into the HttpClient/Transfer's flow. Specifically, Transfers
now maintain a queue of events (start, header, data, end, err) which are
dispatched in HttpClient.tick. The result is that JS callbacks are never
executed in the same stack that initiated the I/O, without needing guards or any
external intervention.
tTwo other benefits come from this. The first is that reentrant libcurl is
eliminated. Instead of "libcurl -> callback", it's now "libcurl -> transfer
event queue THEN tick -> callback" (we don't have to wait until the NEXT tick, we
can just do it later in the tick). HttpClient still has to guard against libcurl
reentrancy, but only because of how WebSocket is implemented, and we should be
able to unify WebSockets to use an event queue too in a follow up PR (which will
eliminate a bunch of guard code).
The transfer queue should also be useful to re-implement streaming, since a
data chunk is just an event in the transfer's event queue. For now, I kept it
as a single buffered event to minimize the change. But since speculative parsing
depends on this, and speculative parsing seems to be the next major performance
tweak we can make, we need to re-introduce streaming.
The other change is the removal of all other layers in favor of a pipeline. This
works well with the existing Transfer.park mechanism, where a parked Transfer
can restart the pipeline for a transfer in an arbitrary point (not as fancy as
it sounds given how simple the flow is). The fallout from this is that we're no
longer creating/wrapping contexts and callbacks: whatever the request was
configured with is all we need.
Because of this, HttpClient.Response is removed. There are no intermediary
responses and no changing context, everything is just the Transfer.
A smaller change is the addition of newRequest + transfer.submit(). The one-shot
HttpClient.request and HttpClient.requestT still exist, but this explicit create
+ submit has some advantage. First, callers can use the transfer.arena (e.g.
Frame using the transfer's arena to set the Referrer header). Second, callers
can holds Transfer immediately, rather than waiting for their startCallback to
be fired. An abort on an XMLHttpRequest called before the start of the transfer
no longer silently fails.
Lightpanda installs no SIGSEGV handler, so a segfault (or the abort() in
the panic path) falls through to the kernel and writes a core dump. When
many instances run under a shared core_pattern crash reporter -- e.g. a
containerized crawl fleet -- those cores become pure storage/alert noise,
and a browser core can capture the contents of arbitrary pages.
Crashes are already reported via telemetry, so this adds an opt-in
LIGHTPANDA_DISABLE_CORE_DUMP env var (mirroring LIGHTPANDA_DISABLE_TELEMETRY)
that zeroes the soft RLIMIT_CORE at startup. Default behavior is unchanged.
Follow up to https://github.com/lightpanda-io/browser/pull/2789 which adds
better multi-page support for runner. When waiting, callers have the choice
to wait for a specific frame, all current frames, or a given list of frames. The
last one, a given list of frames, is the most flexible, allowing callers to
provide an `until` per frame AND receive a per-frame result.
waitForSelector and waitForScript continue to be per frame (which is now
explicitly given).
Like 2789, while the changes aren't insignificant (Runner is doing important
work), a number of files were touched either purely because of changes in tests
or other superficial changes, e.g. `session.runner` is now infallible.
Replaces the 1 (+1 inflight) page design of session for an unlimited number of
pages. The main goal is to support a more efficient async goto (1). Without this
commit, async goto has two implementation: multiple browser, which is resource
intensive (thread + isolate per page), bolting it onto sub-frames (like iframe
or popups). The issue with the 2nd version (which I originally suggested) is
that most resources are tied to the Page. So even if an async goto "page" (which
would map to a Frame) is released, most things stay in memory, including the
DOM (page.factory) and the V8::Context.
This new approach adds multiple page support to sessions. The advantage is
pretty clear: the existing memory model (page-tied resources) becomes a strength
of the design, rather than a weakness.
This change is not tirivla, but the diff is inflated by 2 large mechanial
changes, so it isn't _that_ big either. That said, I'd divide this into four
parts.
1 - The old concept of _active/_pending is now baked into the Page itself. A
Page has a `replaces: ?*Page = null` and `replacement: ?*Page = null` field.
2 - Because of #1 above, the Session now just has an `pages: ArrayList(*Page)`.
As much as possible, single-page APIs, like `removePage` no longer exists. We're
trying to present a consistent multi-page API. See #3.
3 - References to *Page and *Frame have always been dangerous. For example,
`lightpanda.fetch` has a block to scope `frame`:
```zig
{
const frame = try session.createPage();
// frame isn't safe to use after navigate, it can be swapped out
_ = try frame.navigate(...)
}
```
While we still hand out *Page and *Frame (user-beware), some of the more
important APIs now take a frame_id which Session can resolve. Furthermore,
createPage now returns a PageHandle which is a safe wrapper around a page/frame.
4 - Two large mechanical changes were made:
a - Many tests were superficially changed to account for new naming or use
a new test-helper to preserve the single-page illusion (because 99% of
tests _are_ single-page)
b - Many CDP changes where `bc.session.currentFrame()` -> `bc.mainFrame()`).
This isn't to say CDP changes are meaningless, but it's mostly 1 change
about how the "main" target/frame_id is tracked (by CDP itself, rather
than the Session) that required a number of superficial changes to
accomodate
`Runner` remains largely single-page focused. Runner and some MCP/Agent tools
continue to be tied to the "currentFrame" or "primaryPage". The Runner is
something I want to address in a follow-up PR, but we need to figure out what
it means to "wait" for multiple pages.
As much as possible, Session becomes multi-page native and has no concept of
a special/first/primary/main page. Users of session become responsible for
tracking pages of interest.
This change is not trivial, but the diff is inflated by 2 large mechanical
changes:
1 - Many tests
(1) https://github.com/lightpanda-io/browser/pull/2759
I don't know why I thought debug should `unreahcable` while release should
generate a crash report. The crash report always contains more info and whenever
I encounter a failed assertion in debug, the first thing I do is remove the
unreachable to get the more complete crash report.
(The crash report path skips sending the crash report in debug builds already)
Removes the `-i`/`--interactive` CLI flag and live file-based
recording. Instead, the REPL now supports a `/load <path>` command
to run scripts from disk, and `/save` to export the in-memory
session recording.
The `Recorder` is simplified to be purely in-memory, and the script
runtime is moved to `src/script/Runtime.zig`.
BREAKING CHANGE: The `-i`/`--interactive` flag has been removed. Use
the `/save` and `/load` commands within the REPL instead.
We still occasionally see release overflow errors. I think this is a UAF, since
I cannot find any mismatched acquire/releases. So this adds a canary value to
every RC and includes it in the crash dump.
If the canary value is different than what we set it at, then we have a UAF.
If the canary value is set to the poison we set on release, then we have a
double free.
If the canary value is unchanged, then it's inconclusive.
Removes the `--self-heal` CLI option, the `scriptStep` and `scriptHeal`
MCP tools, and associated verification/iterator machinery. Replaces
"PandaScript" terminology with "slash commands" and moves shared
helpers to `tools.zig`.
BREAKING CHANGE: The `--self-heal` CLI flag and the `scriptStep` and
`scriptHeal` MCP tools have been removed.
Replace recorded agent replay with a standalone JavaScript script runtime.
Install synchronous agent primitives in an isolated V8 context, add console
output, return structured extract values as JS objects/arrays, and route script
execution through the new runtime.
Update recording to emit .js function calls, default /save filenames to .js,
drop script-level extract save support, and refresh agent docs/tutorials for
the new format.
Self-healing is disabled for now.
With the new CookieStore, the session must be freed before the notification is.
This is how it works in CDP, but in fetch, we were pretty lazy about it. This
caused the notification to be freed first, and then the cookiestore to try to
unregister: UAF.
- Add diagnostics to command parsing to report invalid field values.
- Fix UTF-8 truncation in spinner to avoid splitting codepoints.
- Fix swapped arguments in log error formatting.
- Distinguish "null" values from missing elements in verifier.
- Handle MCP tool cancellation and timeouts with specific error codes.
This adds help documentation for the --json flag. This is the only thing that
must be kept from this commit.
It uses our testing.zig to streamline the json testing (instead of string
probing). It removes the JsonEnvelope in favor of an anonymous struct (though,
I'm fine with adding it back in if it's needed to resolve something ambiguous).
Finally, I removed the last unit test as, at that point, it's really just
testing Zig's JSON stringifier (could arguably make the same case for the other
two, but there's some logic there about how nulls/empty might be handled).
The `fetch` command is very practical to render pages without needing to
have a long running browser instance.
It is however masking all details on the fetch, most importantly the HTTP status code.
This is a big caveat when leveraging `lightpanda fetch` in a pipeline.
This introduces a `--json` option to provide a structured output that
contains:
* url
* HTTP status code
* response headers
* rendered content as controlled by the `--dump` option
The proposal is to always output the same JSON format even when not
using `--dump` with an option.
Moves script logic to `browser/script/` for shared use. Implements
message rollback on API failure, caches environment variables, and
fixes a potential panic in the spinner.
- Relocate Command, Recorder, and Verifier to `src/browser/script/`
- Implement message rollback on API and synthesis failures in Agent
- Cache `LP_*` environment variables process-wide with mutex protection
- Fix potential panic in Spinner during backward clock jumps
- Improve Recorder to handle write failures and multi-line comments
- Update documentation regarding attachments and path safety
* Prefer `--inject-*` prefix.
* Support injecting multiple scripts (also allows using both variants together).
* Instead of executing scripts in JS context, actually insert them to `<head>` for correct dump output.
Adrià Arrufat