Every caller resolved node.ownerFrame(frame) before building a
SemanticTree or calling getNodeDetails, and every one made the same
decision on a frameless node. Move that into SemanticTree.init, which
returns error.FramelessNode, and turn getNodeDetails into a method so
it goes through the same constructor.
With init as the only entry point the per-method assertOwns is
redundant, so drop it along with its copy of StyleManager's helper.
Compiled patterns are useful anywhere someone else writes the pattern:
the adblock lists today, agent tool arguments next. The wrapper moves
out of the adblock directory and gains an options struct (case, UTF-8
subjects) and a compile diagnostic the caller can log or show. The App
owns the one context every consumer compiles through, the blocker
included.
Follow up to https://github.com/lightpanda-io/browser/pull/3510
Moves the element/node lookups, e.g. `element_class_lists` from Frame to Page.
Elements and nodes can outlive a Frame (it's the reason the identity map lives
on the Page, not the frame). These maps are merely properties on Node/Elements
optimized for a specific usage-pattern (i.e. most Node/Elements don't have these
or they are never materialized from JS). So if a Node/Element can outlive the
Frame, than so too can all of their properties. And, even when an frame is alive
the properties belong to the *Node* or *Element*, NOT the Frame...accessing
those properties across frames should yield the same value / identity.
More mechanically, frame._page => frame.page and all of these lookups lose their
_ prefix. Short summary of _ prefix is:
1 - It's used to deal with Zig not allowing shadowing. This is particularly true
in the WebApis were it happens a bit more often
2 - Early prototype was built as a stand-alone library, and the _ was used to
signal "private" (again, working around Zig). Frame.page shouldn't be
"private" and neither should these lookups (if we aren't going to provide
getter/setters for them).
Adds a `clutter` option to --strip-mode. This is based on readability.js. It
isn't a direct port (e.g. it doesn't strop bylines). It fallsback to `shell` if
it strips too much (and shell itself can fallback to not stripping anything).
But clutter rarely fallback to shell, only when a page is very small or when
it strips out _a lot_.
Also expanded shell to look at class names and ids.
Some additional API changes:
- Add strip-mode support to pdf/png generation.
- Add LP.dump which provides greater content gathering capability to CDP,
exposing most `fetch` dump-related parameters (e.g. format, strip, selector,
...)
The new "--strip-mode shell" is designed to try to remove non-content elements
such as the header and footer. The end goal is to use readibility.js test cases
as a baseline, but this isn't a port of readibility.js.
This is just the basic implementation of this, e.g removing a few key tags, e.g.
<header>, <footer> and considering some specific roles.
Even if --strip-mode shell is used, we might decide to stick with a whole dump:
it's better to strip not enough than to strip too much. This currently works by
measuring the ratio of non-link text of the stripped vs unstripped page.
The js.Execution is the API behind the Frame/WGS split, but the split is
actually held by the underlying js.GlobalScope. Most Execution methods are:
```zig
return switch (self.js.global) {
inline else => |g| g.isSameOrigin(url),
};
```
And that works well, except that in some cases, code has a js.Context, not an
js.Execution, and they need to do the same inline switch.
This commit moves GlobalScope from src/browser/js to src/browser (there's
nothing JS/v8 about it), and moves all those inline switches into it. The
js.Execution API stays the same (it forwards the call to js.global)
but all callers that directly inlined switched the js.global no longer do.
1. Abstract "Driver". Non-CDP things that referenced *CDP now reference a Driver
2. Move the NodeRegistry out of CDP. This created an artificial link between
agent / mcp and CDP
3. Add BiDi driver with enough to navigate to a page
adds --dump screen.pdf and CDP's Page.printToPDF (along with IO.read and
IO.read). Builds ontop of https://github.com/lightpanda-io/browser/pull/3231
This generates a text-based PDF, not just our PNG placed into a PDF. The biggest
change to the existing (isolated) code was adding href information so that links
are actually clickable.
Like 3231 the hard parts are all Claude but isolated. The one place it diverged
from the plan was NOT using the subsetter rust crate for trimming the size of
the embedded font. It felt that the dependencies were too much (e.g. a second
version of skrifa, ...). One subsetter was removed, the pdf rendering, while
still a blackbox, was moved from Rust to Zig (pdf.zig), which is kind of nice.
Extract the Browser+Session+Notification+node Registry quartet, with its
ordering-sensitive init/teardown and isolate enter/exit, into
src/ToolSession.zig. mcp/Server.zig's sessions map now holds
*lp.ToolSession directly; the session id lives only as the map key.
Claude-Session: https://claude.ai/code/session_01M6WGk8wZSE28efFQkYT9SK
Callers pass CallOpts.inline_image; execScreenshot rejects a path-less
call before navigating or rendering, which removes the per-consumer
guards and covers the model-driven tool path that had none. MCP image
content is a protocol type, the screenshot recording rule joins the
recorder's replayRequires predicate, and the viewport-to-Opts mapping,
node-scope ladder and save-path helpers are shared instead of copied.
Review feedback: markdown caps its own output through Opts.max_bytes while
the html dump left it to each caller, so the html tool and fetch each
carried a copy of the LimitedWriter wrapping. root and deep now do it,
with a no-cap fast path so innerHTML/outerHTML are unaffected.
A wait failure on one URL aborted the whole multi-URL fetch with no
output, the wait_ms budget was computed once before the loop so N pages
could take N times the budget, navigation failures exited 0 with an empty
dump, and HTTP 4xx/5xx were indistinguishable from success by exit code.
Each page now has its own error slot: wait, navigation and dump failures
are recorded there instead of returned, every page is still written (the
JSON envelope carries the name under "error"), then the first failure is
returned so the process exits 1. The remaining budget is recomputed per
page. --fail-on-http-error turns a status >= 400 into exit 22, curl's
code for the same condition; the dump is written first either way.
--strip-mode was silently ignored for --dump markdown, and fetch had no
way to scope a dump to one element or cap its size although the markdown
and html tools have both.
--selector dumps the first matching element in any dump mode, --max-bytes
caps html and markdown with the tools' truncation marker, and markdown
now honors strip ui (images; scripts, styles and hidden elements are
never rendered, so the other groups don't apply).
Network has accumulated a bit of bagged. It knows a lot about certificates, it
knows a lot of the cache. I have plans to expand Network, and wanted to clean it
up.
1 - certificate logic moved to dedicated Certificates.zig
2 - Removed CurlDebugAllocator
- this makes setup easier, to Updater can just init libcurl directly
3 - Change Updater to be a single function
4 - Cache initialization is don in the Cache
5 - ?Cache => Cache{.kind = .noop}
Some WPT tests just take more memory, e.g. shadow-dom/declarative/gethtml.html.
Also left some placeholder comments for profiling JS (which I used to make sure
this was just v8 memory usage). These placeholders used to exist but didn't
survive the session multi-page refactor (figured I'd fix it next time I needed
it, and now I needed it).
The implicit default was done (full quiescence), capped at --wait-ms.
Pages with constant background activity (ad timers, live sockets) never
go quiescent, so every default fetch of such a page silently rode the
full 5s cap; the timeout is non-fatal (Runner.waitForAll discards
WaitResult.timeout), so the default was effectively "wait 5s, then dump
whatever is there". load matches the agent/MCP default_nav_wait and its
rationale.
Measured on an ad-heavy news hub (ReleaseFast, interleaved runs):
default now ~3.2s vs 5.8-5.9s with done, with the load-time dump
containing slightly more of the extracted content, not less (the extra
seconds of ad-script execution mutate the page, they do not add to it).
A quiet page dumps byte-identical output under both.
Explicit --wait-until done|networkidle and --wait-selector/--wait-script
are unchanged; the fetch() doc comment now describes when error.Timeout
can actually be raised (selector/script phases only).
1 - Add a metric to track the number of inlight arenas from the pool
2 - Script now use 2 arenas:
- An initial (small) one for the script
- A sized one for the body
Should result in less pressure on our limited .large arenas
3 - DOMPoint and DOMPointRO are now arena free (they live on the slab only)
4 - TextDecoder no longer accumulate garbage in its arena
5 - Response object is much better at picking its arena size, rather than just
using a .large
Introduces a ArenaAllocator wrappre (lp.Arena). This is groundwork for better
memory tracking and reporting memory usage to v8. This is almost purely a
mechanical change to lay a foundation for a follow up PR that will address
https://github.com/lightpanda-io/browser/issues/3027
Some code became a bit leaner: a pooled arena can release itself (it has a
reference to the ArenaPool).
Some code became uglier: The Frame has a `_local_arena: *lp.Arena` and a
`local_arena: Allocator` (same with call_arena, and same with a few other types)
so that consumers aren't impacted (they continue to use `frame.local_arena`).
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
Adds a /metrics endpoint in serve mode. Can be disabled via --disable-metrics.
In fetch, dumps the metrics if --metrics is specified (defaults to false).
None of the metrics being collected are on a hot path, so they're just always
collected using atomic operations (i.e. no `if (!enabled) return;`). Because
the operations are cheap and infrequent enough not to matter.
The one place I want to add more metrics is in HttpClient (bytes, status,
counts, ...) but after the pending HttpClient-related PR is merged.
./lightpanda fetch --log-level fatal --metrics "https://lightpanda.io/"
```
build_info{version="1.0.0-dev.7837+70493ce35"} 1
cdp_connections_total 0
cdp_connection_limit_total 0
cdp_active_connections 0
cdp_commands_total 0
cdp_unknown_commands_total 0
js_heap_limits_total 0
script_errors_total 0
arena_hit_total{size="tiny"} 908
arena_hit_total{size="small"} 35
arena_hit_total{size="medium"} 0
arena_hit_total{size="large"} 27
arena_miss_total{size="tiny"} 193
arena_miss_total{size="small"} 31
arena_miss_total{size="medium"} 5
arena_miss_total{size="large"} 23
navigate_total{type="page"} 1
navigate_total{type="iframe"} 0
navigate_total{type="popup"} 0
js_heap_size_bytes_bucket{le="4194304"} 0
js_heap_size_bytes_bucket{le="8388608"} 0
js_heap_size_bytes_bucket{le="16777216"} 1
js_heap_size_bytes_bucket{le="33554432"} 1
js_heap_size_bytes_bucket{le="67108864"} 1
js_heap_size_bytes_bucket{le="134217728"} 1
js_heap_size_bytes_bucket{le="268435456"} 1
js_heap_size_bytes_bucket{le="536870912"} 1
js_heap_size_bytes_bucket{le="+Inf"} 1
js_heap_size_bytes_sum 11223040
js_heap_size_bytes_count 1
```
The /save script documentation lived as a hand-maintained string in
Agent.zig whose primitives table drifted whenever a tool changed. Move
it to src/script/skill.zig and render the reference (signatures, option
lists, enums, defaults, per-parameter descriptions) from Schema.all()
at first use, keeping the curated per-tool notes behind an exhaustive
switch on Tool so a new or renamed tool is a compile error until its
doc entry exists.
The rendered skill is shared by three consumers:
- the /save and revision system prompts (built lazily in Agent.zig)
- a new mcp://skill/pandascript resource
- `zig build skills`, which writes zig-out/skills/<name>/SKILL.md with
Claude Code frontmatter via a registry-based generator exe, so
future Lightpanda skills are one registry entry each
Schema.FieldEntry now retains per-parameter schema descriptions, which
previously existed only in the raw JSON.
Replaces layering with an inline request pipeline, and transfer queue. This is
meant to simplify the code, reduce footguns, and make future enhancements easier
to implement (e.g. speculative parsing (which requires streaming to fully
leverage)).
Previously, HttpClient implemented deferring as a layer which required special
pumping at various callsites (https://github.com/lightpanda-io/browser/pull/2855,
https://github.com/lightpanda-io/browser/pull/2843, ...). In this new approach,
deferring is built-into the HttpClient/Transfer's flow. Specifically, Transfers
now maintain a queue of events (start, header, data, end, err) which are
dispatched in HttpClient.tick. The result is that JS callbacks are never
executed in the same stack that initiated the I/O, without needing guards or any
external intervention.
tTwo other benefits come from this. The first is that reentrant libcurl is
eliminated. Instead of "libcurl -> callback", it's now "libcurl -> transfer
event queue THEN tick -> callback" (we don't have to wait until the NEXT tick, we
can just do it later in the tick). HttpClient still has to guard against libcurl
reentrancy, but only because of how WebSocket is implemented, and we should be
able to unify WebSockets to use an event queue too in a follow up PR (which will
eliminate a bunch of guard code).
The transfer queue should also be useful to re-implement streaming, since a
data chunk is just an event in the transfer's event queue. For now, I kept it
as a single buffered event to minimize the change. But since speculative parsing
depends on this, and speculative parsing seems to be the next major performance
tweak we can make, we need to re-introduce streaming.
The other change is the removal of all other layers in favor of a pipeline. This
works well with the existing Transfer.park mechanism, where a parked Transfer
can restart the pipeline for a transfer in an arbitrary point (not as fancy as
it sounds given how simple the flow is). The fallout from this is that we're no
longer creating/wrapping contexts and callbacks: whatever the request was
configured with is all we need.
Because of this, HttpClient.Response is removed. There are no intermediary
responses and no changing context, everything is just the Transfer.
A smaller change is the addition of newRequest + transfer.submit(). The one-shot
HttpClient.request and HttpClient.requestT still exist, but this explicit create
+ submit has some advantage. First, callers can use the transfer.arena (e.g.
Frame using the transfer's arena to set the Referrer header). Second, callers
can holds Transfer immediately, rather than waiting for their startCallback to
be fired. An abort on an XMLHttpRequest called before the start of the transfer
no longer silently fails.
Lightpanda installs no SIGSEGV handler, so a segfault (or the abort() in
the panic path) falls through to the kernel and writes a core dump. When
many instances run under a shared core_pattern crash reporter -- e.g. a
containerized crawl fleet -- those cores become pure storage/alert noise,
and a browser core can capture the contents of arbitrary pages.
Crashes are already reported via telemetry, so this adds an opt-in
LIGHTPANDA_DISABLE_CORE_DUMP env var (mirroring LIGHTPANDA_DISABLE_TELEMETRY)
that zeroes the soft RLIMIT_CORE at startup. Default behavior is unchanged.
Follow up to https://github.com/lightpanda-io/browser/pull/2789 which adds
better multi-page support for runner. When waiting, callers have the choice
to wait for a specific frame, all current frames, or a given list of frames. The
last one, a given list of frames, is the most flexible, allowing callers to
provide an `until` per frame AND receive a per-frame result.
waitForSelector and waitForScript continue to be per frame (which is now
explicitly given).
Like 2789, while the changes aren't insignificant (Runner is doing important
work), a number of files were touched either purely because of changes in tests
or other superficial changes, e.g. `session.runner` is now infallible.
Replaces the 1 (+1 inflight) page design of session for an unlimited number of
pages. The main goal is to support a more efficient async goto (1). Without this
commit, async goto has two implementation: multiple browser, which is resource
intensive (thread + isolate per page), bolting it onto sub-frames (like iframe
or popups). The issue with the 2nd version (which I originally suggested) is
that most resources are tied to the Page. So even if an async goto "page" (which
would map to a Frame) is released, most things stay in memory, including the
DOM (page.factory) and the V8::Context.
This new approach adds multiple page support to sessions. The advantage is
pretty clear: the existing memory model (page-tied resources) becomes a strength
of the design, rather than a weakness.
This change is not tirivla, but the diff is inflated by 2 large mechanial
changes, so it isn't _that_ big either. That said, I'd divide this into four
parts.
1 - The old concept of _active/_pending is now baked into the Page itself. A
Page has a `replaces: ?*Page = null` and `replacement: ?*Page = null` field.
2 - Because of #1 above, the Session now just has an `pages: ArrayList(*Page)`.
As much as possible, single-page APIs, like `removePage` no longer exists. We're
trying to present a consistent multi-page API. See #3.
3 - References to *Page and *Frame have always been dangerous. For example,
`lightpanda.fetch` has a block to scope `frame`:
```zig
{
const frame = try session.createPage();
// frame isn't safe to use after navigate, it can be swapped out
_ = try frame.navigate(...)
}
```
While we still hand out *Page and *Frame (user-beware), some of the more
important APIs now take a frame_id which Session can resolve. Furthermore,
createPage now returns a PageHandle which is a safe wrapper around a page/frame.
4 - Two large mechanical changes were made:
a - Many tests were superficially changed to account for new naming or use
a new test-helper to preserve the single-page illusion (because 99% of
tests _are_ single-page)
b - Many CDP changes where `bc.session.currentFrame()` -> `bc.mainFrame()`).
This isn't to say CDP changes are meaningless, but it's mostly 1 change
about how the "main" target/frame_id is tracked (by CDP itself, rather
than the Session) that required a number of superficial changes to
accomodate
`Runner` remains largely single-page focused. Runner and some MCP/Agent tools
continue to be tied to the "currentFrame" or "primaryPage". The Runner is
something I want to address in a follow-up PR, but we need to figure out what
it means to "wait" for multiple pages.
As much as possible, Session becomes multi-page native and has no concept of
a special/first/primary/main page. Users of session become responsible for
tracking pages of interest.
This change is not trivial, but the diff is inflated by 2 large mechanical
changes:
1 - Many tests
(1) https://github.com/lightpanda-io/browser/pull/2759
I don't know why I thought debug should `unreahcable` while release should
generate a crash report. The crash report always contains more info and whenever
I encounter a failed assertion in debug, the first thing I do is remove the
unreachable to get the more complete crash report.
(The crash report path skips sending the crash report in debug builds already)
Adrià Arrufat