The RobotStore is shared by all Browsers. While every browser has a single
flight to prevent duplicate requests to the same robots.txt, that's limited to
that specific browser. So, 2 browsers can ask for the same robots.txt and then
put try to store the result. The RobotStore _is_ thread safe, but it's a simple
last-one-wins which overrites the previous record, without freeing either the
key or value.
This replaces the last-write-wins with a first-write-wins, avoiding the leak.
Add an optional per-host rate limit. This currently only applies to the top-
level navigation. This makes it simpler to implement and simpler to reason
about. The full load of a page is only ever delayed at its head, not
sporadically through the page loading.
The use-case where a RateLimiter is most useful is when the browser is crawling
multiple pages of the same site, and in that case, the top-level rate limit
still applies some degree of limit to any linked resources (e.g. a JS on a
different host).
`--http-nav-delay` is the delay, in milliseconds, to apply to top level
navigates per host. Currently defaults to 0 (disabled).
`--http-nav-burst` is the burst allowed per host. Defaults to 1 (has no impact
when `--http-nav-delay` is disabled).
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
This applies two individual fixes to our robots.txt parser. The first is that
instead of doing an exact match with the User-Agent, it extracts the
"productToken". A UA of Lightpanda/1.0, should match "Lightpanda". The second is
that an empty directive matches.
Given:
User-agent: Lightpanda
Disallow:
User-agent: *
Disallow: /
We would have failed for 2 reasons. Fix#1 is necessary, otherwise
"Lightpanda" != "Lightpanda/1.0". Fix#2 is necessary because this robots.txt
does _not_ disallow Lightpanda from anything (e.g. the fallback doesn't apply
to us).
Parse the HTTP `Link:` response header (RFC 8288) and the robots.txt
`Content-Signal` directive, surfacing both to the driving agent.
- structured_data: extract the registered Link relations (service-doc,
service-desc, api) from the frame's retained response headers into a
new `linkHeaders` field, flowing out via the structuredData MCP tool
and LP.getStructuredData. Emitted only when present.
- Robots: capture Content-Signal preferences on a new content_signals
field, read back via RobotStore.getContentSignals and the new
LP.getContentSignal CDP command. Advisory only - never reaches
isAllowed.
Refs #2680
This is to pave the way for introducing a new "Page" container, which will take
over the page lifecycle currently burdening Session. The ultimate goal of that
is to allow the Session to have multiple pages (mostly for better transitions
between pages), which is hard to do now since the Session has so much state.
This rename was aggressive, e.g. currentPage() -> currentFrame() so that, when
the new Page container is added, you won't see "currentPage()" and wonder:
"Does 'currentPage' mean the new Page container, or the Frame (which
used to be called Page)".
@import("lightpanda") where needed.
Would also like to do this for String, Page, Session and js which all stand out
as types that are use across the codebase.
I know that a few devs are doing this in new work and I haven't heard anyone
voice an objection.
Karl Seguin