Dependencies:
- v8: zig-v8-fork 0.17 branch (lightpanda-io/zig-v8-fork#218).
- sqlite3: build the amalgamation directly; the allyourcodebase wrapper
has no 0.17 support yet.
- pcre2: master commit with 0.17 support (no release yet).
- translate_c package (2.0.0) replaces the deprecated addTranslateC.
default_init is set to keep 0.16's zero-initialized struct fields.
Build:
- b.pathFromRoot/build_root/args/sysroot are gone: resolve paths from
b.root, use addPassthruArgs, declare configure-time file and directory
dependencies, and poison the configure cache for the git version.
- Drop curl config values the template never used (now an error).
- Drop the 0.16 zip-fetch workaround from the Makefile and CI.
- -Doptimize=ReleaseFast -> -Doptimize=fast.
Language and std:
- @intFromEnum/@enumFromInt -> @backingInt/@fromBackingInt.
- Struct-of-arrays @typeInfo (field_names, field_types, decl_names,
param_types, error_names).
- `a ** n` removed: @splat for arrays, string.repeat for strings.
- errdefer captures removed: split WebDriver.run and fulfillRedirect.
- @hasDecl only sees pub decls: make StyleManager's Spec.finish pub,
otherwise it silently stops running.
- SafeAllocator replaces DebugAllocator; the test runner now has to
initialize std.testing.allocator_instance itself.
- std.fmt.allocPrint/bufPrint -> Allocator.print/std.mem.print,
dupeZ -> dupeSentinel, std.builtin -> std.lang, builtin.os ->
builtin.target.os, zon.parse arena API, BufferFirstAllocator, and the
remaining renames (getLastOrNull, bit_set, ascii.find*, meta.Int).
`lightpanda version.io` was rejected as a misspelt `version`, 3 edits
within the length-scaled limit. Like curl, it fetches now: the command check
shares --dump's isUrlLike.
`fetch --dump markdown.com` was rejected as a misspelt `markdown`: the
length-scaled limit allows 4 edits for 12 characters. No format has a `.`,
`/` or `:`, so an argument with one is always the url.
`--log-filter -cdpp` suggested `cdp`; invalidChoice now takes the stripped
prefix and puts it back on the value and the suggestion.
`cli.invalidChoice` logs a bad value with its closest match, and the
generic enum path, --dump, --log-level, --log-format and now --log-filter
all use it. editDistance lowercases its inputs once, and closest strips
exactly a leading `--`, as its doc says.
A flat limit of two edits was too loose for short words and too strict
for long ones: `--dump md` suggested `pdf`, and a bare word within two
edits of `run` or `mcp` was rejected as a mistyped command instead of
being fetched, while `--insecure-disable-tls-verification` got no
suggestion at all. `closest` now allows about one edit per three
characters, as rustc does, not counting a leading `--`, and swapping two
adjacent characters counts as one edit so `--dmup` still finds `--dump`.
`--log-level` and `--log-format` now suggest the closest value too.
A flag given without its value, or an extra positional, failed with
only `FATAL exit err=MissingArgument`, not naming the flag. The parser
now logs which flag is missing its value or which argument is extra,
with a hint pointing at the command's help, and a missing fetch URL is
caught while parsing, next to run's missing script. Since each of these
is logged where it's found, main exits without the generic `exit` line.
`-h` works wherever `--help` does, instead of being taken as a URL.
The --obey-robots tip is for a person at a terminal, so it's skipped
when stderr isn't one; scripts capturing stderr no longer get it on
every run.
A `--task` run that knows where it is going still spent a model turn
navigating there, and the REPL had no way to start anywhere but blank.
`--url` opens the page first, through `browser_tools.call` rather than
around it, so a bad URL fails like any other tool call and the opening
navigation is recorded for `--save` -- which is the first line any
replayable script needs anyway.
The search engine could only be set from the REPL's `/searchEngine`, so a
one-shot run had no way to pin one -- and a benchmark that wants its
results to mean something has to record which API answered.
`/searchEngine` also carried the only warning about key state, which is
the half that matters more. A keyless engine is not an error and starts
fine, so a run that silently falls back to a rate-limited public endpoint
looks exactly like a working one until every search begins failing, and
then it looks like a bad agent. Both messages now fire wherever the
engine is resolved, not just from the command.
`resolveSearchEngine`'s doc comment said there was no CLI flag. There is
one now.
The http max default was 4K with a 16KB hard limit. The default limit is now 1MB
with an initial default of 4K. This is to accommodate larger WebDriver payloads.
This adds the shell for ServiceWorker, behind
`--experimental-features serviceworker`.
It's pretty useless as-is. We don't have the CacheStorage API (next) and don't
have the fetch interceptor (next next). But as-is, the change is quite big but
thankfully largely isolated.
tighten socket ownership (on error paths)
allow reaper to be disabled
Handle window where link is being destroyed, worker is still alive, and client
attempts to re-link.
This is a small step towards WebDriver supports (non-bidi). It allows creating
and deleting a BiDi "Session" (e.g. a worker). It also allows attaching a BiDi
driver to an HTTP-created BiDi session (the typical selenium startup flow).
This change unblocks the most basic setup/teardown of Selenium, so it still
isn't enough to actually use a Selenium script as-is. But it's significant
because it models a worker (thread) that isn't tied to a WebSocket, something we
haven't had before.
A consequence of a pure HTTP Session is that we don't have a clear cleanup
signal. There is no "the socket is disconnected". There's a new HTTP reaper
which kills HTTP Sessions after --http-session-timeout. It's expected that
drivers properly DELETE /session/:id. I imagine we're going to run into
--cdp-max-connections limits and need to tweak this code. BUT, this entire flow
is only enabled with --protocol webdriver, so it won't impact exiting CDP users.
navigator.languages now lists the Accept-Language tags in order, which is
Chrome's contract, instead of a second derivation from the locale tag that
disagreed with the header (--locale de-DE sent de-DE,de,en but reported
["de-DE","de"]). HttpHeaders.AcceptLanguage owns both shapes and is also
the CDP override type.
ICU canonicalizes a BCP 47 tag read from LC_ALL itself, script subtag
included, so the POSIX id conversion is gone; it dropped the script and
turned zh-Hans-TW into Traditional Chinese.
Also: the CDP handler keeps validateUserAgent's verdict instead of scanning
for Mozilla twice, the override is cleared unconditionally on context
teardown instead of through a flag, and the flags are sentinel strings so
Platform passes them to setenv without copying.
navigator.language was hard-coded to en-US and Accept-Language was a
constant, while Intl, toLocaleString and Date followed the host process
environment. On a de_DE host a page saw navigator.language === "en-US"
next to German number formatting, a mismatch fingerprinting scripts look
for, and the same page rendered differently across machines.
Follow Chrome's --lang rule: one configured tag drives navigator.language(s),
the Accept-Language header and ICU's default locale. --locale defaults to
en-US, so Intl is now en-US on every host instead of whatever LANG says.
--timezone sets the IANA zone Date and Intl use; absent, the host zone stays.
Both are applied by writing LC_ALL and TZ before V8 initializes ICU, which
reads them lazily. Platform.init is the first call in App.init, before any
thread exists, so setenv is safe there.
CDP Emulation.setUserAgentOverride.acceptLanguage, which Playwright sends
for its locale option, now overrides the header and navigator.languages
for the browser context's lifetime, mirroring the user agent override, and
applies even when the Mozilla user agent is refused.
Emulation.setLocaleOverride and setTimezoneOverride stay no-ops: changing
ICU's defaults at runtime needs new zig-v8-fork bindings.
A misspelt value for an enum-typed flag now names the nearest tag as
did_you_mean. --dump's peeking validator treats a word within two
edits of a format name as that mistake rather than as the url, so
--dump htmx no longer becomes a second url.
tagNames moves from Config into cli so both can use it.
change --http-timeout default to 15000 (up from 5000)
change --http-connect-timeout default to 8000 (down from 300000, curl's default)
5 second _total_ transfer time can be a little tight. I generally don't see a
good reason to overly limit this value.
Worth noting that help.zon said the default for --http-timeout was 10000, but
it was, in fact, 5000.
Hopefully this improve situations like https://github.com/lightpanda-io/browser/issues/3395
which I believe are due to slow proxies.
In order to support Selenium the way people are used to, it looks like we need
to support both WebDriver classic (WebDriver) and WebDriver BiDi (BiDi). Typical
scripts look like a mix of the two, e.g. using WebDriver to control the browser
and using BiDi to receive notifications. This commit:
1 - adds a --protocol (cdp|webdriver) CLI argument to the `serve` command to
enable one or the other protocol (defaulting to CDP)
2 - adds basic WebDriver endpoint to let a Selenium client connect. This
implementation is hackish and sits on top of our simple Handshake handler.
The handshake handler is well past its original design. Serving /json/version
and /metrics from it was one thing. But Driving the entire browser session? This
will get a follow up PR.
https://github.com/lightpanda-io/browser/issues/3348
When set to 1.1, libcurl is configured to only offer HTTP 1.1. By default, or
when set to "auto", it's up to libcurl to decide how to connect. This maps to
libcurl's CURL_HTTP_VERSION_1_1 and CURL_HTTP_VERSION_NONE.
LP.configureCDP now takes an `httpVersion` field which can be "1.1" or "auto"
to control that specific browser session. Ideally this is called prior to any
navigation.
adds --dump screen.pdf and CDP's Page.printToPDF (along with IO.read and
IO.read). Builds ontop of https://github.com/lightpanda-io/browser/pull/3231
This generates a text-based PDF, not just our PNG placed into a PDF. The biggest
change to the existing (isolated) code was adding href information so that links
are actually clickable.
Like 3231 the hard parts are all Claude but isolated. The one place it diverged
from the plan was NOT using the subsetter rust crate for trimming the size of
the embedded font. It felt that the dependencies were too much (e.g. a second
version of skrifa, ...). One subsetter was removed, the pdf rendering, while
still a blackbox, was moved from Rust to Zig (pdf.zig), which is kind of nice.
By default, iframes and workers no longer loaded. Use `--load-resources iframe`
and `--load-resources worker` to restore the previous behavior. The disabling
makes resource loading more consistent.
To further make things more consistent, Config seems the following changes:
1. remove `--timeout` from `serve` which does nothing but has printed a
deprecation warning for a long time
2. added .deprecated field to CLI config flags which now logs the specified
deprecated warning when used
3. `--log-filter-scopes` is deprecated in favor of `--log-scopes`
4. `--disable_subframes` is deprecated. Iframe loading is disabled by default,
use `--load-resources iframe` to enable iframe loading
5. `--disable_workers` is deprecated. Worker loading is disabled by default,
use `--load-resources worker` to enable worker loading
6. `--enable_external_stylesheets` is deprecated. Stylesheets remain disabled
by default. Use `--load-resources stylesheet` to enable loading external
stylesheets
CLI log parameters now alter the logger behavior on parse. This helps minimze
the window where default log settings are in-play. It also means things like
this work:
```
./lightpanda --log-level fatal --disable_subframes --log-level warn
```
More seriously, there's now an optional `beforeParse` fired once the mode is
known. This is used by mcp to set the default log level to logfmt. Previously
this was done much later and could easily result in a mix of pretty and logfmt
logs.
A wait failure on one URL aborted the whole multi-URL fetch with no
output, the wait_ms budget was computed once before the loop so N pages
could take N times the budget, navigation failures exited 0 with an empty
dump, and HTTP 4xx/5xx were indistinguishable from success by exit code.
Each page now has its own error slot: wait, navigation and dump failures
are recorded there instead of returned, every page is still written (the
JSON envelope carries the name under "error"), then the first failure is
returned so the process exits 1. The remaining budget is recomputed per
page. --fail-on-http-error turns a status >= 400 into exit 22, curl's
code for the same condition; the dump is written first either way.
--strip-mode was silently ignored for --dump markdown, and fetch had no
way to scope a dump to one element or cap its size although the markdown
and html tools have both.
--selector dumps the first matching element in any dump mode, --max-bytes
caps html and markdown with the tools' truncation marker, and markdown
now honors strip ui (images; scripts, styles and hidden elements are
never rendered, so the other groups don't apply).
MCP mode unconditionally set log.opts.format = .logfmt after the
--log_format flag had already been applied, so `--log_format pretty`
was silently ignored. It also ran after the "starting server" line,
which came out in the pre-override format in debug builds.
Move it into Config.logFormat() as a mode-aware default, mirroring
what logLevel() does for agent mode: null -> logfmt for mcp, and an
explicit flag wins.
Add an optional per-host rate limit. This currently only applies to the top-
level navigation. This makes it simpler to implement and simpler to reason
about. The full load of a page is only ever delayed at its head, not
sporadically through the page loading.
The use-case where a RateLimiter is most useful is when the browser is crawling
multiple pages of the same site, and in that case, the top-level rate limit
still applies some degree of limit to any linked resources (e.g. a JS on a
different host).
`--http-nav-delay` is the delay, in milliseconds, to apply to top level
navigates per host. Currently defaults to 0 (disabled).
`--http-nav-burst` is the burst allowed per host. Defaults to 1 (has no impact
when `--http-nav-delay` is disabled).
Adrià Arrufat