An unknown --flag now logs the nearest accepted spelling within two
edits as did_you_mean, painted green next to the red typo in the pretty
log format; logfmt stays plain. A bare first argument within two edits
of a command name is rejected with the same hint instead of being
fetched as a url.
The Levenshtein helper moves from SlashCommand into string.zig so the
agent REPL and the CLI share it, with the table widened to fit the
longest flag name.
Significant rework of the CDP/BiDi server. There are two main changes:
1 - poll replaced with EPoll/Kqueue (1)
2 - make http serving a first class citizen
The change from poll -> epoll/kqueue isn't performance driven, it's just about
tighter code. Both epoll and kqueue let you associate arbitrary data with a
socket, so we don't need to keep arrays in sync in order to associate a socket
with a CDP by index. They both provide some event/notification mechanism, which
is cleaner than the pipe required by poll.
The poll -> epoll/kqueue change could almost have been mechanical. Making HTTP
a first class citizen is the more significant of the two changes
In `main`, a new connection always spawns a thread and, until does its own
little read loop until the connection is upgraded. This is not efficient, it
uses up a connection slot, and it's inconsistent with the final WebSocket
connection which _is_ polled off the main loop. Using up a slot means that
keepalive isn't possible, else HTTP connections would quickly use up all
available slots/threads.
This commit parses and serves HTTP requests on the main thread (safe
because none of the processing is blocking). The approach is better streamlined
for HTTP requests which never upgrade (/metrics, WebDriver) without causing
any performance overhead for those that do. It simplifies some things (e.g. an
"http" socket or a "websocket" socket is monitored and read in a similar manner
(on the main loop)). It makes other things more complicated; the flow is no
longer accept -> spawn -> upgrade -> websocket loop. It's loop -> accept -> loop
-> process -> (http | ws).
This is built ontop of the BiDi branch because (a) WebDriver is what needs
better HTTP support and (b) some of the more mechanical changes already exist
in that branch (e.g. src/cdp/, src/server.zig -> src/server/*)
(1) kqueue landing in 2 commits from now on this branch.
We currently have 1 note: it prints the server's listening address:port. Note
is a special un-ignorable level. This keeps the "note" level, but logs it under
a new scope: "note", so that it _can_ be silenced with a `--log-filter note`.
Add a new note, on startup, that displays tips. Currently, only displays when
--obey-robots is not enabled:
NOTE note : config tips . . . . . . . . . . . . . . . . . . . [+0ms]
robots = use '--obey-robots' to use a sites robots.txt
meta = use '--log-filter note' to silence this message
1. Abstract "Driver". Non-CDP things that referenced *CDP now reference a Driver
2. Move the NodeRegistry out of CDP. This created an artificial link between
agent / mcp and CDP
3. Add BiDi driver with enough to navigate to a page
By default, iframes and workers no longer loaded. Use `--load-resources iframe`
and `--load-resources worker` to restore the previous behavior. The disabling
makes resource loading more consistent.
To further make things more consistent, Config seems the following changes:
1. remove `--timeout` from `serve` which does nothing but has printed a
deprecation warning for a long time
2. added .deprecated field to CLI config flags which now logs the specified
deprecated warning when used
3. `--log-filter-scopes` is deprecated in favor of `--log-scopes`
4. `--disable_subframes` is deprecated. Iframe loading is disabled by default,
use `--load-resources iframe` to enable iframe loading
5. `--disable_workers` is deprecated. Worker loading is disabled by default,
use `--load-resources worker` to enable worker loading
6. `--enable_external_stylesheets` is deprecated. Stylesheets remain disabled
by default. Use `--load-resources stylesheet` to enable loading external
stylesheets
CLI log parameters now alter the logger behavior on parse. This helps minimze
the window where default log settings are in-play. It also means things like
this work:
```
./lightpanda --log-level fatal --disable_subframes --log-level warn
```
More seriously, there's now an optional `beforeParse` fired once the mode is
known. This is used by mcp to set the default log level to logfmt. Previously
this was done much later and could easily result in a mix of pretty and logfmt
logs.
1 - TEST_VERBOSE is now off by default
2 - There's a afterEach callback that is automatically run after each tests, it:
a - clears the log filters
b - resets the test arena
3 - LogFilter replace with
a - testing.silenceLog(&.{...scopes...}); to silence all logs for the given
scopes.
b - testing.expectLog(&.{...scopes}); to set log expectations, 1 per log.
The goal here isn't so much to expect logs (though, you can do that),
but rather to silence an expected # of logs, without silencing more.
Zig 0.16 changed clocks/timestamps. Our migration to Zig 0.16 took the path of
least resistance. For example, we kept 'monotonic' and 'real' even though Zig
0.16 renamed them to 'boot' and 'clock'
This commit tries to standardize all timestamp usage to: lp.datetime.timestamp
and lp.datetime.milliTimestamp using the new zig 0.16 names ('boot' and 'clock')
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
`--log-filter-scopes` previously took a comma-separated list of scopes to
suppress. Extend it with explicit include/exclude directives applied
left-to-right:
-X filter out scope X
X alias for -X (backward compatible)
+X filter in scope X
all target every scope
This makes "filter everything except one scope" expressible, e.g.
`-all,+cdp` suppresses all logs except `cdp`. Bare scope names keep their
old meaning, so existing invocations like `http,unknown_prop` are
unaffected.
Internally, the suppressed-scope list is replaced by a resolved per-scope
`scope_enabled` boolean array built via `log.resolveFilterScopes()`, so the
hot-path `enabled()` check is a single array index. Filtering remains
Debug-only, as before. `testing.LogFilter` is adapted to the new
representation; all existing call sites are unchanged.
Adds a unit test for resolveFilterScopes (default, backward-compat,
-all,+cdp, ordering) and updates the help text.
`note` is the highest log level (e.g. cannot be turned off), and thus will
always be displayed. The purpose is to make sure that people who use --port 0
can see the listening port WITHOUT giving up log-level filtering (e.g. having
to use --log-level info JUST to see the listening port).
I'm personally a fan of using .note only when (port == 0), but we can tweak it
later if people complain about it showing up despite using a warn/error/fatal
log level.
Experiment to see if this shrinks the build by any significant amount. Some
investigation did surface this as some of the code we control directly which
did stand out.
- Add diagnostics to command parsing to report invalid field values.
- Fix UTF-8 truncation in spinner to avoid splitting codepoints.
- Fix swapped arguments in log error formatting.
- Distinguish "null" values from missing elements in verifier.
- Handle MCP tool cancellation and timeouts with specific error codes.
This is to pave the way for introducing a new "Page" container, which will take
over the page lifecycle currently burdening Session. The ultimate goal of that
is to allow the Session to have multiple pages (mostly for better transitions
between pages), which is hard to do now since the Session has so much state.
This rename was aggressive, e.g. currentPage() -> currentFrame() so that, when
the new Page container is added, you won't see "currentPage()" and wonder:
"Does 'currentPage' mean the new Page container, or the Frame (which
used to be called Page)".
This adds an app.storage which is a union around configurable storage engine
(currently, only sqlite).
It is _not_ being used anywhere right now. The goal is to get feedback on
the implementation and then move cache to it.
This doesn't expose a generic query API. The goal is that the storage will
expose high level methods, e.g. `cacheGet(req: CacheGetRequest)` and every
storage engine will translate the `storage.CacheGetRequest` as needed.
A thin wrapper around the Sqlite C api is included, e.g. exec(SQL, .{args}) a
`rows` and `row` fetcher. A connection pool is included. By default, an
in-memory DB is currently created. And a `migrations` table with an id of `1`
is created/inserted. I don't imagine needing fancy migratations.
Adds a new `mcp` run mode to start an MCP server over stdio.
Implements tools for navigation and JS evaluation, along with
resources for HTML and Markdown page content.
It now returns a Caught struct which contains all information. The Caught struct
can be logged directly, providing more consistent logs for caught errors.
Currently, this hooks a single log.Interceptor into the logging framework, but
changing it to take a list shouldn't be too hard. Biggest issue is who will own
it, as we'd need an allocator to maintain a list / lookup (which log doesn't
currently have).
Uses logFmt format, and, for now, always filters out debug messages and a few
particularly verbose scopes.
There is some risk to this change. The first is that I made a mistake. The
other is that one of the APIs that doesn't currently return an error changes
in the future.
Follows up on https://github.com/lightpanda-io/browser/pull/994 and replaces
the jsRunner with a new page.navigation-based test runner.
Currently only implemented for the Window tests, looking for feedback and
converting every existing test will take time - so for a while, newRunner (to be
renamed) will sit side-by-side with jsRunner.
In addition to the benefits outlined in 994, largely around code simplicity and
putting more of the actual code under tests, I think our WebAPI tests
particularly benefit from:
1 - No need to recompile when modifying the html tests
2 - Much better assertions, e.g. you can assert that something is actually an
array, not just a string representation of an array
3 - Ability to test some edge cases (e.g. dynamic script loading)
I've put some effort into testing.js to make sure that, if the encapsulating
zig test passes, it's because it actually passed, not because it didn't run.
For the time being, console tests are removed. I think it's more useful to have
access to the console within tests, than it is to test the console (which is
just a wrapper around log, which is both tested and heavily used).
There's a flaky performance test that I wanted to fix (1). This led to a couple
changes.
1 - Add timestamp() and milliTimestamp() to datetime.zig. Reduce some code
duplication and use better clock_ids where available
2 - Change Performance API to use milliTimestamp and store a u64 instead of a
f64. While the spec says a float, Firefox deals with u64 and implicit
conversion is always available. Makes our APIs simpler.
(1) - https://github.com/lightpanda-io/browser/actions/runs/17313296490/job/49151366798#step:4:131
Test speed has been improved only slightly by tweaking a 2-second running tests.
Build has been improved by:
1 - moving logFunctionCallError out of js.Caller and to a standalone function
2 - removing some non-generic code from the generic portions of the logger
Caller.getter and Caller.setter have been removed in favor or calling
Caller.method. This wasn't previously possible - prior to our v8 upgrade, they
had different signatures.
Also removed a largely unused parser/str.zig file.
1 - Add a custom console.lp function to make our debug logs stand out from
script logs.
2 - In some cases, significantly improve how JavaScript values are serialized
in debug logs and in console.log.
1 - Make log_level a runtime option (not a build-time)
2 - Make log_format a runtime option
3 - In Debug mode, allow for log scope filtering
Improve the general usability of scopes. Previously, the scope was more or less
based on the file that the log was in. Now they are more logically grouped.
Consider the case where you want to silence HTTP request information, previously
you'd have to filter out the `page`, `xhr` and `http_client` scopes, but that
would also elimiate other page, xhr and http_client logs. Now, you can just
filter out the `http` scope.
Adrià Arrufat