If the server gives a status reason, report it as-is. Only default to the
zig code->reason map when one isn't given.
Also, don't force a 407 status code when auth_challenge is present.
Responses with specific status (e.g. 204) should always have a null body. Also
adds validation to Response constructor (e.g. can't provide an invalid status).
Improves a handful of WPT tests:
fetch/api/response/response-error.any.html
fetch/api/response/response-static-json.any.html
Directly improve a couple slow tests (crash_handler actually generating crash
dumps on some systems, e.g. mine). The runWebApiTest prefers to poll for work
when possible rather than a blind sleep -> check loop. (e.g. if we have
websocket connections, prefer an http tick).
For me, it's 22s -> 17s.
This adds the shell for ServiceWorker, behind
`--experimental-features serviceworker`.
It's pretty useless as-is. We don't have the CacheStorage API (next) and don't
have the fetch interceptor (next next). But as-is, the change is quite big but
thankfully largely isolated.
Currently, the HttpClient owns the inbox and its borrowed by the Link. This is
a bit backwards, but it also means that we can't eagerly create a Link: the
Link needs the inbox, so it needs the HttpClient, which is created by the
Browser (which creates an Isolate).
Remember, the Inbox is one of the few things shared between the main thread
and the worker, so either end can own it and the other can borrow it.
This switches the ownership so that the HttpClient now borrows the Inbox from
the Server's side of the Link (the WebSocket).
The main goal of this change is to prepare for more advanced HTTP WebDriver
flows. The more we can create _without_ a Browser, the fewer edge cases we have
to deal with (Browser because it's expensive and has to be created on the
Worker thread due to how V8::Isolate works).
Give it one pass through some Claude fuzz testing. Add a max message size,
protect against weird interactions during a shutdown and we had some pending
accepts. Put a time limit on blocked writes.
Adds resource timing, e.g. `performance.getEntriesByType("resource")`.
The `resource-timing` WPT category is currently at 4.6%, and this is a first
step at improving it. It also hopefully fixes https://github.com/lightpanda-io/browser/issues/3359
This is more complicated than I thought because there's a "Timing-Allow-Origin"
header that a server can include which hides some of the data if the request
doesn't come from the listed origin. And that, of course, interacts with
redirects.
(The DOMException change is seemingly random, but it came up in one of the WPT
cases I was looking at).
window.stop() is less destructive than other mechanisms we have. For one, it
seems largely isolated to pending or inflight HTTP requests. For anther, it
keeps the page intact.
To achieve this, HttpClient gains an `cancelRequests` which is a gentler version
of `abortOwner`. It cancels inflight/pending HTTP requests, which results in
error callbacks (not shutdown callbacks) firing.
Just like https://github.com/lightpanda-io/browser/pull/3189 I ran into the
problem that I couldn't distinguish between an HTTP request that was canceled
because of user-action (e.g. calling window.stop(), or xhr.abort()) and an HTTP
request that was internally aborted. These now have distinct errors/flows so
that we can present the correct state. Most places that aborted now all
transfer.cancel() which results in a distinct `error.TransferCanceled` (some
places still abort -> `error.Abort`). It should be possible to revisit 3189 now.
The CDP "Page.stopLoading" now hooks into this new behavior. Fixes
https://github.com/lightpanda-io/browser/issues/3351
In order to support Selenium the way people are used to, it looks like we need
to support both WebDriver classic (WebDriver) and WebDriver BiDi (BiDi). Typical
scripts look like a mix of the two, e.g. using WebDriver to control the browser
and using BiDi to receive notifications. This commit:
1 - adds a --protocol (cdp|webdriver) CLI argument to the `serve` command to
enable one or the other protocol (defaulting to CDP)
2 - adds basic WebDriver endpoint to let a Selenium client connect. This
implementation is hackish and sits on top of our simple Handshake handler.
The handshake handler is well past its original design. Serving /json/version
and /metrics from it was one thing. But Driving the entire browser session? This
will get a follow up PR.
1. Abstract "Driver". Non-CDP things that referenced *CDP now reference a Driver
2. Move the NodeRegistry out of CDP. This created an artificial link between
agent / mcp and CDP
3. Add BiDi driver with enough to navigate to a page
https://github.com/lightpanda-io/browser/pull/3320 made Mime.zig aware of
`application/octet-stream`, but Mime is still lossy, e.g.
"text/html; charset=utf-8" -> "text/html".
This commit dupes the response's content-type in the XHR's arena and use that
value as-is.
By default, iframes and workers no longer loaded. Use `--load-resources iframe`
and `--load-resources worker` to restore the previous behavior. The disabling
makes resource loading more consistent.
To further make things more consistent, Config seems the following changes:
1. remove `--timeout` from `serve` which does nothing but has printed a
deprecation warning for a long time
2. added .deprecated field to CLI config flags which now logs the specified
deprecated warning when used
3. `--log-filter-scopes` is deprecated in favor of `--log-scopes`
4. `--disable_subframes` is deprecated. Iframe loading is disabled by default,
use `--load-resources iframe` to enable iframe loading
5. `--disable_workers` is deprecated. Worker loading is disabled by default,
use `--load-resources worker` to enable worker loading
6. `--enable_external_stylesheets` is deprecated. Stylesheets remain disabled
by default. Use `--load-resources stylesheet` to enable loading external
stylesheets
CLI log parameters now alter the logger behavior on parse. This helps minimze
the window where default log settings are in-play. It also means things like
this work:
```
./lightpanda --log-level fatal --disable_subframes --log-level warn
```
More seriously, there's now an optional `beforeParse` fired once the mode is
known. This is used by mcp to set the default log level to logfmt. Previously
this was done much later and could easily result in a mix of pretty and logfmt
logs.
Our `Arena` from the `ArenaPool` now tracks if it's already been released. On
a subsequent release, it panics then and there. Without this, the code will
almost certainly panic anyways, but it will panic in a seemingly unrelated
place. Hopefully this makes identifying future cases of this easier (since we'll
get the stack trace of the re-releaser).
Fix 3 separate memory issues, all edge cases.
1 - an XHR error handler that re-open/sends would incorrectly null the new
transfer (maybe this isn't that odd, maybe it's a common retry-on-error).
2 - On a curl_easy_pause error (from WebSocket.zig) we now unqueue the just-
queued message, because the error will errdefer the message arena to be
cleaned up.
3 - ScriptManager now cleans up after itself on a failure prior to submit()
being called.
Network is less cohesive than https://github.com/lightpanda-io/browser/pull/3242
would indicate. It has two distinct and _completely_ separate responsibilities.
1 - It acts as the base for each HttpClient, providing a shared connection pool
(for http and ws) and access to the process wide Cache, RobotStore,
WebBothAuth, certificates, ...
2 - It accepts, polls and reads from CDP connection
There is zero relationship between these, and it's a particularly bad place for
this duality to exist because both parts are, in their own way, the main
multi-threaded junction in the system.
This commit is purely mechanical it:
1 - Keeps network as the base for each HttpClient.
2 - Extract the CDP interaction into the existing Server.zig
3 - Gives mcp's HttpServer its own accept loop
Builds ontop of https://github.com/lightpanda-io/browser/pull/3200 to centralize
header enforcement and standardize merge vs overwrite header logic.
The API is still a `setHeader` and `appendHeader`, with a source, but set/append
both are thin wrappers around private `putHeader`. putHeader blocks overwriting
restricted headers (user-agent). The `source` acts as a priority (ordered enum)
which further restricts the header AND depending on whether set or append were
called, controls if the value is overwritten or appended to.
3200 had an always-append which can cause problems, e.g. a script setting
Accept-Language: fr would have results in the value being appended to the
default, e.g.: 'en-US,en;q=0.9, fr'.
On a redirect, recalculate the Referer header based on the new target.
Also, allow SVG anchors to be clicked (doesn't seem related, but it came up
in referrer-policy WPT tests).
1 - TEST_VERBOSE is now off by default
2 - There's a afterEach callback that is automatically run after each tests, it:
a - clears the log filters
b - resets the test arena
3 - LogFilter replace with
a - testing.silenceLog(&.{...scopes...}); to silence all logs for the given
scopes.
b - testing.expectLog(&.{...scopes}); to set log expectations, 1 per log.
The goal here isn't so much to expect logs (though, you can do that),
but rather to silence an expected # of logs, without silencing more.
BodyInit.extract returned an empty body for ReadableStream request
bodies, so fetch POST with duplex:half silently sent Content-Length: 0.
Drain closed streams synchronously via collectBodyBytes; reject open,
locked, or errored streams with TypeError instead of sending nothing.
Fixeslightpanda-io/browser#3052
Expand mime type to be aware of more XML types. DOMParser and XHR now use the
existing Parser.parseXML (via a Frame helper), rather than the HTML parser. Both
these APIs predate the addition of parseXML.
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
Builds on top of the recently added support for <link rel=preload...> and
<link rel=modulePreload...> to scan the HTML for script tags to preload. I.e.
adds script preloading without actually having any preload hits.
At least for this first pass, I opted for a simple approach which leverages
are fully buffered HTML body and html5ever's tokenizer to prescan the body and
kickoff any script fetching before starting the complete parse.
There are doubtless cases where this will either decrease performance and/or
increase memory usage. E.g. a site with no script gets its html scanned twice
and loading multiple blocking scripts in parallel obvious uses more memory
than loading them sequentially. But for most sites and I think most use-cases,
the impact should range between neutral to significantly faster loads.
This is something most browsers do.
This fixes a bug with preload script so that they're actually used for
async/defer scripts (previously, only used for blocking scripts).
More importantly, this cleans up the ScriptManager's addFromElement. For example
inline scripts are handled in their own function, which means we aren't weaving
the two modes in a single function. It also allows the inline-script to use
a better-sized arena. The goal for this cleanup is the follow up commit which
will bring a pre-parse step to preload scripts without an explicit <link> hint.
The EventSource API only requires a simple line-based parser. It's pretty
straightforward. It's RC'd by v8 and itself, but also by scheduled tasks (e.g.
to reconnect).
The bigger change is to the HttpClient to support streaming requests. This
changes to things:
1 - A transfer doesn't only deliver() once
2 - A transfer can be in the multi while delivering
Neither of these are surprising, but they both add complexity.
When an iframe has "loading=lazy" it no longer delays the parent's "load" event.
It's still loaded (subject to `--disable-subframes` parameter).
Many sites use loading=lazy for non-critical content (e.g. ads). On Chrome/
Firefox, these are only loaded when they're in the viewport, so sites usually
can't depend/rely on them being loaded.
Frame.loadExternalStylesheet fetches an external <link rel=stylesheet>
synchronously, which registers a blocking request for the frame. While that
blocking request is active the DeferringLayer holds back the completion
callbacks of every other in-flight transfer for the frame, so they don't run
JS on the parser stack. The blocking-<script> path and the worker path flush
those deferred completions once their synchronous fetch returns, but the
external-stylesheet path did not. As a result a <script defer> that finished
loading during the stylesheet's blocking window stayed at complete == false:
the deferred-script queue never drained, so the deferred scripts never ran and
DOMContentLoaded / the load event / readyState -> "complete" never fired —
the document was stuck at readyState "loading" even though every request
completed with HTTP 200.
Flush the frame's deferring layer after the synchronous stylesheet fetch,
mirroring the other two synchronous-request call sites.
Closes#2842
Follow up to https://github.com/lightpanda-io/browser/pull/2789 which adds
better multi-page support for runner. When waiting, callers have the choice
to wait for a specific frame, all current frames, or a given list of frames. The
last one, a given list of frames, is the most flexible, allowing callers to
provide an `until` per frame AND receive a per-frame result.
waitForSelector and waitForScript continue to be per frame (which is now
explicitly given).
Like 2789, while the changes aren't insignificant (Runner is doing important
work), a number of files were touched either purely because of changes in tests
or other superficial changes, e.g. `session.runner` is now infallible.
Replaces the 1 (+1 inflight) page design of session for an unlimited number of
pages. The main goal is to support a more efficient async goto (1). Without this
commit, async goto has two implementation: multiple browser, which is resource
intensive (thread + isolate per page), bolting it onto sub-frames (like iframe
or popups). The issue with the 2nd version (which I originally suggested) is
that most resources are tied to the Page. So even if an async goto "page" (which
would map to a Frame) is released, most things stay in memory, including the
DOM (page.factory) and the V8::Context.
This new approach adds multiple page support to sessions. The advantage is
pretty clear: the existing memory model (page-tied resources) becomes a strength
of the design, rather than a weakness.
This change is not tirivla, but the diff is inflated by 2 large mechanial
changes, so it isn't _that_ big either. That said, I'd divide this into four
parts.
1 - The old concept of _active/_pending is now baked into the Page itself. A
Page has a `replaces: ?*Page = null` and `replacement: ?*Page = null` field.
2 - Because of #1 above, the Session now just has an `pages: ArrayList(*Page)`.
As much as possible, single-page APIs, like `removePage` no longer exists. We're
trying to present a consistent multi-page API. See #3.
3 - References to *Page and *Frame have always been dangerous. For example,
`lightpanda.fetch` has a block to scope `frame`:
```zig
{
const frame = try session.createPage();
// frame isn't safe to use after navigate, it can be swapped out
_ = try frame.navigate(...)
}
```
While we still hand out *Page and *Frame (user-beware), some of the more
important APIs now take a frame_id which Session can resolve. Furthermore,
createPage now returns a PageHandle which is a safe wrapper around a page/frame.
4 - Two large mechanical changes were made:
a - Many tests were superficially changed to account for new naming or use
a new test-helper to preserve the single-page illusion (because 99% of
tests _are_ single-page)
b - Many CDP changes where `bc.session.currentFrame()` -> `bc.mainFrame()`).
This isn't to say CDP changes are meaningless, but it's mostly 1 change
about how the "main" target/frame_id is tracked (by CDP itself, rather
than the Session) that required a number of superficial changes to
accomodate
`Runner` remains largely single-page focused. Runner and some MCP/Agent tools
continue to be tied to the "currentFrame" or "primaryPage". The Runner is
something I want to address in a follow-up PR, but we need to figure out what
it means to "wait" for multiple pages.
As much as possible, Session becomes multi-page native and has no concept of
a special/first/primary/main page. Users of session become responsible for
tracking pages of interest.
This change is not trivial, but the diff is inflated by 2 large mechanical
changes:
1 - Many tests
(1) https://github.com/lightpanda-io/browser/pull/2759
Browser.setDownloadBehavior was a noop, so Lightpanda had no file-download
path. A response with Content-Disposition: attachment is now streamed to disk
under downloadPath, and Page.downloadWillBegin / Browser.downloadProgress are
emitted when eventsEnabled.
Fixes#2701
`--log-filter-scopes` previously took a comma-separated list of scopes to
suppress. Extend it with explicit include/exclude directives applied
left-to-right:
-X filter out scope X
X alias for -X (backward compatible)
+X filter in scope X
all target every scope
This makes "filter everything except one scope" expressible, e.g.
`-all,+cdp` suppresses all logs except `cdp`. Bare scope names keep their
old meaning, so existing invocations like `http,unknown_prop` are
unaffected.
Internally, the suppressed-scope list is replaced by a resolved per-scope
`scope_enabled` boolean array built via `log.resolveFilterScopes()`, so the
hot-path `enabled()` check is a single array index. Filtering remains
Debug-only, as before. `testing.LogFilter` is adapted to the new
representation; all existing call sites are unchanged.
Adds a unit test for resolveFilterScopes (default, backward-compat,
-all,+cdp, ordering) and updates the help text.
Every 300-399 status was routed into the redirect path, where the missing
Location header fails the transfer with error.LocationNotFound — the old
document is destroyed and nothing replaces it. Per the fetch standard's
HTTP-redirect fetch ("If locationURL is null, then return response") and
RFC 9110 §15.4, a 3xx without Location is a normal final response whose
body must be delivered. Guard both the redirect dispatch and the 3xx
skip-body branch on the header's presence so such responses fall through
to the regular completion path.
Closes#2713
Karl Seguin