ScriptManager, XMLHttpRequest.zig, Fetch, Workers, etc. all take ownership (aka
dupe) the HTTP response from HTTPClient. They all have a headerCallback that
does something like:
```zig
if (transfer.getContentLength()) |cl| {
try self.body.ensureTotalCapacity(self.arena, cl);
}
```
But in all non-streaming cases (which is most cases), the HttpClient buffers
the response and only calls the headerCallback _after_ the body has been
received. Rather than relying on "Content-Length" header, the body buffer can
be sized to the exact body length. Why does this matter? Because the
Content-Length is the length of the body on the wire, and if the body is
compressed (like almost all .js files are), it will under-report the final
body length AND, because most callers are using an arena, the buffer growth
will retain more memory than it should.
This adds a `transfer.bodyLen()` method. Callers which dupe the body now use
this rather than the Content-Length (Content-Length is still used, e.g. for
XHR progress report).
Adds a `clutter` option to --strip-mode. This is based on readability.js. It
isn't a direct port (e.g. it doesn't strop bylines). It fallsback to `shell` if
it strips too much (and shell itself can fallback to not stripping anything).
But clutter rarely fallback to shell, only when a page is very small or when
it strips out _a lot_.
Also expanded shell to look at class names and ids.
Some additional API changes:
- Add strip-mode support to pdf/png generation.
- Add LP.dump which provides greater content gathering capability to CDP,
exposing most `fetch` dump-related parameters (e.g. format, strip, selector,
...)
The new "--strip-mode shell" is designed to try to remove non-content elements
such as the header and footer. The end goal is to use readibility.js test cases
as a baseline, but this isn't a port of readibility.js.
This is just the basic implementation of this, e.g removing a few key tags, e.g.
<header>, <footer> and considering some specific roles.
Even if --strip-mode shell is used, we might decide to stick with a whole dump:
it's better to strip not enough than to strip too much. This currently works by
measuring the ratio of non-link text of the stripped vs unstripped page.
tighten socket ownership (on error paths)
allow reaper to be disabled
Handle window where link is being destroyed, worker is still alive, and client
attempts to re-link.
This is a small step towards WebDriver supports (non-bidi). It allows creating
and deleting a BiDi "Session" (e.g. a worker). It also allows attaching a BiDi
driver to an HTTP-created BiDi session (the typical selenium startup flow).
This change unblocks the most basic setup/teardown of Selenium, so it still
isn't enough to actually use a Selenium script as-is. But it's significant
because it models a worker (thread) that isn't tied to a WebSocket, something we
haven't had before.
A consequence of a pure HTTP Session is that we don't have a clear cleanup
signal. There is no "the socket is disconnected". There's a new HTTP reaper
which kills HTTP Sessions after --http-session-timeout. It's expected that
drivers properly DELETE /session/:id. I imagine we're going to run into
--cdp-max-connections limits and need to tweak this code. BUT, this entire flow
is only enabled with --protocol webdriver, so it won't impact exiting CDP users.
Three parameters every Playwright and Stagehand session sends were parsed
and then only logged as not implemented:
- Page.addScriptToEvaluateOnNewDocument runImmediately now also evaluates
the script in the current document, in the requested world.
- Emulation.setDeviceMetricsOverride screenWidth/screenHeight now back
window.screen, kept on the viewport override next to width/height; 0
keeps the current value, as for the other dimensions.
- Browser.setDownloadBehavior browserContextId is checked against the
loaded context, as the Storage commands already do.
Drivers focus a node before typing (chromedp's SendKeys calls DOM.focus,
then Input.dispatchKeyEvent). The method was unknown, so the keystrokes
went to the previously active element and every chromedp form fill was a
no-op. Resolve the node like the other DOM commands and call Element.focus,
which already handles focusability and the blur/focus event sequence.
With the StyleManager memo, a VisibilityCache/PointerEventsCache hit
costs the same hash probe as a memo hit, so the caches only added a
second probe per ancestor, a call_arena allocation per element, and
plumbing through SemanticTree, AXNode, CDP, links, ResizeObserver and
elementFromPoint. checkVisibilityCached becomes isVisible.
navigator.languages now lists the Accept-Language tags in order, which is
Chrome's contract, instead of a second derivation from the locale tag that
disagreed with the header (--locale de-DE sent de-DE,de,en but reported
["de-DE","de"]). HttpHeaders.AcceptLanguage owns both shapes and is also
the CDP override type.
ICU canonicalizes a BCP 47 tag read from LC_ALL itself, script subtag
included, so the POSIX id conversion is gone; it dropped the script and
turned zh-Hans-TW into Traditional Chinese.
Also: the CDP handler keeps validateUserAgent's verdict instead of scanning
for Mozilla twice, the override is cleared unconditionally on context
teardown instead of through a flag, and the flags are sentinel strings so
Platform passes them to setenv without copying.
navigator.language was hard-coded to en-US and Accept-Language was a
constant, while Intl, toLocaleString and Date followed the host process
environment. On a de_DE host a page saw navigator.language === "en-US"
next to German number formatting, a mismatch fingerprinting scripts look
for, and the same page rendered differently across machines.
Follow Chrome's --lang rule: one configured tag drives navigator.language(s),
the Accept-Language header and ICU's default locale. --locale defaults to
en-US, so Intl is now en-US on every host instead of whatever LANG says.
--timezone sets the IANA zone Date and Intl use; absent, the host zone stays.
Both are applied by writing LC_ALL and TZ before V8 initializes ICU, which
reads them lazily. Platform.init is the first call in App.init, before any
thread exists, so setenv is safe there.
CDP Emulation.setUserAgentOverride.acceptLanguage, which Playwright sends
for its locale option, now overrides the header and navigator.languages
for the browser context's lifetime, mirroring the user agent override, and
applies even when the Mozilla user agent is refused.
Emulation.setLocaleOverride and setTimezoneOverride stay no-ops: changing
ICU's defaults at runtime needs new zig-v8-fork bindings.
In main, there's a `getId`, and `getClassName` (etc...) getter on Element. But
these all `orelse ""`, because that's what the WebAPI wants. Internally though,
most code want the optional. The result is that _many_ places do:
```zig
el.getAttributeSafe(comptime .wrap("id"))
```
instead of:
```zig
el.getId()
```
This is a bit tedious AND, it means that when we improve `Element.getId` (1) no
internal caller benefits from it. This commit makes the element getters return
the optional (`?[]const u8`) and updates every callsite to use the new getter.
The `orelse ""` needed by the WebAPI is moved to the JsApi bridge.
(1) https://github.com/lightpanda-io/browser/pull/3457
The js.Execution is the API behind the Frame/WGS split, but the split is
actually held by the underlying js.GlobalScope. Most Execution methods are:
```zig
return switch (self.js.global) {
inline else => |g| g.isSameOrigin(url),
};
```
And that works well, except that in some cases, code has a js.Context, not an
js.Execution, and they need to do the same inline switch.
This commit moves GlobalScope from src/browser/js to src/browser (there's
nothing JS/v8 about it), and moves all those inline switches into it. The
js.Execution API stays the same (it forwards the call to js.global)
but all callers that directly inlined switched the js.global no longer do.
Tried to not to change function signature by relying on saturating addition; could've implemented differently, though I'm not sure if returning an error here would make a huge difference.
A trusted keydown for ArrowLeft/ArrowRight/Home/End (and ArrowUp/ArrowDown
on a single-line <input>) fell through Frame.user_input.editKey, which only
handled Backspace/Delete and printable keys, so the caret never moved, Shift
never extended the selection, and a plain arrow never collapsed one.
innerInsert's no-selection arm also appended to the end of the value instead
of inserting at the caret. Add a moveCaret helper to the shared text entry
mixin (byte offsets stepping over whole UTF-8 sequences, line moves bounded
by '\n'), route the keys to it from editKey, and insert at the caret.
Closes#3423
`select()`, `setSelectionRange()` and `howSelected()` in the TextEntry mixin
read the `_value` slot directly. That slot only holds an *assigned* value, so
a `<textarea>abcdef</textarea>` straight out of the parser has none and every
selection call collapsed the caret to 0 — while `textarea.value` returned
"abcdef" all along, since `getValue()` falls back to the child text node.
Route the three sites through `getValue()`, matching `innerDelete()`, which
already did. Per HTML, "set the selection range" clamps against the element's
API value, and a textarea's API value is its raw value.
`howSelected()` now also clamps: `selectionStart`/`selectionEnd` store their
argument verbatim, so an offset can outlive a shorter value and reach
`innerInsert()`'s `.partial` arm as an out-of-range slice index.
Closes#3421
macOS adds an internal 'was written' bit to F_GETFL once the fd has been
written to, so the exact comparison fails there (expected 6, found 65542)
even though the send timed out as intended and O_NONBLOCK is still set.
macOS delivers loopback traffic asynchronously, so LoopTest.accept could
call Server.accept before the handshake ACK landed (NotAccepted) and the
tests could read before the request bytes landed (WouldBlock, RST, and a
double disconnect that cast a -1 socket to usize in KQueue.socketEvent).
Poll for readiness first. Also accept any non-zero SO_KEEPALIVE: BSD
getsockopt returns the option bit, not 1.
InputEvent.initWithTrusted overwrote _bubbles/_cancelable/_composed for
every InputEvent right after Event.populatePrototypes had applied the
caller's options, forcing _cancelable = false. user_input.zig#allowEdit
asks for a cancelable beforeinput so a listener can veto the edit, but
preventDefault() is a no-op on a non-cancelable event, so the character
was inserted and `input` fired regardless.
Guard the flag block with `if (trusted)` — the shape KeyboardEvent
already uses — and derive _cancelable from the event type: `beforeinput`
is cancelable, `input` is not. Constructed events and
document.createEvent('InputEvent') now follow the EventInit dictionary
defaults instead of being forced to bubbling and composed.
Closes#3413
Karl Seguin