Adds a `clutter` option to --strip-mode. This is based on readability.js. It
isn't a direct port (e.g. it doesn't strop bylines). It fallsback to `shell` if
it strips too much (and shell itself can fallback to not stripping anything).
But clutter rarely fallback to shell, only when a page is very small or when
it strips out _a lot_.
Also expanded shell to look at class names and ids.
Some additional API changes:
- Add strip-mode support to pdf/png generation.
- Add LP.dump which provides greater content gathering capability to CDP,
exposing most `fetch` dump-related parameters (e.g. format, strip, selector,
...)
The new "--strip-mode shell" is designed to try to remove non-content elements
such as the header and footer. The end goal is to use readibility.js test cases
as a baseline, but this isn't a port of readibility.js.
This is just the basic implementation of this, e.g removing a few key tags, e.g.
<header>, <footer> and considering some specific roles.
Even if --strip-mode shell is used, we might decide to stick with a whole dump:
it's better to strip not enough than to strip too much. This currently works by
measuring the ratio of non-link text of the stripped vs unstripped page.
Debug and release builds each compiled their own copy of every C
dependency and of the Rust staticlib, because build.zig threaded the
top-level optimize mode into all of them. Under dev_fast the deps also
picked up the bundled-CRT target query, so even the same mode could not
share objects with a plain build.
Dependencies now build in ReleaseFast for the requested target, the way
the prebuilt V8 archive already works. Debug and release builds share
one set of cached dependency objects, and debug binaries run TLS, HTML
parsing, regex and sqlite optimized. -Ddebug_deps restores the old
behaviour for stepping into a dependency.
The Rust staticlib can only be shared by dropping the Debug-only memstats
feature: its single export, html5ever_get_memory_usage, was declared on
the Zig side but never called, and it pulled a jemalloc build into every
cold debug build. The Makefile override that existed for jemalloc's
nested make goes with it.
tighten socket ownership (on error paths)
allow reaper to be disabled
Handle window where link is being destroyed, worker is still alive, and client
attempts to re-link.
This is a small step towards WebDriver supports (non-bidi). It allows creating
and deleting a BiDi "Session" (e.g. a worker). It also allows attaching a BiDi
driver to an HTTP-created BiDi session (the typical selenium startup flow).
This change unblocks the most basic setup/teardown of Selenium, so it still
isn't enough to actually use a Selenium script as-is. But it's significant
because it models a worker (thread) that isn't tied to a WebSocket, something we
haven't had before.
A consequence of a pure HTTP Session is that we don't have a clear cleanup
signal. There is no "the socket is disconnected". There's a new HTTP reaper
which kills HTTP Sessions after --http-session-timeout. It's expected that
drivers properly DELETE /session/:id. I imagine we're going to run into
--cdp-max-connections limits and need to tweak this code. BUT, this entire flow
is only enabled with --protocol webdriver, so it won't impact exiting CDP users.
Drivers focus a node before typing (chromedp's SendKeys calls DOM.focus,
then Input.dispatchKeyEvent). The method was unknown, so the keystrokes
went to the previously active element and every chromedp form fill was a
no-op. Resolve the node like the other DOM commands and call Element.focus,
which already handles focusability and the blur/focus event sequence.
An alternation or a repeat keeps only whether its text may start and
end with a token character, and read that off one marker. An optional
non-token stretch there (`\/?x`) was taken as a definite non-token,
so `\/ads(\/?x|\/y)` was filed under "ads" while `/adsx` carries no
such token. What follows the stretch answers now.
Attempt to improve https://github.com/lightpanda-io/browser/issues/3199
Meant an leaner alternative to https://github.com/lightpanda-io/browser/pull/3440
3440 is close, but it adds 16 bytes to every VisibilityRule which isn't ideal
especially since the majority of these rules have no custom properties.
Let's make a few reasonable assumptions:
1 - Most custom properties don't have many distinct selectors
(--background-color might be defined on, :root, and .card and a few others)
2 - Most custom properties aren't queried from JS (Tailwind can define thousands
of custom properties, but they're used by the rendering engine, not from JS)
3 - Most selectors don't have custom properties
The design is to inverse what we do for visibility: lookup per property. We end
up with a property-name -> [(selector, value), (selector, value)] lookup. If
there's a query for --foreground, we O(1) to get the list of (selector, value)
and then iterate through this (hopefully) short list to get the value.
PLUS, we only parse the selector on the first query. So for those sites with
thousands of custom-properties, we're not wastefully storing / parsing the
selector on every build.
With the StyleManager memo, a VisibilityCache/PointerEventsCache hit
costs the same hash probe as a memo hit, so the caches only added a
second probe per ancestor, a call_arena allocation per element, and
plumbing through SemanticTree, AXNode, CDP, links, ResizeObserver and
elementFromPoint. checkVisibilityCached becomes isVisible.
Every ancestor-aware visibility query (checkVisibility, offsetWidth,
getBoundingClientRect, getComputedStyle().visibility, pointer-events, and
the Zig tree walkers) re-resolved each ancestor from scratch. Resolve an
element's display/visibility/opacity/pointer-events once and reuse it
while Page.style_version is unchanged.
The inline style attribute is scanned once for all four properties, the
pointer-events scan shares the same path, and a materialize-mode hit still
creates the inline style object that layout reads.
Bumped through Frame.styleChanged: with dom_version, and by the state
changes that alter a selector match without changing what live
collections see: checked/indeterminate, input values, custom element
definitions, character data (:empty), parser insertions and stylesheet
changes. Stamps the visibility memo added next.
navigator.languages now lists the Accept-Language tags in order, which is
Chrome's contract, instead of a second derivation from the locale tag that
disagreed with the header (--locale de-DE sent de-DE,de,en but reported
["de-DE","de"]). HttpHeaders.AcceptLanguage owns both shapes and is also
the CDP override type.
ICU canonicalizes a BCP 47 tag read from LC_ALL itself, script subtag
included, so the POSIX id conversion is gone; it dropped the script and
turned zh-Hans-TW into Traditional Chinese.
Also: the CDP handler keeps validateUserAgent's verdict instead of scanning
for Mozilla twice, the override is cleared unconditionally on context
teardown instead of through a flag, and the flags are sentinel strings so
Platform passes them to setenv without copying.
navigator.language was hard-coded to en-US and Accept-Language was a
constant, while Intl, toLocaleString and Date followed the host process
environment. On a de_DE host a page saw navigator.language === "en-US"
next to German number formatting, a mismatch fingerprinting scripts look
for, and the same page rendered differently across machines.
Follow Chrome's --lang rule: one configured tag drives navigator.language(s),
the Accept-Language header and ICU's default locale. --locale defaults to
en-US, so Intl is now en-US on every host instead of whatever LANG says.
--timezone sets the IANA zone Date and Intl use; absent, the host zone stays.
Both are applied by writing LC_ALL and TZ before V8 initializes ICU, which
reads them lazily. Platform.init is the first call in App.init, before any
thread exists, so setenv is safe there.
CDP Emulation.setUserAgentOverride.acceptLanguage, which Playwright sends
for its locale option, now overrides the header and navigator.languages
for the browser context's lifetime, mirroring the user agent override, and
applies even when the Mozilla user agent is refused.
Emulation.setLocaleOverride and setTimezoneOverride stay no-ops: changing
ICU's defaults at runtime needs new zig-v8-fork bindings.
Attr.cloneNode() copied _element, so the clone still claimed the original
owner. Element.setAttributeNode() then tried to remove it from that owner,
which never held it, and threw NotFoundError. Per the DOM spec a cloned Attr
has a null ownerElement.
While there, make setAttributeNode() throw InUseAttributeError when the Attr
belongs to another element instead of silently moving it, matching the spec
and both Chrome and Firefox.
Mozilla's Readability.js hits this on every page (_setNodeTag,
_simplifyNestedElements), as does any code that copies attributes with
el.setAttributeNode(attr.cloneNode()).
Karl Seguin