render: add "shell" option to --strip-mode

Some additional API changes:
- Add strip-mode support to pdf/png generation.
- Add LP.dump which provides greater content gathering capability to CDP,
  exposing most `fetch` dump-related parameters (e.g. format, strip, selector,
  ...)

The new "--strip-mode shell" is designed to try to remove non-content elements
such as the header and footer. The end goal is to use readibility.js test cases
as a baseline, but this isn't a port of readibility.js.

This is just the basic implementation of this, e.g removing a few key tags, e.g.
<header>, <footer> and considering some specific roles.

Even if --strip-mode shell is used, we might decide to stick with a whole dump:
it's better to strip not enough than to strip too much. This currently works by
measuring the ratio of non-link text of the stripped vs unstripped page.
This commit is contained in:
Karl Seguin committed 2026-09-11 06:35:26 +08:00
1 parent c21462cc38
commit cdceca37c3
11 files changed
+447 -27

No files matched your search

+5 -6
View File
@@ -118,15 +118,14 @@
\\ Defaults to false.
\\ --strip-mode <STRIP>
\\ Tag group to remove from dump. Can be passed multiple times.
\\ In markdown only 'ui' changes the output: scripts, styles and
\\ hidden elements are never rendered.
\\ Defaults to no-strip.
\\ Defaults to stripping nothing.
\\ Allowed values:
\\ js Script and link[as=script, rel=preload].
\\ ui Includes img, picture, video, CSS and SVG.
\\ css Includes style and link[rel=stylesheet].
\\ js Script and link[as=script, rel=preload].
\\ invisible Best-effort (e.g. display:none) hidden elements
\\ full Strip everything.
\\ shell Heuristic to attempt to remove headers, footer,
\\ and other non-content decoration.
\\ ui Includes img, picture, video, CSS and SVG.
\\ --terminate-ms <INT>
\\ Hard deadline in milliseconds. After this time elapses, JavaScript
\\ execution is forcibly terminated (e.g. for pages with endless scripts).