--strip-mode was silently ignored for --dump markdown, and fetch had no
way to scope a dump to one element or cap its size although the markdown
and html tools have both.
--selector dumps the first matching element in any dump mode, --max-bytes
caps html and markdown with the tools' truncation marker, and markdown
now honors strip ui (images; scripts, styles and hidden elements are
never rendered, so the other groups don't apply).
Add an optional per-host rate limit. This currently only applies to the top-
level navigation. This makes it simpler to implement and simpler to reason
about. The full load of a page is only ever delayed at its head, not
sporadically through the page loading.
The use-case where a RateLimiter is most useful is when the browser is crawling
multiple pages of the same site, and in that case, the top-level rate limit
still applies some degree of limit to any linked resources (e.g. a JS on a
different host).
`--http-nav-delay` is the delay, in milliseconds, to apply to top level
navigates per host. Currently defaults to 0 (disabled).
`--http-nav-burst` is the burst allowed per host. Defaults to 1 (has no impact
when `--http-nav-delay` is disabled).
--http-cache-entry-limit is currently implemented as a hard-limit via a check
on every put. For a large --http-cache-entry-limit, this can result in a non-
trivial delay on every http get.
This commit calls cache.maintenance() on Browser.deinit and on startup.
maintenance:
1 - purges stale entries
2 - enforces the limit
With concurrent requests, the number of entries can easily exceed the limit by
hundreds of entries (but this is temporary).
The implicit default was done (full quiescence), capped at --wait-ms.
Pages with constant background activity (ad timers, live sockets) never
go quiescent, so every default fetch of such a page silently rode the
full 5s cap; the timeout is non-fatal (Runner.waitForAll discards
WaitResult.timeout), so the default was effectively "wait 5s, then dump
whatever is there". load matches the agent/MCP default_nav_wait and its
rationale.
Measured on an ad-heavy news hub (ReleaseFast, interleaved runs):
default now ~3.2s vs 5.8-5.9s with done, with the load-time dump
containing slightly more of the extracted content, not less (the extra
seconds of ad-script execution mutate the page, they do not add to it).
A quiet page dumps byte-identical output under both.
Explicit --wait-until done|networkidle and --wait-selector/--wait-script
are unchanged; the fetch() doc comment now describes when error.Timeout
can actually be raised (selector/script phases only).
Adds a /metrics endpoint in serve mode. Can be disabled via --disable-metrics.
In fetch, dumps the metrics if --metrics is specified (defaults to false).
None of the metrics being collected are on a hot path, so they're just always
collected using atomic operations (i.e. no `if (!enabled) return;`). Because
the operations are cheap and infrequent enough not to matter.
The one place I want to add more metrics is in HttpClient (bytes, status,
counts, ...) but after the pending HttpClient-related PR is merged.
./lightpanda fetch --log-level fatal --metrics "https://lightpanda.io/"
```
build_info{version="1.0.0-dev.7837+70493ce35"} 1
cdp_connections_total 0
cdp_connection_limit_total 0
cdp_active_connections 0
cdp_commands_total 0
cdp_unknown_commands_total 0
js_heap_limits_total 0
script_errors_total 0
arena_hit_total{size="tiny"} 908
arena_hit_total{size="small"} 35
arena_hit_total{size="medium"} 0
arena_hit_total{size="large"} 27
arena_miss_total{size="tiny"} 193
arena_miss_total{size="small"} 31
arena_miss_total{size="medium"} 5
arena_miss_total{size="large"} 23
navigate_total{type="page"} 1
navigate_total{type="iframe"} 0
navigate_total{type="popup"} 0
js_heap_size_bytes_bucket{le="4194304"} 0
js_heap_size_bytes_bucket{le="8388608"} 0
js_heap_size_bytes_bucket{le="16777216"} 1
js_heap_size_bytes_bucket{le="33554432"} 1
js_heap_size_bytes_bucket{le="67108864"} 1
js_heap_size_bytes_bucket{le="134217728"} 1
js_heap_size_bytes_bucket{le="268435456"} 1
js_heap_size_bytes_bucket{le="536870912"} 1
js_heap_size_bytes_bucket{le="+Inf"} 1
js_heap_size_bytes_sum 11223040
js_heap_size_bytes_count 1
```
`lightpanda agent <script>` replays a saved script token-free (no LLM),
but running it under `agent` reads wrong. Add `run` as a dedicated,
script-only command: `lightpanda run <script>`.
`run` exposes only a required [SCRIPT] positional plus the common
options; its LLM-session flags (--task/--save/--list-models/REPL) are
absent, so they're rejected at parse time. `parseArgs` normalizes a
`run` invocation to an `.agent` command, reusing the existing token-free
replay path unchanged, so nothing downstream needs a `.run` case.
`agent <script>` keeps working. Telemetry records `run` as its own mode
code (R) via a new Config.command field, so its adoption is measurable
separately from agent script replay (AR).
Introduces an HTTP transport option to serve multipleagents from a
single process. Each connection is routed to its own isolated
browsing session using the `Mcp-Session-Id` header.
Also adds new session management tools (`session_new`, `session_list`,
`session_close`) and refactors the MCP server to support multiple V8
isolates with parking.
Adds a new command line argument `--watchdog-ms` which, when set, will terminate
any JS that appears to be hung.
When configured, a new thread is started. Workers heartbeat this thread to
signal activity (e.g. not stuck in a JS loop). However, workers also block for
their own reason (e.g. network polling), so they can signal the watchdog that
they are "entering a wait" and, when complete, that they are "existing a wait".
During such waits, the watchdog will not signal the isolate to terminate.
Obviously, it's important for workers to signal aliveness and whenever they plan
on doing a non-JS wait. So you could say we introduce safe points where the
watchdog (for that browser) is disabled. We could do the opposite: enable the
watchdog whenever we enter JS ("hey, I'm about to execute JS, monitor me). But
there are _a lot_ more place where this happens.
The main addition in this commit is that we hook into the Isolate's
AddNearHeapLimitCallback callback and try to force the isolate to shutdown
rather than letting v8 hit an OOM which would take down the entire process.
In support of this, we now support a `--v8-max-heap-mb` command line option to
set an explicit heap limit. As a simple way to test this feature, load a
relatively heavy JS page with `--v8-max-heap-mb 1`.
There's also a `--v8-flags-unsafe` which is a mechanism to pass arbitrary
flags to v8 via its `SetFlagsFromString`. The parameter is called `unsafe`
because some [of the many] configurable flags could conflict with how the
snapshot is built and result in crashes. The snapshot creator also gains a
`--v8-flags-unsafe` flag, so advance users COULD create their snapshot and
run lightpanda with the same set of flags.
Places the options in alphabetic order and applies the same indentation for
all commands (agent was formatted quite differently than the others).
While I would also like "more common" commands to be listed first, since they
can only be sorted one way, I find alphabetic to be the most generally useful
and the easier for us to maintain.
Feel free to reject this if you disagree.
max http connections 10 -> 40
max http connetions per host 4 -> 6
These are just the defaults and can still be adjusted by the command line
arguments. 6 appears to be both Chrome and FireFox's default per host (which
is probably the more important of the two settings).
The limits are really use-case specific. A use case that it multi-threading
different domains can benefit from a conservative max-host with a very large
max conn. A use case that is multi-threading the same host will need to decide
if it's safe to raise max-host.
Adds a new strip-mode "invisible" which consults the StyleManager to remove
hidden elements. However, unlike the general isHidden check in StyleManager
which is spec-compliant and will automatically hide a number of tags, e.g.
"meta", this check is explicitly against explicit styles. This seems more
practical to me, while it might be useful to remove some tags by default, e.g.
template, some tags like head, title and meta are too important to strip out.
`--log-filter-scopes` previously took a comma-separated list of scopes to
suppress. Extend it with explicit include/exclude directives applied
left-to-right:
-X filter out scope X
X alias for -X (backward compatible)
+X filter in scope X
all target every scope
This makes "filter everything except one scope" expressible, e.g.
`-all,+cdp` suppresses all logs except `cdp`. Bare scope names keep their
old meaning, so existing invocations like `http,unknown_prop` are
unaffected.
Internally, the suppressed-scope list is replaced by a resolved per-scope
`scope_enabled` boolean array built via `log.resolveFilterScopes()`, so the
hot-path `enabled()` check is a single array index. Filtering remains
Debug-only, as before. `testing.LogFilter` is adapted to the new
representation; all existing call sites are unchanged.
Adds a unit test for resolveFilterScopes (default, backward-compat,
-all,+cdp, ordering) and updates the help text.
The agent already supports these providers (via zenai), but the agent
command's help text still listed only anthropic/openai/gemini/huggingface/
ollama. Bring --help in sync:
- Add vercel, mistral, and llama_cpp to the --provider allowed values
- Add AI_GATEWAY_API_KEY and MISTRAL_API_KEY to the auto-detect and
env-key lists
- Document llama.cpp as a keyless local server (base http://localhost:8080/v1)
- Note that a provider can set its own effort default (Mistral -> none)
This realigns --help with the docs at lightpanda.io/docs/usage/agent.
Move away from hard-coded 512KB (WS) and 4K (http) limits. Introduces two new
serve-specific command line arguments:
--cdp-max-message-size <INT>
Maximum allowed incoming websocket message size.
Defaults to 1048576 (1MB)
--cdp-max-http-message-size <INT>
Maximum allowed HTTP request size
Defaults to 4096 (maximum allowed: 16383)
--cdp-max-message-size has been bumped from 512KB to 1MB default.
Meant to provide a more robust solution than https://github.com/lightpanda-io/browser/pull/2717
Adrià Arrufat