Commit Graph

8472 Commits

Author SHA1 Message Date
Karl Seguin
62ecfb48d8 Merge pull request #2880 from lightpanda-io/script_replay_nav_fixes
script: fix two replay crashes on tool-triggered navigation and $LP_* goto URLs
2026-07-04 10:27:24 +08:00
Karl Seguin
a082dc2b07 update v8 2026-07-04 07:29:40 +08:00
Adrià Arrufat
cfd61ca69e tools: resolve $LP_* placeholders in startGoto
The script runtime's async goto path calls startGoto directly,
bypassing the substituteStringArgs pass that call() applies to every
other tool, so $LP_* placeholders in goto URLs (a base URL, a username
in the path) stayed literal and the navigation failed on replay.
2026-07-03 23:52:40 +02:00
Adrià Arrufat
ae2359c1af script: fix segfault when a tool call triggers navigation
A press/click that submits a form or follows a link commits a
replacement Page inside the tool call itself (finalizeAction ->
awaitQueuedNavigation), freeing the Frame that
Session._tool_frame_override still pointed at; finalizeAction then
dereferenced it via requireFrame -> currentFrame. Any script whose
press/click navigated crashed on replay.

Store a frame id instead of a pointer and resolve it at every
currentFrame call. The replacement page keeps the frame id (see
commitPendingPage), so the same handle stays routable across the swap.
2026-07-03 23:52:29 +02:00
Adrià Arrufat
ff7545a8c3 agent: add Google Vertex AI support
Supports both express mode (via VERTEX_API_KEY) and project mode
(via GOOGLE_CLOUD_PROJECT and gcloud auth). Handles automatic
OAuth token retrieval and refresh.
2026-07-03 20:23:51 +02:00
Karl Seguin
f6fdec3300 Merge pull request #2876 from lightpanda-io/content-type-makrdown
handle text/makrdown as plain text
2026-07-03 22:46:48 +08:00
Karl Seguin
213126e08b Merge pull request #2875 from lightpanda-io/rm-100-continue
Disable Expect: 100-continue with curl
2026-07-03 22:24:30 +08:00
Karl Seguin
933dc61bf6 Merge pull request #2869 from lightpanda-io/closest_cached_selector
perf: used cachedParse for closest and waitForSelector
2026-07-03 21:18:23 +08:00
Pierre Tachoire
c74f18acc2 handle text/makrdown as plain text 2026-07-03 13:25:31 +02:00
Karl Seguin
3f16812797 perf: Remove Seletor cache from Runner's waitForSelector
From feedback, using cachedParse in waitForSelector outside the loop risks
having the cached selector freed. For this reason, I'm reverting this part of
the previous commit. waitForSelector already pre-parsed the selector and I think
this is the better choice.

Pre-parsed means that we avoid the cache lookup in the [potentially] tight wait
loop. The advantage of hitting the cache inside the loop is only if
waitForSelector is called multiple times, which doesn't typically happen.
2026-07-03 19:14:41 +08:00
Adrià Arrufat
2170b34ea9 ci: agent-regression is manual-only, drop the nightly cron
Claude-Session: https://claude.ai/code/session_01SG5kc3gGurhT7N6f63cjMq
2026-07-03 12:01:05 +02:00
Pierre Tachoire
e37e13b364 Disable Expect: 100-continue with curl
Curl sends by default an `Expect: 100-continue` header with some `POST`
requests. But it can slow down the whole request if the server doesn't
handle it correclty.
Forcing an `Expect:` header disable the curl's default behavior.

see https://everything.curl.dev/http/post/expect100.html or
https://gms.tf/when-curl-sends-100-continue.html
2026-07-03 11:42:35 +02:00
Karl Seguin
aa3b30754d Merge pull request #2873 from lightpanda-io/tweak_parser_node_insert
refactor: Cleanup parser/non-parser Frame._insertNodeRelative
2026-07-03 17:00:22 +08:00
Karl Seguin
917a6b5176 Merge pull request #2871 from lightpanda-io/reclaim_large_ws_buffer
mem: reclaim large WS buffer
2026-07-03 17:00:00 +08:00
Karl Seguin
8c4219c2b5 Merge pull request #2870 from lightpanda-io/default_max_connections
perf: Increase default max connections
2026-07-03 16:59:42 +08:00
Adrià Arrufat
f7bcdace77 ci: drop temporary push trigger, cross-repo wiring validated green
Claude-Session: https://claude.ai/code/session_01SG5kc3gGurhT7N6f63cjMq
2026-07-03 10:13:43 +02:00
Adrià Arrufat
bd95d9a610 test: consume agent regression suite from lightpanda-io/demo
The suite moved to the demo repo (machinery in agent/, targets the
existing public/ sites — lightpanda-io/demo#206). Delete the in-repo
copy and point the agent-deterministic job, agent-regression.yml, and
make test-agent at the demo checkout, passing the binary via LPD_PATH
(the same variable wptrunner uses) — same consumption model as
runner/integration/wpt.

Includes a temporary push trigger on agent-regression.yml to validate
the cross-repo wiring pre-merge; dropped before merging.
2026-07-03 09:59:29 +02:00
Karl Seguin
bf4d4f7ede refactor: Cleanup parser/non-parser Frame._insertNodeRelative
_insertNodeRelative weaved the from_parser comptime through a different
conditions. It made reading the code difficult, but also optimizing it. This
commit extracts the common code (notifyChildInserted) and handles from_parser
in one cohesive chunk.

One benefit of this is that isConnected() is no longer called when from_parser
is true.
2026-07-03 13:24:30 +08:00
Karl Seguin
8a20823e78 Merge pull request #2868 from SyntaxSawdust/fix/charset-unicode-mojibake
Fix charset=unicode HTML prescan fallback
2026-07-03 11:41:32 +08:00
Karl Seguin
5a182d96f4 webapi: Add 6 more interfaces to Worker
CustomEvent, FileList, MessagePort, MessageChannel, Notification, ProgressEvent
2026-07-03 11:12:02 +08:00
Karl Seguin
23ab2df58e mem: reclaim large WS buffer
We'll grow the WS read buffer up to cdp-max-message-size (1MB default), but
never reclaim that space. A lot of drivers send large message upfront and then
settle into sending smaller ones.

This commit shrinks the buffer to 256KB (or cdp-max-message-size, whichever is
smaller) after 8 consecutive small message (tracking consecutive messages to
prevent a spat of allocation -> shink -> allocation -> shrink ->...)
2026-07-03 10:46:39 +08:00
Karl Seguin
f1e49aae6e perf: Increase default max connections
max http connections 10 -> 40
max http connetions per host 4 -> 6

These are just the defaults and can still be adjusted by the command line
arguments. 6 appears to be both Chrome and FireFox's default per host (which
is probably the more important of the two settings).

The limits are really use-case specific. A use case that it multi-threading
different domains can benefit from a conservative max-host with a very large
max conn. A use case that is multi-threading the same host will need to decide
if it's safe to raise max-host.
2026-07-03 09:43:13 +08:00
Karl Seguin
adf474774e perf: used cachedParse for closest and waitForSelector
closest was already hitting the cache, but it was doing it inside the loop.
Moving it outside the loop avoids the cache lookup.

Inversely, Runner.waitForSelector was parsing outside the loop, but it wasn't
caching. Repeated calls to waitForSelector with the same selector would not
leverage the cache.
2026-07-03 09:36:21 +08:00
Karl Seguin
c5a5991b1c Merge pull request #2864 from lightpanda-io/cache-computed-style
cache the computed-style object per element
2026-07-03 08:28:22 +08:00
Karl Seguin
c4d6c84cdb Merge pull request #2865 from lightpanda-io/network-idle-per-frame
frame: emit networkIdle lifecycle events for child frames
2026-07-03 08:19:53 +08:00
Karl Seguin
6708d5b961 Merge pull request #2862 from lightpanda-io/global_enumerable
webapi: Make all interfaces non-enumerable
2026-07-03 07:06:54 +08:00
Dustin Persek
f8934f575b Fix charset unicode HTML prescan fallback 2026-07-02 17:43:04 -04:00
Pierre Tachoire
ca0579413e Merge pull request #2837 from lightpanda-io/cache-simple-heuristic
Cache Simple Heuristic
2026-07-02 17:42:19 +02:00
Pierre Tachoire
656e2ee495 frame: emit networkIdle lifecycle events for child frames
Puppeteer's networkidle0 requires the networkIdle/networkAlmostIdle
lifecycle events on every frame that started loading, like Chrome
emits them — not just the root frame. Run the idle-notification
checks recursively over the frame tree in Runner._tick.

Fixes goto({waitUntil: 'networkidle0'}) timing out on pages with
iframes (e.g. reddit.com post pages).
2026-07-02 17:06:12 +02:00
Karl Seguin
53767d2dad Merge pull request #2852 from lightpanda-io/local_arena
mem: add local_arena
2026-07-02 22:46:11 +08:00
Pierre Tachoire
7e353927b9 cache the computed-style object per element
window.getComputedStyle allocated a CSSStyleProperties +
CSSStyleDeclaration pair per call; pages polling it grew the page
arena without bound. The computed variant is a stateless lazy view,
so hand out one per element from a frame-level map. This also matches
Chrome, where repeated calls return the identical object.
Pseudo-element requests keep the fresh-object path.
2026-07-02 16:33:58 +02:00
Karl Seguin
25d8b7d5c7 mem: add local_arena
The `call_arena` is currently our shortest-lived arena. Its promise is that it
will be valid for at least 1 v8 -> zig -> v8 function call, which makes it ideal
for getting values from v8 into zig, temporary work, and getting values from
zig to v8.

VBut `call_arena`'s lifetime is actually much longer. It's only reset when the
call_depth reaches 0. And the reason for that are Zig functions that invoke
v8 callbacks. If you just do it when a function ends, then if you allocate in
ZigA and then call CallbackA which calls ZigB, then when ZigB ends, your ZigA
allocation is cleared. This is something we could solve by reaching into
Zig's ArenaAllocator to capture a position to rollback from.

So, `call_arena` can end up living relatively long and accumulating quite a bit
of memory. But most calls _don't_ invoke callbacks. Hence, `local_arena` IS
reset at the end of every function.

Using `local_arena` _is_ dangerous, mostly for the case where some of our APIs
receive a *Frame (or *Execution) and thus might use a `local_arena` because they
know they aren't invoking a JS callback. BUT, those APIs might not know that
they're also invoked by some other Zig code that _could_ be invoking JS.

Dangerous? Sure. But, github has code that looks like:

```js
function onDelegatedClick(e) {
  for (const el of document.querySelectorAll('[data-action]')) {
    if (el.matches('.menu > .item:not(.disabled) a[href]')) {
      handle(el);
    }
    el.closest('.panel');
  }
}
```

Anything allocated in the `call_arena` will only be freed when the caller of
`onDelegatedClick` ends. The `querySelectorAll` returns hundreds of elements
and thus builds hundreds of parsed CSS and other scrap (x2 for `matches` and
closest`). `call_arena` peaks at 15MB. With the local_arena? 1MB. If 10x more
elements were returned, the peak would be 10x higher. With local_arena, it
stays 1MB.
2026-07-02 21:06:51 +08:00
Karl Seguin
3fbe460b04 Merge pull request #2861 from lightpanda-io/MessageEvent_getSource
crash, worker: Fix crash when MessageEvent.source is called from worker
2026-07-02 21:04:45 +08:00
Karl Seguin
0db0b89def Merge pull request #2858 from lightpanda-io/structuredClone
webapi: structuredClone for host (aka Zig) objects
2026-07-02 21:00:57 +08:00
Karl Seguin
69547db8ba webapi: Make all interfaces non-enumerable
When I added this, I was going through the list in /dom/interface-objects.html
thinking that was exhaustive. But no, no interfaces should be enumerable and
various other WPT tests (usually the idlharness ones) assert that for their
respective types.

Make it _always_ non enumerable means we no longer need Meta.enumerable to
be declared true/false (it's always false).
2026-07-02 20:54:33 +08:00
Karl Seguin
8f8df00b0f crash, worker: Fix crash when MessageEvent.source is called from worker
The source for a worker is always null. The getter cannot receive a *Frame since
it can be called from a Worker's context.
2026-07-02 19:28:09 +08:00
Pierre Tachoire
5fcfb4e344 Merge pull request #2859 from lightpanda-io/crypto-random-leak
use getrandom syscall for std.crypto.random
2026-07-02 12:03:30 +02:00
Adrià Arrufat
f9c3e4bc2a ci: drop temporary push trigger, suite validated green in CI 2026-07-02 12:00:19 +02:00
Pierre Tachoire
37a7f034c8 use writerStreaming in log 2026-07-02 11:23:45 +02:00
Pierre Tachoire
e56a4f6259 use getrandom syscall for std.crypto.random
std.crypto.random's default backend mmaps a thread-local 528-byte state
page on first use and never unmaps it — there is no thread-exit hook.
With one detached thread per CDP connection (Server.handleConnection),
that leaks one resident page per connection (uuidv4 in
Page.getOrCreateOrigin touches it), ~4KB/conn of unbounded RSS growth.
Route every std.crypto.random call to the getrandom syscall instead.
.crypto_always_getrandom = true,
2026-07-02 11:19:53 +02:00
Adrià Arrufat
e219581791 test: surface agent stderr on live-layer failures 2026-07-02 10:33:28 +02:00
Adrià Arrufat
86ef37e12b ci: temporary push trigger to validate agent-regression pre-merge 2026-07-02 10:24:59 +02:00
Adrià Arrufat
f191410773 test: add agent regression suite
Two layers driven by test/agent/run.sh:

- deterministic: replay a golden PandaScript against frozen local HN
  fixtures and diff the returned JSON exactly against a golden file.
  No API key, no network; gates every PR via e2e-test.yml.

- live (GOOGLE_API_KEY): closed-form Q&A over a local fixture page
  (substring match), and a live Hacker News scrape saved with --save,
  replayed token-free and validated against a jq shape invariant.
  Runs nightly and on demand via agent-regression.yml.

The live output contract lives in cases/hn-live.task (prompt) +
schemas/hn-live.jq (invariant). Model pinned to gemini-3.5-flash in
run.sh; per-task timeouts and a $usage token ceiling guard runaway
loops. make test-agent for local runs.
2026-07-02 09:40:22 +02:00
Halil Durak
60aacfa2b0 Merge pull request #2851 from lightpanda-io/nikneym/network-x509-store
networking: prefer `X509_STORE` instead of `ca_blob`
2026-07-02 09:10:15 +03:00
Halil Durak
c86dad5430 libcrypto: remove unused utilities 2026-07-02 08:46:22 +03:00
Halil Durak
5e567adf1b ci: run serve command with TLS host verification disabled 2026-07-02 08:46:21 +03:00
Halil Durak
fd0bff3af2 http: remove dead code 2026-07-02 08:46:21 +03:00
Halil Durak
820f5684cb networking: replace ca_blob with X509_STORE
Idea here is to skip re-parsing that happen for each connection; we already use BoringSSL, so we can take more advantage of it by directly mutating cert store of `SSL_CTX`.
2026-07-02 08:46:21 +03:00
Halil Durak
00123d5102 libcurl: more bindings
* Make curl_easy_setopt aware of SSL_CTX_FUNCTION and SSL_CTX_DATA,
* Add CURLE_* errors.
2026-07-02 08:46:21 +03:00
Halil Durak
6e75a0caa9 libcrypto: bind couple X509, X509_STORE and SSL_CTX helpers 2026-07-02 08:46:20 +03:00