Commit Graph
2505 Commits
Author SHA1 Message Date
Karl Seguin 9bd8b2fc43 fix wpt runner 2025-08-13 19:39:49 +08:00
Karl Seguin 5a3d5f5512 improve elapsed display for larger numbers 2025-08-13 18:17:59 +08:00
Karl Seguin ca9e850ac7 Create Client.Transfer earlier.
On client.request(req) we now immediately wrap the request into a Transfer. This
results in less copying of the Request object. It also makes the transfer.uri
available, so CDP no longer needs to std.Uri(request.url) anymore.

The main advantage is that it's easier to manage resources. There was a use-
after free before due to the sensitive nature of the tranfer's lifetime. There
were also corner cases where some resources might not be freed. This is
hopefully fixed with the lifetime of Transfer being extended.
2025-08-13 18:05:00 +08:00
Karl Seguin 2dc09c799f Merge pull request #930 from lightpanda-io/request_interception
request interception
2025-08-13 14:44:26 +08:00
sjorsdonkers a49154acf4 http_request_fail 2025-08-12 15:20:48 +02:00
sjorsdonkers 77eee7f087 Cookies 2025-08-12 14:40:23 +02:00
sjorsdonkers 03694b54f0 3# This is a combination of 3 commits.
intercept continue and abort

feedback

First version of headers, no cookies yet
2025-08-12 13:49:20 +02:00
Karl Seguin bed320204d Merge pull request #939 from lightpanda-io/raw-done
finalize document loading with non-HTML pages
2025-08-12 19:09:31 +08:00
Pierre Tachoire 971524fa3b finalize document loading with non-HTML pages
Avoid infinite the loop of loading non-HTML documents with CDP.
2025-08-12 12:55:44 +02:00
Karl Seguin ea0bbaf332 Revert "Treat pending requests as active"
This reverts commit 19c908035b.
2025-08-12 11:27:28 +08:00
Karl Seguin 19c908035b Treat pending requests as active
This ensures that page.wait won't unblock too early. As-is, this isn't an issue
since active can only be 0 if there are no active OR pending requests. However,
with request interception (https://github.com/lightpanda-io/browser/pull/930)
it's possible to have no active requests and no pending requests - from the
http client's point of view - but still have pending-on-intercept requests.

An alternative to this would be to undo these changes, and instead change
Page.wait to be intercept-aware. That is, Page.wait would continue to block on
http activity and scheduled tasks, as well as intercepted requests. However,
since the Page doesn't know anything about CDP right now, and it does know
about the http client, maybe doing this in the client is fine.
2025-08-12 11:13:19 +08:00
Muki Kiboigo 05192b6850 update flake 2025-08-11 12:09:22 -07:00
Karl Seguin 079ce5e9de whitelist application/ld+json 2025-08-11 21:38:36 +08:00
Karl Seguin ff742c0169 don't allow concurrent blocking calls 2025-08-11 21:38:36 +08:00
Karl Seguin 332e264437 remove unimportant todos 2025-08-11 21:38:34 +08:00
Karl Seguin 3554634c1c cleanup optional request headers 2025-08-11 21:37:03 +08:00
Karl Seguin c96fb3c2f2 support CDP proxy override 2025-08-11 21:37:03 +08:00
Karl Seguin 1e612e4166 Add command line options to control HTTP client
http_timeout_ms
http_connect_timeout_ms
http_max_host_open
http_max_concurrent
2025-08-11 21:37:03 +08:00
Karl Seguin 06984ace21 fix overflow and debug units 2025-08-11 21:37:03 +08:00
Karl Seguin cabd4fa718 re-enable datauris 2025-08-11 21:37:03 +08:00
Karl Seguin ddb549cb45 cookie support 2025-08-11 21:37:02 +08:00
Karl Seguin c7484c69c0 Increase max concurrent request to 10
Improve wait analysis dump.

De-prioritize secondary schedules.

Don't log warning for application/json scripts

Change pretty log timer to display time from start.
2025-08-11 21:37:02 +08:00
Karl Seguin 9876d79680 Add Accept-Encoding
This is necessary because of CloudFront which will send gzip content even if
we don't ask for it.

Properly handle scripts that are both async and defer.

Add a helper to print state of page wait. This can be helpful in identifying
what's causing the page to hang on page.wait.
2025-08-11 21:37:02 +08:00
Karl Seguin 32566ccc80 Set window location on load
Set SUPPRESS_CONNECT_HEADERS option.
2025-08-11 21:37:02 +08:00
Karl Seguin 7f9e309ae8 Shutdown clean async scripts
Set parent current script
2025-08-11 21:37:02 +08:00
Karl Seguin 7831aabe5a connect proxy 2025-08-11 21:37:02 +08:00
Karl Seguin 74b40b97ec fix ScriptManager wrong order execution 2025-08-11 21:37:02 +08:00
Karl Seguin f45726d61f ScriptManager & HttpClient support for JS modules
Improve cleanup/shutdown (*cough* memory leaks *cough*)
2025-08-11 21:37:01 +08:00
Karl Seguin 3c0d027306 dynamic script support 2025-08-11 21:37:01 +08:00
Karl Seguin dc83765808 fix build 2025-08-11 21:37:01 +08:00
Karl Seguin 4244b572d1 Improve page.wait
Allow page.wait to transition page mode.

Optimize initial page load. No point running scheduler until the initial
page is loaded.

Support ISO-8859-1 charset
2025-08-11 21:37:01 +08:00
Karl Seguin 77475ca5e4 Re-enable --insecure_disable_tls_host_verification
Better error logs on http callback error

Fix wait timing
2025-08-11 21:37:01 +08:00
Karl Seguin 3555680335 Working navigation events (clicks, form submission) 2025-08-11 21:37:01 +08:00
Karl Seguin f65a39a3e3 Re-enable telemetry
Start work on supporting navigation events (clicks, form submission).
2025-08-11 21:37:00 +08:00
Karl Seguin 94e8964f69 add custom scheduler 2025-08-11 21:37:00 +08:00
Karl Seguin 254d22e2cc don't poll libcurl if we have no running transfers 2025-08-11 21:37:00 +08:00
Karl Seguin 54ab1326e5 Switch XHR to new http client
get puppeteer/cdp.js working again

make test are all passing
2025-08-11 21:37:00 +08:00
Karl Seguin b0fe5d60ab Initial work on integrating libcurl and making all http nonblocking 2025-08-11 21:36:56 +08:00
Karl Seguin 6a2dd1111c Merge pull request #928 from lightpanda-io/lit_compat
lit compatibility
2025-08-11 08:30:34 +08:00
Karl Seguin f5da89b50b lit compatibility
Aims to improve compatibility for the lit framework (e.g. what Reddit is using).

1 - Adds support for adoptedStyleSheets to the Document and ShadowRoot
2 - Adds mock support for replace and replaceSync to the CSSStyleSheet
3 - Optionally include shadowroot in dump
4 - Special-case setting innerHTML on a TemplateElement
2025-08-09 07:43:27 +08:00
Karl Seguin bede244598 Merge pull request #934 from lightpanda-io/with-base
add a --with_base option to fetch
2025-08-09 07:37:32 +08:00
Karl Seguin 4df48c9695 Merge pull request #935 from lightpanda-io/mouse-event-log
use internal logger instead of std.log
2025-08-09 07:36:35 +08:00
Karl Seguin 05ad77ffbe Merge pull request #936 from lightpanda-io/runtime-empty-array
Fix crashes with empty array
2025-08-09 07:36:09 +08:00
Pierre Tachoire dc23a74e7b add <base> in the DOM tree 2025-08-08 18:34:14 +02:00
Pierre Tachoire f463cb16da runtime: handle empty array parameter 2025-08-08 17:50:18 +02:00
Pierre Tachoire b785884cd8 runtime: fix returning an empty array crash 2025-08-08 17:26:39 +02:00
Pierre Tachoire f09caec09a use internal logger instead of std.log 2025-08-08 16:21:23 +02:00
Pierre TachoireandKarl Seguin 5e30a3997e typo fix
Co-authored-by: Karl Seguin <karlseguin@users.noreply.github.com>
2025-08-08 16:17:52 +02:00
Karl Seguin 8552a5797c Merge pull request #933 from lightpanda-io/document_fragment_get_element_by_id
Add DocumentFragment getElementById
2025-08-08 22:06:02 +08:00
Karl Seguin a0d528981e Merge pull request #932 from lightpanda-io/libdom_element_attributes
Updates libdom
2025-08-08 22:05:42 +08:00