Commit Graph

22 Commits

Author SHA1 Message Date
Muki Kiboigo
572267802d fire RobotsBlocked in nextTickRun 2026-06-05 07:25:32 -07:00
Karl Seguin
e333e458b1 Merge pull request #2597 from lightpanda-io/remove_unused_imports
remove unused imports
2026-06-02 06:24:30 +08:00
Karl Seguin
37a846d91d remove unused imports 2026-06-01 22:42:39 +08:00
Muki Kiboigo
2f1362ac19 remove missing queue panic on RobotsLayer 2026-06-01 06:57:14 -07:00
Karl Seguin
1cdd2bb324 Cleanup Transfer flag
Replaces 4 boolean flags with a state. Makes it easier to figure out what the
state of the transfer is, and removes the possibility of inconsistent flags
.e.g queued + loop_owned.

loop_owned -> state != .created
_queued -> state == .queued
_perform -> state == .completing
aborted -> state == .aborted
2026-05-20 20:37:21 +08:00
Karl Seguin
a5162bea8f Cleanup HttpClient.Transfer
This is just moving fields around. The end result is that there's a
`transfer.req` and a `transfer.res`.

On the Request side, we use to have a nested `params: RequestParam` resulting
in a lot of `transfer.req.params.url`. This is now `transfer.req.url`. On the
Response side, we had the exact opposite: response fields splattered directly
in the transfer, `transfer.response_header`. This is now `transfer.res.header`.

There is now an HttpClient.Response, which is the actual final response (which
could be for a transfer or something else, e.g the cache). And an
HttpClient.Transfer.Response which captures the inflight response data (and is
one of the polymorphic variants of the HttpClient.Response). Probably still not
ideal, but I'm not sure how to make it cleaner, and even if this is just an
intermediary step, I consider it an small win.
2026-05-15 12:55:47 +08:00
Muki Kiboigo
4a45b4d866 fix crash on robots.txt request fufilled immediately 2026-05-12 21:50:05 -07:00
Karl Seguin
5e0976bbd6 fix use-after-free on robotslayer shutdown 2026-05-12 19:26:24 +08:00
Karl Seguin
82a4fc752b HttpClient Improvements
1 - Track owner of a request (for simpler / more accurate abort (TBD))

2 - Create Transfer upfront, make everything work on Transfer (not Request)
    This helps remove ambiguity about cleanup and simplifies layers. For example
    Robots request is just another normal request, not a special case. This gives
    everything a stable address (the *Transfer which can be looked up by id)
2026-05-12 19:26:24 +08:00
Adrià Arrufat
eab9ae0243 RobotsLayer: use managed ArrayList 2026-05-04 08:01:22 +02:00
Muki Kiboigo
1ab445843c better arena management in Robots Layer and Context 2026-04-28 07:01:43 -07:00
Muki Kiboigo
3fe774fbfb pass error all the way up to Layer chain to clean 2026-04-28 07:01:42 -07:00
Muki Kiboigo
c719a522b8 use lightpanda module log in layers 2026-04-28 07:01:42 -07:00
Muki Kiboigo
152a792c18 use Request Arena in RobotsLayer 2026-04-28 07:01:41 -07:00
Muki Kiboigo
bb9e238f6c Requests now use arenas from the arena pool 2026-04-28 07:01:41 -07:00
Muki Kiboigo
175c2cc288 ensure robots params have arena and request id 2026-04-28 07:01:41 -07:00
Muki Kiboigo
ca08f0c56d remove blocking from RequestParams 2026-04-28 07:01:40 -07:00
Muki Kiboigo
9c826159a0 crude InterceptionLayer 2026-04-28 07:01:40 -07:00
Muki Kiboigo
6d41ea6fd0 move arena up to Request instead of Transfer 2026-04-28 07:01:39 -07:00
Muki Kiboigo
e988e49136 remove Context and thread *Client 2026-04-28 07:01:39 -07:00
Muki Kiboigo
46d0b34c54 add RequestParams and SyncRequest 2026-04-28 07:01:39 -07:00
Muki Kiboigo
5dd15aa2cf use layers for Cache, Robots and WebBotAuth 2026-04-28 07:01:39 -07:00