mirror of
https://github.com/pnpm/pnpm.git
synced 2026-06-28 09:55:39 -04:00
c199198e94
Closes #12234. This PR has two parts. The headline turned out to be the **benchmark**, not the feature. ## Part 1 — `/v1/resolve` streaming (the #12234 feature) `POST /v1/resolve` now streams **NDJSON** instead of buffering the whole lockfile: one `package` frame per resolved tarball as the server's tree walk yields it, then a terminal `done` (full lockfile + stats), `error`, or `violations` frame. The client fetches each tarball as its frame arrives, overlapping the server's resolution — matching the native `PrefetchingResolver` shape. - Server: new `ResolutionObserver`/`ObservingResolver` in `package-manager`, threaded through `Install`; `handle_resolve` runs the resolve in a detached task whose observer pushes frames into the response channel; `application/x-ndjson` is excluded from the gzip layer so frames flush incrementally. - Client: `resolve_streaming(opts, on_package)`; `install_via_pnpr` drives a new `TarballPrefetcher` that warms the shared mem cache as frames arrive. - Breaking change within protocol v1 (no version bump — experimental pnpr allows it). **Honest caveat:** streaming only helps when the *server's* resolve is slow (cold/distant server). Against a warm server it's inert — see the results below (`pnpr@HEAD` ≈ `pnpr@main` everywhere). Whether to keep this commit or defer it is an open question; the benchmark fixes below stand on their own. ## Part 2 — make the integrated benchmark actually measure pnpr While validating Part 1, the benchmark turned out not to be exercising pnpr **at all**, plus it was serving the registry far faster than reality. Fixes: - **`pnpr@<rev>` targets never routed through pnpr.** `.pnpr-env` exported a bare `PNPR_SERVER`, but pacquet reads config env vars only under the `PNPM_CONFIG_*` prefix, so `config.pnpr_server` was always `None` and every pnpr row was a silent duplicate of its direct row. Fixed the env var name; added a post-run guard that fails the benchmark if a pnpr target's `pnpr-storage` is empty (proof it never served a resolve). - **Emulate a real registry link for every client.** The latency proxy modeled RTT but not throughput, and fronted only direct targets. Generalized it to a `LinkProfile` (one-way delay + per-direction bandwidth cap), added `--registry-bandwidth-mbps`, and routed *all* registry traffic through it (direct installs, the pnpr server's resolve, the pnpr client's fetches) so the registry-mock is uniformly as remote as real npm. CI runs it at 50 ms + 200 Mbit/s (≈ the measured public-npm peak). - **Make "cold cache" cold for resolution.** Forced `cacheDir` bench-local and wipe it in cold-cache scenarios, so a direct install actually pays the packument-fetch waterfall (previously the global metadata mirror survived every wipe). - **New scenario** `fresh-install.cold-cache.hot-store` that isolates resolution (cold metadata, hot store → no download to mask it). ## Results (Linux CI, after the fixes) | Scenario | pacquet@HEAD | pnpr@HEAD | pnpr speedup | |---|---:|---:|---:| | fresh-install · cold cache · **hot store** | 5.06 s | 0.69 s | **7.4×** | | fresh-install · cold cache · cold store | 5.36 s | 2.03 s | **2.65×** | | fresh-install · hot cache · hot store | 1.46 s | 0.69 s | **2.1×** | | fresh-restore (frozen) · cold cache · cold store | 10.08 s | 5.13 s | **2.0×** | | fresh-restore (frozen) · hot cache · hot store | 0.71 s | 0.80 s | 0.89× (slower) | pnpr offloads the client's resolution (and, on the frozen path, lockfile verification) to its warm server: 2–7× faster wherever the client would otherwise pay that cost. The lone regression is the fully-warm frozen install, where there's nothing to offload and pnpr's one round trip is pure overhead. `pnpr@HEAD` vs `pnpr@main` is flat throughout — i.e. the streaming commit (Part 1) adds ~nothing against a warm server, while the base pnpr win (offload to a warm server) is large.