mirror of
https://github.com/pnpm/pnpm.git
synced 2026-07-02 11:55:17 -04:00
c199198e94
Closes #12234. This PR has two parts. The headline turned out to be the **benchmark**, not the feature. ## Part 1 — `/v1/resolve` streaming (the #12234 feature) `POST /v1/resolve` now streams **NDJSON** instead of buffering the whole lockfile: one `package` frame per resolved tarball as the server's tree walk yields it, then a terminal `done` (full lockfile + stats), `error`, or `violations` frame. The client fetches each tarball as its frame arrives, overlapping the server's resolution — matching the native `PrefetchingResolver` shape. - Server: new `ResolutionObserver`/`ObservingResolver` in `package-manager`, threaded through `Install`; `handle_resolve` runs the resolve in a detached task whose observer pushes frames into the response channel; `application/x-ndjson` is excluded from the gzip layer so frames flush incrementally. - Client: `resolve_streaming(opts, on_package)`; `install_via_pnpr` drives a new `TarballPrefetcher` that warms the shared mem cache as frames arrive. - Breaking change within protocol v1 (no version bump — experimental pnpr allows it). **Honest caveat:** streaming only helps when the *server's* resolve is slow (cold/distant server). Against a warm server it's inert — see the results below (`pnpr@HEAD` ≈ `pnpr@main` everywhere). Whether to keep this commit or defer it is an open question; the benchmark fixes below stand on their own. ## Part 2 — make the integrated benchmark actually measure pnpr While validating Part 1, the benchmark turned out not to be exercising pnpr **at all**, plus it was serving the registry far faster than reality. Fixes: - **`pnpr@<rev>` targets never routed through pnpr.** `.pnpr-env` exported a bare `PNPR_SERVER`, but pacquet reads config env vars only under the `PNPM_CONFIG_*` prefix, so `config.pnpr_server` was always `None` and every pnpr row was a silent duplicate of its direct row. Fixed the env var name; added a post-run guard that fails the benchmark if a pnpr target's `pnpr-storage` is empty (proof it never served a resolve). - **Emulate a real registry link for every client.** The latency proxy modeled RTT but not throughput, and fronted only direct targets. Generalized it to a `LinkProfile` (one-way delay + per-direction bandwidth cap), added `--registry-bandwidth-mbps`, and routed *all* registry traffic through it (direct installs, the pnpr server's resolve, the pnpr client's fetches) so the registry-mock is uniformly as remote as real npm. CI runs it at 50 ms + 200 Mbit/s (≈ the measured public-npm peak). - **Make "cold cache" cold for resolution.** Forced `cacheDir` bench-local and wipe it in cold-cache scenarios, so a direct install actually pays the packument-fetch waterfall (previously the global metadata mirror survived every wipe). - **New scenario** `fresh-install.cold-cache.hot-store` that isolates resolution (cold metadata, hot store → no download to mask it). ## Results (Linux CI, after the fixes) | Scenario | pacquet@HEAD | pnpr@HEAD | pnpr speedup | |---|---:|---:|---:| | fresh-install · cold cache · **hot store** | 5.06 s | 0.69 s | **7.4×** | | fresh-install · cold cache · cold store | 5.36 s | 2.03 s | **2.65×** | | fresh-install · hot cache · hot store | 1.46 s | 0.69 s | **2.1×** | | fresh-restore (frozen) · cold cache · cold store | 10.08 s | 5.13 s | **2.0×** | | fresh-restore (frozen) · hot cache · hot store | 0.71 s | 0.80 s | 0.89× (slower) | pnpr offloads the client's resolution (and, on the frozen path, lockfile verification) to its warm server: 2–7× faster wherever the client would otherwise pay that cost. The lone regression is the fully-warm frozen install, where there's nothing to offload and pnpr's one round trip is pure overhead. `pnpr@HEAD` vs `pnpr@main` is flat throughout — i.e. the streaming commit (Part 1) adds ~nothing against a warm server, while the base pnpr win (offload to a warm server) is large.
简体中文 | 日本語 | 한국어 | Italiano | Português Brasileiro
Fast, disk space efficient package manager:
- Fast. Up to 2x faster than the alternatives (see benchmark).
- Efficient. Files inside
node_modulesare linked from a single content-addressable storage. - Great for monorepos.
- Strict. A package can access only dependencies that are specified in its
package.json. - Deterministic. Has a lockfile called
pnpm-lock.yaml. - Works as a Node.js version manager. See pnpm runtime.
- Works everywhere. Supports Windows, Linux, and macOS.
- Battle-tested. Used in production by teams of all sizes since 2016.
- See the full feature comparison with npm and Yarn.
To quote the Rush team:
Microsoft uses pnpm in Rush repos with hundreds of projects and hundreds of PRs per day, and we’ve found it to be very fast and reliable.
Platinum Sponsors
|
Gold Sponsors
|
|
|
|
|
|
|
|
Silver Sponsors
|
|
|
|
|
|
|
⏱️ Time.now |
Support this project by becoming a sponsor.
Background
pnpm uses a content-addressable filesystem to store all files from all module directories on a disk. When using npm, if you have 100 projects using lodash, you will have 100 copies of lodash on disk. With pnpm, lodash will be stored in a content-addressable storage, so:
- If you depend on different versions of lodash, only the files that differ are added to the store.
If lodash has 100 files, and a new version has a change only in one of those files,
pnpm updatewill only add 1 new file to the storage. - All the files are saved in a single place on the disk. When packages are installed, their files are linked from that single place consuming no additional disk space. Linking is performed using either hard-links or reflinks (copy-on-write).
As a result, you save gigabytes of space on your disk and you have a lot faster installations!
If you'd like more details about the unique node_modules structure that pnpm creates and
why it works fine with the Node.js ecosystem, read this small article: Flat node_modules is not the only way.
💖 Like this project? Let people know with a tweet
Getting Started
Benchmark
pnpm is up to 2x faster than npm and Yarn classic. See all benchmarks here.
Benchmarks on an app with lots of dependencies:
License
MIT, except the pnpr/ directory, which is source-available under the PolyForm Shield License 1.0.0.