Zoltan Kochan
5353fcbf01
Closes #11832. On the `alotta-files` benchmark (1362 nodes, warm cache, GVS on), pacquet was 3-5× behind the TypeScript pnpm CLI whenever resolution ran (`firstInstall`, `withWarmCache`, `withWarmModules`, `updatedDependencies`). Wall-clock dropped from ~11.83s to ~5.03s on this branch; pnpm sits at ~4.16s, and the remaining gap is concentrated in the resolver's per-node `pick_package` walk (3.1s of the 5.03s — see #11843 for the `peekManifestFromStore` follow-up that would close it). The branch is a series of small wins rather than one big rewrite. The original `PrefetchingResolver` (commitf375c916) was replaced by a batched store-index prefetch (461a4c02) — same throughput, far less plumbing. ## What's in this PR ### Resolve-phase - **Packument fetch dedup** (386a90b5) — `PackumentFetchLocker` (per-cache-key `DashMap<String, Arc<Semaphore>>`) so concurrent picks of the same `(registry, name)` coalesce into one HTTP GET. Mirrors pnpm's `runLimited(pkgMirror, …)` in `pickPackage.ts`. Pacquet was firing N parallel GETs for the same packument per cluster of cross-referencing deps; now it's one. - **Conditional GET on upgrade fetch** (58f49c90) — forward `etag` / `modified` so the registry can answer `304 Not Modified` on the abbreviated-to-full re-fetch path. - **Off-tokio mirror disk reads** (6cb50b4f) — the packument cache's mirror read moves to `spawn_blocking` instead of running on the tokio worker. - **Picked-manifest serialisation dedup** (387b8721) — `PickedManifestCache` `Arc<DashMap<String, Arc<Value>>>` so duplicate picks of the same `name@version` reuse the already-serialised `Arc<Value>` instead of re-running `serde_json::to_value`. - **Arc-shared resolver outputs** (743c718f,53e3cde6,5d6a4207) — `Package`, `ResolveResult.manifest`, and `ResolveResult` itself are now shared via `Arc` so the tree walk's per-occurrence clones become refcount bumps. - **`std::sync::Mutex` on `TreeCtx`** (a7c94a90) — the per-package dedupe gate is a short `HashMap` insert with no `await` inside, so a sync mutex is the right tool. Tokio's async mutex was paying per-acquire overhead once per visit per ctx field on the resolve hot path. ### Install-phase - **Batched store-index prefetch** (461a4c02) — one `SELECT … WHERE key IN (…)` against `index.db` at install start, rayon-parallel verify, drops the SQLite mutex before any fs work. Replaces the per-snapshot `spawn_blocking` fan-out that was serialising on `Arc<Mutex<StoreIndex>>` and queueing in tokio's blocking pool. - **Single `pnpm:progress` per URL** (e54208e1) — `run_with_mem_cache` was emitting `fetched` twice when the in-memory cache hit; mirrors pnpm's `packageRequester` shape where the emit fires exactly once. - **Retain prefetched manifests for bin linking** (b0bf5970) — the fresh-install bin linker now drives `LinkVirtualStoreBins` with the prefetched bundled-manifests map (and built lockfile snapshots), so per-child `package.json` disk reads on warm hits are gone. Skips the `read_dir` enumeration too. Updated snapshots reflect the now-present `<slot>/node_modules/<pkg>/node_modules/.bin/<pkg>` self-shim the lockfile-driven path writes per pnpm's `linkBinsOfDependencies`. - **Skip per-snapshot deep clone on warm prefetch** (3f9c1bb5) — `run_with_mem_cache` returns the prefetched `Arc<HashMap>` straight through instead of going via `run_without_mem_cache`'s deep-clone path + redundant `Arc::new`. At 1k+ snapshots that's one per-file map allocation and one `Arc::new` saved per snapshot. ### Correctness - **Registry-scoped `PickedManifestCache` key** (57c3094e) — the shared cache key was `{name}@{version}` only; two registries (default + JSR + named-registry) serving different artifacts under the same `name@version` (private + public collisions, forks) would hand one resolver the other's manifest. Now keyed `{registry}\x00{name}@{version}`, matching `PackageMetaCache`'s shape. Regression test included. ### Diagnostics - **Per-phase timing logs** (57864d2c) — `pacquet::install::phase` `tracing` events with `elapsed_ms` for `resolve_importer`, `prefetch_cas_paths`, `build_fresh_lockfile`, `virtual_store_layout_new`, `install_subtree`. Made the profiling for this branch tractable and stays in for future work (the same per-phase trace is what motivated #11843). ### Review cleanup - **Refactor + comment fixes** (e7b3e6ca) — `build_resolve_result` now takes a `BuildResolveResult` struct instead of 9 positional args (`#[allow(clippy::too_many_arguments)]` gone); resolver-side `Arc` bindings are dropped explicitly before the install pass so the packument cache actually frees; the `resolve_dependency_tree` doc comment was wrong about "skipping the recursion" on dedupe hits. ### Doc + Dylint fixes - **CI compliance** (0343a472) — broken doc links resolved; single-letter generics, `Arc.clone()` direct, unicode ellipsis in doc comments fixed for Dylint. ## Scope - **Fresh-lockfile install path only.** The frozen-lockfile path already had the batched store-index prefetch; nothing else here changes its behaviour. - **No user-visible behavior change** — lockfile format, error codes, CLI surface, `MemCache` semantics unchanged. The cache-key bug fix doesn't change the on-disk lockfile; it only prevents an in-memory mix-up that would otherwise produce a wrong `ResolveResult.manifest` field. ## Follow-ups - **#11843** — port pnpm's `peekManifestFromStore` fast path. The store-index row carries the bundled `package.json` (name, version, deps, bin, engines, etc.) but no publish-time, so the fast path is safe only when no `published_by` / `minimumReleaseAge` policy is in effect, no `--update`, and the wanted lockfile pins a tarball+integrity. With ~95% of nodes short-circuiting, the `resolve_importer` phase (currently 3.1s on warm cache) drops dramatically — this is the single biggest unimplemented win and the most likely path to parity with pnpm. ## Benchmark Wall-clock progression on the `alotta-files` warm-cache + GVS-on scenario (this branch vs `main`): | Stage | Wall | | ----------------------------------- | ------- | | `main` | ~11.83s | | + packument fetch dedup | ~8.21s | | + batched store-index prefetch | ~6.39s | | + Arc-shared resolver outputs | ~5.67s | | + std Mutex + manifest serial dedup | ~5.03s | | pnpm CLI baseline | ~4.16s | The remaining ~0.87s gap is concentrated in `resolve_importer` and is what #11843 targets.
简体中文 | 日本語 | 한국어 | Italiano | Português Brasileiro
Fast, disk space efficient package manager:
- Fast. Up to 2x faster than the alternatives (see benchmark).
- Efficient. Files inside
node_modulesare linked from a single content-addressable storage. - Great for monorepos.
- Strict. A package can access only dependencies that are specified in its
package.json. - Deterministic. Has a lockfile called
pnpm-lock.yaml. - Works as a Node.js version manager. See pnpm runtime.
- Works everywhere. Supports Windows, Linux, and macOS.
- Battle-tested. Used in production by teams of all sizes since 2016.
- See the full feature comparison with npm and Yarn.
To quote the Rush team:
Microsoft uses pnpm in Rush repos with hundreds of projects and hundreds of PRs per day, and we’ve found it to be very fast and reliable.
Platinum Sponsors
|
Gold Sponsors
|
|
|
|
|
|
|
|
Silver Sponsors
|
|
|
|
|
|
|
⏱️ Time.now |
Support this project by becoming a sponsor.
Background
pnpm uses a content-addressable filesystem to store all files from all module directories on a disk. When using npm, if you have 100 projects using lodash, you will have 100 copies of lodash on disk. With pnpm, lodash will be stored in a content-addressable storage, so:
- If you depend on different versions of lodash, only the files that differ are added to the store.
If lodash has 100 files, and a new version has a change only in one of those files,
pnpm updatewill only add 1 new file to the storage. - All the files are saved in a single place on the disk. When packages are installed, their files are linked from that single place consuming no additional disk space. Linking is performed using either hard-links or reflinks (copy-on-write).
As a result, you save gigabytes of space on your disk and you have a lot faster installations!
If you'd like more details about the unique node_modules structure that pnpm creates and
why it works fine with the Node.js ecosystem, read this small article: Flat node_modules is not the only way.
💖 Like this project? Let people know with a tweet
Getting Started
Benchmark
pnpm is up to 2x faster than npm and Yarn classic. See all benchmarks here.
Benchmarks on an app with lots of dependencies:
