mirror of
https://github.com/pnpm/pnpm.git
synced 2026-06-28 18:05:29 -04:00
df77f649ee
* fix(pacquet/fs): serialize concurrent CAS writes to the same path
Two install snapshots whose tarballs ship identical file content
(e.g. a shared LICENSE across sibling packages like
`@pnpm.e2e/hello-world-js-bin` and `@pnpm.e2e/hello-world-js-bin-parent`)
hash to the same CAS path and call `ensure_file` concurrently. Without
serialization the second writer's `O_CREAT|O_EXCL` hits `AlreadyExists`
while the first writer is mid-`write_all`, falls into
`verify_or_rewrite`'s `meta.len() != content.len()` arm, and runs
`write_atomic` — which renames a temp file over the live source.
On Linux/ext4 the partial-size observation is fast enough that this
window opens often, surfacing as flaky CI failures of the form:
failed to import "<store>/v11/files/65/<hash>" to ".../LICENSE":
No such file or directory (os error 2)
emitted by `link_file` whose `reflink`/`fs::hard_link` raced the
rename. macOS/APFS and Windows tests pass because their `stat` cadence
and CI runner parallelism don't reliably open the window.
Port pnpm v11's `locker: Map<string, number>` semantics — slightly
stronger in pacquet, as a per-path `Mutex` rather than a dedup cache —
so two writers of the same CAS path serialize through it. The second
caller acquires the lock after the first writer's `write_all` has
finished, then takes the byte-match fast path inside `verify_or_rewrite`
and never has to rewrite. The previous docstring at the bottom of the
"Differences from pnpm" list explicitly acknowledged the locker
omission and predicted it would matter; this change closes that gap.
Add a regression test (`concurrent_writers_of_same_path_do_not_swap_the_inode`)
that fires 32 threads at one path with identical content and asserts
the inode never swaps — the observable signal that no writer ever took
the `write_atomic` rename path.
* fix(pacquet/fs): unlink intra-doc references to private items
`ensure_file` is public; references to private `verify_or_rewrite`,
`write_atomic`, and `cas_write_lock` only resolve under
`--document-private-items`, which trips `rustdoc::private-intra-doc-links`
under `-D warnings`. Drop the link form and keep the names as plain
backticked identifiers — the docstring still reads correctly and
`cargo doc` no longer fails.
* style(pacquet/fs): rename single-letter N to WRITER_COUNT in test
Address review feedback on #11758 — single-letter constants are
opaque; `WRITER_COUNT` reads as the loop count of concurrent
writers the test fires at one CAS path.
* docs(pacquet/fs): tighten ensure_file / cas_write_lock / test docs
Address review feedback on #11758: drop body-narrating prose, keep
only the contract and the non-obvious why. The test docstring no
longer references "pre-fix code" so it reads independent of this
PR's history.
* test(pacquet/fs): pre-create + per-thread inode capture in concurrent test
Address CodeRabbit's and Copilot's review feedback on #11758. The
previous assert compared two metadata reads taken at the same moment
after join — tautological. Pre-creating the file gives an
`original_ino` reference taken before the contended run, and each
writer also captures the path's inode immediately after its own
`ensure_file` returns so a mid-run rename swap from another writer
is visible to at least one of those observations.
* style(pacquet/fs): add trailing comma in multi-line assert! macro
Address `perfectionist::macro_trailing_comma` warning surfaced by
the Dylint CI on #11758 — multi-line macro invocations must end with
a trailing comma. (My earlier local `dylint --all` run missed this
because the perfectionist library hadn't fully recompiled against the
new test code.)
* test(pacquet/fs): drop pre-create from concurrent writer test
Address Copilot review feedback on #11758. The pre-create made every
contender take the byte-match fast path against a complete file, so
the lock made no observable difference and the test couldn't even
weakly distinguish lock from no-lock. Removing it leaves the
fresh-dirent shape (`O_CREAT|O_EXCL` race + verify_or_rewrite for
the rest), and the per-thread inode observations can now diverge
under a multi-rename race without the lock.
Honest about the limitation in the docstring: any observation taken
after `ensure_file` returns has already missed the rename window, so
a single-rename race converges on one inode and slips past. The
test catches the multi-rename case and validates the "no deadlock,
all writers see correct content" baseline.
---------
Co-authored-by: Claude <noreply@anthropic.com>
简体中文 | 日本語 | 한국어 | Italiano | Português Brasileiro
Fast, disk space efficient package manager:
- Fast. Up to 2x faster than the alternatives (see benchmark).
- Efficient. Files inside
node_modulesare linked from a single content-addressable storage. - Great for monorepos.
- Strict. A package can access only dependencies that are specified in its
package.json. - Deterministic. Has a lockfile called
pnpm-lock.yaml. - Works as a Node.js version manager. See pnpm runtime.
- Works everywhere. Supports Windows, Linux, and macOS.
- Battle-tested. Used in production by teams of all sizes since 2016.
- See the full feature comparison with npm and Yarn.
To quote the Rush team:
Microsoft uses pnpm in Rush repos with hundreds of projects and hundreds of PRs per day, and we’ve found it to be very fast and reliable.
Platinum Sponsors
|
Gold Sponsors
|
|
|
|
|
|
|
|
Silver Sponsors
|
|
|
|
|
|
|
⏱️ Time.now |
Support this project by becoming a sponsor.
Background
pnpm uses a content-addressable filesystem to store all files from all module directories on a disk. When using npm, if you have 100 projects using lodash, you will have 100 copies of lodash on disk. With pnpm, lodash will be stored in a content-addressable storage, so:
- If you depend on different versions of lodash, only the files that differ are added to the store.
If lodash has 100 files, and a new version has a change only in one of those files,
pnpm updatewill only add 1 new file to the storage. - All the files are saved in a single place on the disk. When packages are installed, their files are linked from that single place consuming no additional disk space. Linking is performed using either hard-links or reflinks (copy-on-write).
As a result, you save gigabytes of space on your disk and you have a lot faster installations!
If you'd like more details about the unique node_modules structure that pnpm creates and
why it works fine with the Node.js ecosystem, read this small article: Flat node_modules is not the only way.
💖 Like this project? Let people know with a tweet
Getting Started
Benchmark
pnpm is up to 2x faster than npm and Yarn classic. See all benchmarks here.
Benchmarks on an app with lots of dependencies:
