## Problem
Every file extracted to the CAS goes through a temp-file-plus-rename cycle: `writeFile(temp, buffer)` then `renameOverwriteSync(temp, fileDest)`. For a typical cold install with ~30k files, this adds ~30k extra rename syscalls.
## Solution
Use `writeFileExclusive()` with `{ flag: 'wx' }` (O_CREAT|O_EXCL) to write directly to the final CAS path when the file doesn't exist — skipping the temp+rename overhead. For recovery paths (corrupt/partial files, EEXIST races), fall back to the existing atomic temp+rename via `optimisticRenameOverwrite`.
### Write paths
- **File doesn't exist (common cold-install path)** → `writeFileExclusive` writes directly, no rename
- **File exists with correct integrity** → return immediately, no write
- **File exists with wrong integrity (corruption/crash)** → atomic temp+rename recovery
- **EEXIST (concurrent write)** → verify integrity; if OK return, otherwise atomic temp+rename recovery
### Concurrent safety
- `writeFileExclusive` (`O_CREAT|O_EXCL`) ensures only one process creates a given CAS file
- Recovery overwrites use the battle-tested `optimisticRenameOverwrite` + `pathTemp` for atomic replacement
- `verifyFileIntegrity` is non-destructive (no `unlinkSync` on mismatch), safe when another process may be mid-write
- A crash mid-`writeFileExclusive` can leave a partial file, recovered on next access via atomic temp+rename
---------
Co-authored-by: Zoltan Kochan <z@kochan.io>
* fix(cafs): update locker cache when file exists with correct integrity
The CAS locker cache was not updated when a file already existed on disk
with correct integrity. This caused repeated verifyFileIntegrity calls
on subsequent lookups within the same process, adding unnecessary I/O.
* fix(test): assert locker cache value not just key existence
Strengthen the test to verify locker.get() returns the correct
checkedAt timestamp, not just that the key exists.
* perf(cafs): optimize hot path string operations
Replace path.join with string concatenation in contentPathFromHex and
getFilePathByModeInCafs. These functions are called ~30k times per
install and the simpler string operations avoid path.join's argument
validation overhead.
Increase gunzipSync chunk size from default 16KB to 128KB for faster
tarball decompression with fewer zlib iterations.
* refactor: remove dead Buffer.isBuffer check in tarball path
tarballBuffer is typed as Buffer, so the isBuffer/Buffer.from
fallback was unreachable dead code.
* docs: add comments explaining path.join bypass and chunkSize choice
Address review feedback:
- Explain why string concat is used instead of path.join in CAS hot path
- Document why 128KB chunkSize was chosen (microbenchmarks, diminishing
returns at larger sizes, bounded memory cost)
* fix: cspell — use 'Benchmarks' instead of 'Microbenchmarks'
* fix(cafs): restore Buffer.isBuffer check for worker thread compatibility
The structured clone algorithm converts Buffer to Uint8Array when sent
via postMessage to worker threads. parseTarball relies on
Buffer.prototype.toString('utf8', ...) which doesn't exist on
Uint8Array — Uint8Array.toString() returns comma-separated decimal
values, causing parseOctal to misparse tar headers.
* fix: ensure PNPM_HOME/bin is in PATH during pnpm setup
When upgrading from old pnpm (global bin = PNPM_HOME) to new pnpm
(global bin = PNPM_HOME/bin), `pnpm setup` would fail because the
spawned `pnpm add -g` checks that the global bin dir is in PATH.
Prepend PNPM_HOME/bin to PATH in the spawned process env so the
check passes during the transition.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update pnpm to v11 beta 2
* chore: update pnpm to v11 beta 2
* chore: update pnpm to v11 beta 2
* chore: update pnpm to v11 beta 2
* fix: lint
* refactor: rename _-prefixed scripts to .-prefixed scripts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: update root package.json to use .test instead of _test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: update action-setup
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update all dependencies to latest versions
Update all outdated dependencies across the monorepo catalog and fix
breaking changes from major version bumps.
Notable updates:
- ESLint 9 → 10 (fix custom rule API, disable new no-useless-assignment)
- @stylistic/eslint-plugin 4 → 5 (auto-fixed indent changes)
- @cyclonedx/cyclonedx-library 9 → 10 (adapt to removed SPDX API)
- esbuild 0.25 → 0.27
- TypeScript 5.9.2 → 5.9.3
- Various @types packages, test utilities, and build tools
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: update unified/remark/mdast imports for v11/v4 API changes
Update imports in get-release-text for the new ESM named exports:
- mdast-util-to-string: default → { toString }
- unified: default → { unified }
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: resolve typecheck errors from dependency updates
- isexe v4: use named import { sync } instead of default export
- remark-parse/remark-stringify v11: add vfile as packageExtension
dependency so TypeScript can resolve type declarations
- get-release-text: remove unused @ts-expect-error directives
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: revert runtime dependency major version bumps
Revert major version bumps for runtime dependencies that are bundled
into pnpm to fix test failures where pnpm add silently fails:
- bin-links: keep ^5.0.0 (was ^6.0.0)
- cli-truncate: keep ^4.0.0 (was ^5.2.0)
- delay: keep ^6.0.0 (was ^7.0.0)
- filenamify: keep ^6.0.0 (was ^7.0.1)
- find-up: keep ^7.0.0 (was ^8.0.0)
- isexe: keep 2.0.0 (was 4.0.0)
- normalize-newline: keep 4.1.0 (was 5.0.0)
- p-queue: keep ^8.1.0 (was ^9.1.0)
- ps-list: keep ^8.1.1 (was ^9.0.0)
- string-length: keep ^6.0.0 (was ^7.0.1)
- symlink-dir: keep ^7.0.0 (was ^9.0.0)
- terminal-link: keep ^4.0.0 (was ^5.0.0)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: restore runtime dependency major version bumps
Re-apply all runtime dependency major version bumps that were
previously reverted. All packages maintain their default exports
except isexe v4 which needs named imports.
Updated runtime deps:
- bin-links: ^5.0.0 → ^6.0.0
- cli-truncate: ^4.0.0 → ^5.2.0
- delay: ^6.0.0 → ^7.0.0
- filenamify: ^6.0.0 → ^7.0.1
- find-up: ^7.0.0 → ^8.0.0
- isexe: 2.0.0 → 4.0.0 (fix: use named import { sync })
- normalize-newline: 4.1.0 → 5.0.0
- p-queue: ^8.1.0 → ^9.1.0
- ps-list: ^8.1.1 → ^9.0.0
- string-length: ^6.0.0 → ^7.0.1
- symlink-dir: ^7.0.0 → ^9.0.0
- terminal-link: ^4.0.0 → ^5.0.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: revert tempy to 3.0.0 to fix bundle hang
tempy 3.2.0 pulls in temp-dir 3.0.0 which uses async fs.realpath()
inside its module init. When bundled by esbuild into the __esm lazy
init pattern, this causes a deadlock during module initialization,
making the pnpm binary hang silently on startup.
Keeping tempy at 3.0.0 which uses temp-dir 2.x (sync fs.realpathSync).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add comment explaining why tempy cannot be upgraded
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: revert nock to 13.3.4 for node-fetch compatibility
nock 14 changed its HTTP interception mechanism in a way that doesn't
properly intercept node-fetch requests, causing audit tests to hang
waiting for responses that are never intercepted.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add comment explaining why nock cannot be upgraded
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: update symlink-dir imports for v10 ESM named exports
symlink-dir v10 removed the default export and switched to named
exports: { symlinkDir, symlinkDirSync }.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: revert @typescript/native-preview to working version
Newer tsgo dev builds (>= 20260318) have a regression where
@types/node cannot be resolved, breaking all node built-in types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: vulnerabilities
* fix: align comment indentation in runLifecycleHook
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: pin msgpackr to 1.11.8 for TypeScript 5.9 compatibility
msgpackr 1.11.9 has broken type definitions that use Iterable/Iterator
without required type arguments, causing compile errors with TS 5.9.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: reduce noisy warnings in test output
- Suppress ExperimentalWarning and DEP0169 via --disable-warning in NODE_OPTIONS
- Fix MaxListenersExceededWarning by raising limit in StoreIndex when adding exit listeners
- Update meta-updater to generate the new _test scripts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: stop streaming pnpm subprocess output during CLI tests
Buffer stdout/stderr from execPnpm instead of writing to the parent
process in real time. Output is still included in the error message on
failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: pipe all subprocess output in CLI tests
Use stdio: 'pipe' for all pnpm/pnpx spawn helpers so subprocess output
is buffered instead of printed. Output is still included in error
messages on failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove duplicate @pnpm/installing.env-installer in pnpm/package.json
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: use pipe stdio in dlx and errorHandler tests
Replace stdio: 'inherit' and [null, 'pipe', 'inherit'] with 'pipe' to
prevent subprocess output from leaking into test output.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: skip maxListeners adjustment when set to unlimited (0)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: rename workspace.sort-packages and workspace.pkgs-graph
- workspace.sort-packages -> workspace.projects-sorter
- workspace.pkgs-graph -> workspace.projects-graph
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: rename packages/ to core/ and pkg-manifest.read-package-json to reader
- Rename packages/ directory to core/ for clarity
- Rename pkg-manifest/read-package-json to pkg-manifest/reader (@pnpm/pkg-manifest.reader)
- Update all tsconfig, package.json, and lockfile references
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: consolidate runtime resolvers under engine/runtime domain
- Remove unused @pnpm/engine.runtime.node.fetcher package
- Rename engine/runtime/node.resolver to node-resolver (dash convention)
- Move resolving/bun-resolver to engine/runtime/bun-resolver
- Move resolving/deno-resolver to engine/runtime/deno-resolver
- Update all package names, tsconfig paths, and lockfile references
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update lockfile after removing node.fetcher
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: sort tsconfig references and package.json deps alphabetically
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: auto-fix import sorting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update __typings__ paths in tsconfig.lint.json for moved resolvers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove deno-resolver from deps of bun-resolver
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add n/prefer-node-protocol rule and autofix all bare builtin imports
to use the node: prefix. Simplify the simple-import-sort builtins
pattern to just ^node: since all imports now use the prefix.
Add eslint-plugin-simple-import-sort to enforce consistent import ordering:
- Node.js builtins first
- External packages second
- Relative imports last
- Named imports sorted alphabetically within each statement
Update all packages from zkochan/packages to their latest major versions
and exclude them from minimumReleaseAge requirement. This includes
updating catalog entries, adapting to breaking API changes (default
exports replaced with named exports, sync functions renamed with Sync
suffix), and updating type declarations.
## Summary
Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries.
Closes#10826
## Design
### New package: `@pnpm/store.index`
A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage.
Key design decisions:
- **WAL mode** enables concurrent reads from workers while the main process writes.
- **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access.
- **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`.
- **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`).
- **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits.
- **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space.
### Key format
Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`.
### Shared StoreIndex instance
A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access.
### Worker architecture
Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads.
### SQLite ExperimentalWarning suppression
`node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`.
### No migration from `.mpk` files
Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store).
## Changed packages
121 files changed across these areas:
- **`store/index/`** — New `@pnpm/store.index` package
- **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process
- **`store/package-store/`** — StoreIndex creation and lifecycle management
- **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed)
- **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files
- **`store/plugin-commands-store/`** — `store status` uses StoreIndex
- **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex
- **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading
- **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter
- **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex
- **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex
- **`cache/api/`** — Cache view uses StoreIndex
- **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression
## Test plan
- [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching
- [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle
- [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index
- [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes
- [x] `pnpm --filter @pnpm/headless test` — Headless install
- [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching
- [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index
- [x] `pnpm --filter @pnpm/license-scanner test` — License scanning
- [x] e2e tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
This PR overhauls `pnpm env` use to route through pnpm's own install machinery instead of maintaining a parallel code path with manual symlink/shim/hardlink logic.
```
pnpm env use -g <version>
```
now runs:
```
pnpm add --global node@runtime:<version>
```
via `@pnpm/exec.pnpm-cli-runner`. All manual symlink, hardlink, and cmd-shim code in `envUse.ts` is gone (~1000 lines removed across the package).
### Changes
**npm and npx shims on all platforms**
Added `getNodeBinsForCurrentOS(platform)` to `@pnpm/constants`, returning a `Record<string, string>` with the correct relative paths for `node`, `npm`, and `npx` inside a Node.js distribution. `BinaryResolution.bin` is widened from `string` to `string | Record<string, string>` in `@pnpm/resolver-base` and `@pnpm/lockfile.types`, so the node resolver can set all three entries and pnpm's bin-linker creates shims for each automatically.
**Windows npm/npx fix**
`addFilesFromDir` was skipping root-level `node_modules/` (to avoid storing a package's own dependencies), which stripped the bundled `npm` from Node.js Windows zip archives. Added an `includeNodeModules` option and enabled it from the binary fetcher so Windows distributions keep their full contents.
**Removed subcommands**
`pnpm env add` and `pnpm env remove` are removed. `pnpm env use` handles both installing and activating a version. `pnpm env list` now always lists remote versions (the `--remote` flag is no longer required, though it is kept for backwards compatibility).
**musl support**
On Alpine Linux and other musl-based systems, the musl variant of Node.js is automatically downloaded from [unofficial-builds.nodejs.org](https://unofficial-builds.nodejs.org).
* fix: retry filesystem operations on EAGAIN
filesystem operations can raise EAGAIN to tell the application to try
again later. This is especially often the case under ZFS.
fix: move wrapped functions to graceful-fs directly
* fix: retry filesystem operations on EAGAIN
* fix: retry filesystem operations on EAGAIN
* fix: indexed-pkg-importer
* test: fix
* docs: add changeset
---------
Co-authored-by: Zoltan Kochan <z@kochan.io>
* chore: `package.json` add type field
* chore: add type field to every package.json
* chore: add type field to every package.json
---------
Co-authored-by: Zoltan Kochan <z@kochan.io>
