Commit Graph

491 Commits

Author SHA1 Message Date
Zoltan Kochan
73d9a83f52 Revert "perf: move resolver CPU work to worker thread"
This reverts commit 2c4e9cc263.
2026-04-01 19:05:28 +02:00
Zoltan Kochan
2c4e9cc263 perf: move resolver CPU work to worker thread
Move SQLite reads/writes, JSON.parse, semver matching, and version
picking from the main thread to the existing worker pool.

The main thread is now a thin orchestrator that only does network I/O.
Resolution uses at most 2 round-trips to the worker:
1. Worker checks SQLite cache → cache hit returns immediately
2. On cache miss, main thread fetches from registry, sends raw JSON
   to worker → worker parses, writes cache, picks version

This unblocks the main event loop during resolution — network I/O,
tarball downloads, and linking can proceed while the worker does
CPU-heavy parsing and semver matching.
2026-04-01 18:59:18 +02:00
Zoltan Kochan
cbf7dfcf65 fix: guard MetadataCache methods against use after close
Add early-return guards to getHeaders, getIndex, getManifest, and
updateCachedAt when the DB has been closed. This prevents "statement
has been finalized" errors when the process exit handler closes the
DB while async operations are still in flight.

Also change store controller close to flush (not close) the metadata
DB, since the exit handler handles cleanup.
2026-04-01 16:55:07 +02:00
Zoltan Kochan
3ea70465a1 fix: address Copilot review feedback
- Set meta.modified from DB row in loadMetaFromDb so If-Modified-Since
  headers are sent even when modified comes from the DB column
- Use || instead of && for header lookup so both etag and modified are
  populated even when one is already available from in-memory metadata
- Check pending writes in MetadataCache.get/getHeaders instead of
  flushing in loadMetaFromDb, avoiding synchronous DB writes during
  resolution
- Close MetadataCache when store controller is closed to prevent
  resource leaks
- Add defensive slash-index guards in cache API functions
2026-04-01 14:55:49 +02:00
Zoltan Kochan
ed6bae5b9c refactor: remove legacy metadata directory constants, update remaining tests
- Remove ABBREVIATED_META_DIR, FULL_META_DIR, FULL_FILTERED_META_DIR
  from @pnpm/constants — no longer used
- Update lockfileOnly test to check MetadataCache DB instead of files
- Update prune to also remove metadata.db and its WAL/SHM files
2026-04-01 14:55:19 +02:00
Zoltan Kochan
89ea578e68 perf: replace file-based metadata cache with SQLite
Replace the per-package JSON file cache (metadata-v1.4/, metadata-ff-v1.4/)
with a single SQLite database (metadata.db) for registry metadata caching.

Benefits:
- Cheap conditional request header lookups (etag/modified) without parsing
  the full metadata JSON — enables If-None-Match/If-Modified-Since with
  minimal I/O overhead
- Full metadata can serve abbreviated requests — if a package was previously
  fetched as full (e.g., for trustPolicy or resolutionMode), the resolver
  reuses it instead of making another registry request
- Eliminates hundreds of individual file read/write/rename operations per
  install, replaced by SQLite WAL-mode transactions
- Removes the runLimited/metafileOperationLimits concurrency machinery —
  SQLite handles concurrent access natively

New package: @pnpm/cache.metadata — SQLite-backed MetadataCache class
modeled after @pnpm/store.index, with getHeaders() for cheap lookups,
get() with abbreviated→full fallback, and set()/updateCachedAt().
2026-04-01 14:54:13 +02:00
Zoltan Kochan
968724fc0b perf: use abbreviated metadata for minimumReleaseAge (#11160)
* perf: use abbreviated metadata for minimumReleaseAge when possible

Instead of always fetching full package metadata when minimumReleaseAge
is set, fetch the smaller abbreviated document first and check the
top-level `modified` field. If the package was last modified before the
release age cutoff, all versions are mature and no per-version time
filtering is needed. Only re-fetch full metadata for the rare case of
recently-modified packages.

Also uses fs.stat() to check cache file mtime instead of reading and
parsing the JSON to check cachedAt, avoiding unnecessary I/O.

* fix: validate modified date and handle abbreviated metadata edge cases

- Validate meta.modified date to prevent invalid dates from bypassing
  minimumReleaseAge filtering
- Skip full metadata refetch for packages excluded by publishedByExclude
- Allow ERR_PNPM_MISSING_TIME from cached abbreviated metadata to fall
  through to the network fetch path instead of throwing

* fix: cache abbreviated metadata before re-fetching full metadata

Save the abbreviated metadata to disk before re-fetching full metadata
so subsequent runs benefit from the mtime cache fast-path.

* fix: resolve type narrowing for conditional metadata fetch result
2026-04-01 14:47:31 +02:00
Zoltan Kochan
d6b8e281b6 chore: use pn instead of pnpm (#11124) 2026-03-28 11:55:51 +01:00
Victor Sumner
878a7734a0 perf(cafs): skip rename on cold CAS writes, use atomic rename for recovery (#11087)
## Problem

Every file extracted to the CAS goes through a temp-file-plus-rename cycle: `writeFile(temp, buffer)` then `renameOverwriteSync(temp, fileDest)`. For a typical cold install with ~30k files, this adds ~30k extra rename syscalls.

## Solution

Use `writeFileExclusive()` with `{ flag: 'wx' }` (O_CREAT|O_EXCL) to write directly to the final CAS path when the file doesn't exist — skipping the temp+rename overhead. For recovery paths (corrupt/partial files, EEXIST races), fall back to the existing atomic temp+rename via `optimisticRenameOverwrite`.

### Write paths

- **File doesn't exist (common cold-install path)** → `writeFileExclusive` writes directly, no rename
- **File exists with correct integrity** → return immediately, no write
- **File exists with wrong integrity (corruption/crash)** → atomic temp+rename recovery
- **EEXIST (concurrent write)** → verify integrity; if OK return, otherwise atomic temp+rename recovery

### Concurrent safety

- `writeFileExclusive` (`O_CREAT|O_EXCL`) ensures only one process creates a given CAS file
- Recovery overwrites use the battle-tested `optimisticRenameOverwrite` + `pathTemp` for atomic replacement
- `verifyFileIntegrity` is non-destructive (no `unlinkSync` on mismatch), safe when another process may be mid-write
- A crash mid-`writeFileExclusive` can leave a partial file, recovered on next access via atomic temp+rename

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
2026-03-25 15:20:34 +01:00
Victor Sumner
6656baaea3 fix(cafs): update locker cache when file exists with correct integrity (#11085)
* fix(cafs): update locker cache when file exists with correct integrity

The CAS locker cache was not updated when a file already existed on disk
with correct integrity. This caused repeated verifyFileIntegrity calls
on subsequent lookups within the same process, adding unnecessary I/O.

* fix(test): assert locker cache value not just key existence

Strengthen the test to verify locker.get() returns the correct
checkedAt timestamp, not just that the key exists.
2026-03-25 02:12:00 +01:00
Victor Sumner
f8e6774273 perf(cafs): optimize hot path string operations (#11086)
* perf(cafs): optimize hot path string operations

Replace path.join with string concatenation in contentPathFromHex and
getFilePathByModeInCafs. These functions are called ~30k times per
install and the simpler string operations avoid path.join's argument
validation overhead.

Increase gunzipSync chunk size from default 16KB to 128KB for faster
tarball decompression with fewer zlib iterations.

* refactor: remove dead Buffer.isBuffer check in tarball path

tarballBuffer is typed as Buffer, so the isBuffer/Buffer.from
fallback was unreachable dead code.

* docs: add comments explaining path.join bypass and chunkSize choice

Address review feedback:
- Explain why string concat is used instead of path.join in CAS hot path
- Document why 128KB chunkSize was chosen (microbenchmarks, diminishing
  returns at larger sizes, bounded memory cost)

* fix: cspell — use 'Benchmarks' instead of 'Microbenchmarks'

* fix(cafs): restore Buffer.isBuffer check for worker thread compatibility

The structured clone algorithm converts Buffer to Uint8Array when sent
via postMessage to worker threads. parseTarball relies on
Buffer.prototype.toString('utf8', ...) which doesn't exist on
Uint8Array — Uint8Array.toString() returns comma-separated decimal
values, causing parseOctal to misparse tar headers.
2026-03-25 02:08:19 +01:00
Zoltan Kochan
421ceac0b3 chore: compile pnpm CLI bundle before tests that use it (#11059)
Packages whose tests spawn the local pnpm CLI (pnpm/bin/pnpm.mjs) need
the bundle (pnpm/dist/pnpm.mjs) to exist. Add `pnpm --filter pnpm run
compile` to their test scripts so the bundle is built before tests run.
2026-03-22 10:56:36 +01:00
btea
2f98ec84f4 feat: store prune displays the total size of removed files (#11047)
* feat: store prune displays the total size of removed files

* test: update
2026-03-21 20:01:58 +01:00
Zoltan Kochan
cd2dc7d481 refactor: prefix internal scripts with . to hide them (#11051)
* fix: ensure PNPM_HOME/bin is in PATH during pnpm setup

When upgrading from old pnpm (global bin = PNPM_HOME) to new pnpm
(global bin = PNPM_HOME/bin), `pnpm setup` would fail because the
spawned `pnpm add -g` checks that the global bin dir is in PATH.
Prepend PNPM_HOME/bin to PATH in the spawned process env so the
check passes during the transition.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update pnpm to v11 beta 2

* chore: update pnpm to v11 beta 2

* chore: update pnpm to v11 beta 2

* chore: update pnpm to v11 beta 2

* fix: lint

* refactor: rename _-prefixed scripts to .-prefixed scripts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update root package.json to use .test instead of _test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* ci: update action-setup

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 14:30:56 +01:00
Zoltan Kochan
0d88df854f chore: update all dependencies to latest versions (#11032)
* chore: update all dependencies to latest versions

Update all outdated dependencies across the monorepo catalog and fix
breaking changes from major version bumps.

Notable updates:
- ESLint 9 → 10 (fix custom rule API, disable new no-useless-assignment)
- @stylistic/eslint-plugin 4 → 5 (auto-fixed indent changes)
- @cyclonedx/cyclonedx-library 9 → 10 (adapt to removed SPDX API)
- esbuild 0.25 → 0.27
- TypeScript 5.9.2 → 5.9.3
- Various @types packages, test utilities, and build tools

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update unified/remark/mdast imports for v11/v4 API changes

Update imports in get-release-text for the new ESM named exports:
- mdast-util-to-string: default → { toString }
- unified: default → { unified }

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: resolve typecheck errors from dependency updates

- isexe v4: use named import { sync } instead of default export
- remark-parse/remark-stringify v11: add vfile as packageExtension
  dependency so TypeScript can resolve type declarations
- get-release-text: remove unused @ts-expect-error directives

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert runtime dependency major version bumps

Revert major version bumps for runtime dependencies that are bundled
into pnpm to fix test failures where pnpm add silently fails:
- bin-links: keep ^5.0.0 (was ^6.0.0)
- cli-truncate: keep ^4.0.0 (was ^5.2.0)
- delay: keep ^6.0.0 (was ^7.0.0)
- filenamify: keep ^6.0.0 (was ^7.0.1)
- find-up: keep ^7.0.0 (was ^8.0.0)
- isexe: keep 2.0.0 (was 4.0.0)
- normalize-newline: keep 4.1.0 (was 5.0.0)
- p-queue: keep ^8.1.0 (was ^9.1.0)
- ps-list: keep ^8.1.1 (was ^9.0.0)
- string-length: keep ^6.0.0 (was ^7.0.1)
- symlink-dir: keep ^7.0.0 (was ^9.0.0)
- terminal-link: keep ^4.0.0 (was ^5.0.0)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: restore runtime dependency major version bumps

Re-apply all runtime dependency major version bumps that were
previously reverted. All packages maintain their default exports
except isexe v4 which needs named imports.

Updated runtime deps:
- bin-links: ^5.0.0 → ^6.0.0
- cli-truncate: ^4.0.0 → ^5.2.0
- delay: ^6.0.0 → ^7.0.0
- filenamify: ^6.0.0 → ^7.0.1
- find-up: ^7.0.0 → ^8.0.0
- isexe: 2.0.0 → 4.0.0 (fix: use named import { sync })
- normalize-newline: 4.1.0 → 5.0.0
- p-queue: ^8.1.0 → ^9.1.0
- ps-list: ^8.1.1 → ^9.0.0
- string-length: ^6.0.0 → ^7.0.1
- symlink-dir: ^7.0.0 → ^9.0.0
- terminal-link: ^4.0.0 → ^5.0.0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert tempy to 3.0.0 to fix bundle hang

tempy 3.2.0 pulls in temp-dir 3.0.0 which uses async fs.realpath()
inside its module init. When bundled by esbuild into the __esm lazy
init pattern, this causes a deadlock during module initialization,
making the pnpm binary hang silently on startup.

Keeping tempy at 3.0.0 which uses temp-dir 2.x (sync fs.realpathSync).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add comment explaining why tempy cannot be upgraded

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert nock to 13.3.4 for node-fetch compatibility

nock 14 changed its HTTP interception mechanism in a way that doesn't
properly intercept node-fetch requests, causing audit tests to hang
waiting for responses that are never intercepted.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add comment explaining why nock cannot be upgraded

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update symlink-dir imports for v10 ESM named exports

symlink-dir v10 removed the default export and switched to named
exports: { symlinkDir, symlinkDirSync }.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert @typescript/native-preview to working version

Newer tsgo dev builds (>= 20260318) have a regression where
@types/node cannot be resolved, breaking all node built-in types.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: vulnerabilities

* fix: align comment indentation in runLifecycleHook

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: pin msgpackr to 1.11.8 for TypeScript 5.9 compatibility

msgpackr 1.11.9 has broken type definitions that use Iterable/Iterator
without required type arguments, causing compile errors with TS 5.9.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 23:28:53 +01:00
Zoltan Kochan
1701a65845 chore: reduce noisy warnings in test output (#11022)
* chore: reduce noisy warnings in test output

- Suppress ExperimentalWarning and DEP0169 via --disable-warning in NODE_OPTIONS
- Fix MaxListenersExceededWarning by raising limit in StoreIndex when adding exit listeners
- Update meta-updater to generate the new _test scripts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: stop streaming pnpm subprocess output during CLI tests

Buffer stdout/stderr from execPnpm instead of writing to the parent
process in real time. Output is still included in the error message on
failure.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: pipe all subprocess output in CLI tests

Use stdio: 'pipe' for all pnpm/pnpx spawn helpers so subprocess output
is buffered instead of printed. Output is still included in error
messages on failure.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove duplicate @pnpm/installing.env-installer in pnpm/package.json

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: use pipe stdio in dlx and errorHandler tests

Replace stdio: 'inherit' and [null, 'pipe', 'inherit'] with 'pipe' to
prevent subprocess output from leaking into test output.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: skip maxListeners adjustment when set to unlimited (0)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 10:43:12 +01:00
Zoltan Kochan
dba4153767 refactor: rename packages and consolidate runtime resolvers (#10999)
* refactor: rename workspace.sort-packages and workspace.pkgs-graph

- workspace.sort-packages -> workspace.projects-sorter
- workspace.pkgs-graph -> workspace.projects-graph

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: rename packages/ to core/ and pkg-manifest.read-package-json to reader

- Rename packages/ directory to core/ for clarity
- Rename pkg-manifest/read-package-json to pkg-manifest/reader (@pnpm/pkg-manifest.reader)
- Update all tsconfig, package.json, and lockfile references

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: consolidate runtime resolvers under engine/runtime domain

- Remove unused @pnpm/engine.runtime.node.fetcher package
- Rename engine/runtime/node.resolver to node-resolver (dash convention)
- Move resolving/bun-resolver to engine/runtime/bun-resolver
- Move resolving/deno-resolver to engine/runtime/deno-resolver
- Update all package names, tsconfig paths, and lockfile references

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update lockfile after removing node.fetcher

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: sort tsconfig references and package.json deps alphabetically

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: auto-fix import sorting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: update __typings__ paths in tsconfig.lint.json for moved resolvers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: remove deno-resolver from deps of bun-resolver

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-18 00:19:58 +01:00
Zoltan Kochan
4a36b9a110 refactor: rename internal packages to @pnpm/<domain>.<leaf> convention (#10997)
## Summary

Rename all internal packages so their npm names follow the `@pnpm/<domain>.<leaf>` convention, matching their directory structure. Also rename directories to remove redundancy and improve clarity.

### Bulk rename (94 packages)

All `@pnpm/` packages now derive their name from their directory path using dot-separated segments. Exceptions: `packages/`, `__utils__/`, and `pnpm/artifacts/` keep leaf names only.

### Directory renames (removing redundant prefixes)

- `cli/cli-meta` → `cli/meta`, `cli/cli-utils` → `cli/utils`
- `config/config` → `config/reader`, `config/config-writer` → `config/writer`
- `fetching/fetching-types` → `fetching/types`
- `lockfile/lockfile-to-pnp` → `lockfile/to-pnp`
- `store/store-connection-manager` → `store/connection-manager`
- `store/store-controller-types` → `store/controller-types`
- `store/store-path` → `store/path`

### Targeted renames (clarity improvements)

- `deps/dependency-path` → `deps/path` (`@pnpm/deps.path`)
- `deps/calc-dep-state` → `deps/graph-hasher` (`@pnpm/deps.graph-hasher`)
- `deps/inspection/dependencies-hierarchy` → `deps/inspection/tree-builder` (`@pnpm/deps.inspection.tree-builder`)
- `bins/link-bins` → `bins/linker`, `bins/remove-bins` → `bins/remover`, `bins/package-bins` → `bins/resolver`
- `installing/get-context` → `installing/context`
- `store/package-store` → `store/controller`
- `pkg-manifest/manifest-utils` → `pkg-manifest/utils`

### Manifest reader/writer renames

- `workspace/read-project-manifest` → `workspace/project-manifest-reader` (`@pnpm/workspace.project-manifest-reader`)
- `workspace/write-project-manifest` → `workspace/project-manifest-writer` (`@pnpm/workspace.project-manifest-writer`)
- `workspace/read-manifest` → `workspace/workspace-manifest-reader` (`@pnpm/workspace.workspace-manifest-reader`)
- `workspace/manifest-writer` → `workspace/workspace-manifest-writer` (`@pnpm/workspace.workspace-manifest-writer`)

### Workspace package renames

- `workspace/find-packages` → `workspace/projects-reader`
- `workspace/find-workspace-dir` → `workspace/root-finder`
- `workspace/resolve-workspace-range` → `workspace/range-resolver`
- `workspace/filter-packages-from-dir` merged into `workspace/filter-workspace-packages` → `workspace/projects-filter`

### Domain moves

- `pkg-manifest/read-project-manifest` → `workspace/project-manifest-reader`
- `pkg-manifest/write-project-manifest` → `workspace/project-manifest-writer`
- `pkg-manifest/exportable-manifest` → `releasing/exportable-manifest`

### Scope

- 1206 files changed
- Updated: package.json names/deps, TypeScript imports, tsconfig references, changeset files, renovate.json, test fixtures, import ordering
2026-03-17 21:50:40 +01:00
Zoltan Kochan
7a304b17c4 refactor: rename directories and unify command packages per domain (#10993)
- Rename `installing/core` → `installing/deps-installer` and `installing/headless` → `installing/deps-restorer` for clearer naming
- Rename all `plugin-commands-*` directories to use `-commands` suffix convention
- Merge multiple command packages per domain into a single `commands/` directory (one commands package per domain rule):
  - `building/{build-commands,policy-commands}` → `building/commands`
  - `deps/compliance/{audit-commands,licenses-commands,sbom-commands}` → `deps/compliance/commands`
  - `deps/inspection/{listing-commands,outdated-commands}` → `deps/inspection/commands`
  - `store/{store-commands,inspecting-commands}` → `store/commands`
  - `releasing/{publish-commands,deploy-commands}` → `releasing/commands`
  - `cli/{completion-commands,doctor-commands}` → `cli/commands`
  - `engine/pm/{self-updater-commands,setup-commands}` → `engine/pm/commands`
  - `engine/runtime/{runtime-commands,env-commands}` → `engine/runtime/commands`
  - `cache/cache-commands` → `cache/commands`
- Fix relative paths in merged test files (pnpmBin, __typings__ references)
- Update jest config to ignore `utils/` dirs at any nesting depth under `test/`
- Fix stale package names in changeset files
2026-03-17 17:42:20 +01:00
Zoltan Kochan
f47ef4b125 refactor: reorganize monorepo domain structure (#10987)
Reorganize the monorepo's top-level domain directories for clarity:

- pkg-manager/ split into:
  - installing/ (core, headless, client, resolve-dependencies, etc.)
  - installing/linking/ (hoist, direct-dep-linker, modules-cleaner, etc.)
  - bins/ (link-bins, package-bins, remove-bins)
- completion/ merged into cli/
- dedupe/ moved to installing/dedupe/
- env/ renamed to engine/ with subdomains:
  - engine/runtime/ (node.fetcher, node.resolver, plugin-commands-env, etc.)
  - engine/pm/ (plugin-commands-setup, plugin-commands-self-updater)
- env.path moved to shell/
- tools/ and runtime/ dissolved
- reviewing/ and lockfile audit packages moved to deps/:
  - deps/inspection/ (list, outdated, dependencies-hierarchy)
  - deps/compliance/ (audit, licenses, sbom)
- registry/ moved to resolving/registry/
- semver/peer-range moved to deps/
- network/fetching-types moved to fetching/
- packages/ slimmed down, moving packages to proper domains:
  - calc-dep-state, dependency-path -> deps/
  - parse-wanted-dependency -> resolving/
  - git-utils -> network/
  - naming-cases -> text/
  - make-dedicated-lockfile -> lockfile/
  - render-peer-issues -> installing/
  - plugin-commands-doctor -> cli/
  - plugin-commands-init -> workspace/

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-17 13:45:54 +01:00
Zoltan Kochan
5d5818e44f style: enforce node: protocol for builtin imports (#10951)
Add n/prefer-node-protocol rule and autofix all bare builtin imports
to use the node: prefix. Simplify the simple-import-sort builtins
pattern to just ^node: since all imports now use the prefix.
2026-03-13 07:59:51 +01:00
Zoltan Kochan
1c8c4e49f5 style: add eslint-plugin-simple-import-sort (#10947)
Add eslint-plugin-simple-import-sort to enforce consistent import ordering:
- Node.js builtins first
- External packages second
- Relative imports last
- Named imports sorted alphabetically within each statement
2026-03-13 02:02:38 +01:00
Zoltan Kochan
62f760ec3d fix(dlx): fix race conditions in parallel dlx calls sharing Global Virtual Store (#10939)
## Summary

Fixes intermittent failures in the `parallel dlx calls of the same package` test, especially on Windows CI. Multiple race conditions were discovered when concurrent `pnpm dlx` processes share the same Global Virtual Store (GVS):

- **Content-verified skip in GVS mode**: When `safeToSkip` is true and a rename fails because the target already exists (ENOTEMPTY/EEXIST/EPERM), verify all files match (inode or content comparison) before skipping. Falls through to `renameOverwriteSync` if content doesn't match.
- **Tolerate EPERM during bin creation on Windows**: `cmd-shim`'s `chmod` can fail with EPERM when another process holds the `.bin` file. Warn instead of crashing.
- **Handle EPERM in DLX cache symlink**: Added EPERM to the list of tolerated errors when creating the DLX cache symlink, as Windows can throw this when another process has the symlink open.

## Test plan

- [x] `parallel dlx calls of the same package` test passes on Windows CI
- [x] Full test suite passes on both Ubuntu and Windows
2026-03-12 21:06:32 +01:00
Zoltan Kochan
3a5bfaa94f chore: update zkochan packages to latest versions (#10930)
Update all packages from zkochan/packages to their latest major versions
and exclude them from minimumReleaseAge requirement. This includes
updating catalog entries, adapting to breaking API changes (default
exports replaced with named exports, sync functions renamed with Sync
suffix), and updating type declarations.
2026-03-11 13:47:46 +01:00
Zoltan Kochan
2fccb03fbe refactor: consolidate build-related packages into building/ domain (#10918)
* refactor: rename rebuildSelectedPkgs/rebuildProjects to buildSelectedPkgs/buildProjects

The "rebuild" prefix is redundant now that these functions live in
@pnpm/building.after-install.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: rename Rebuild option types to Build (RebuildOptions → BuildOptions, etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: rename plugin-commands-rebuild and exec.build-commands to building domain

- @pnpm/plugin-commands-rebuild → @pnpm/building.build-commands
- @pnpm/exec.build-commands → @pnpm/building.policy-commands

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: move build-modules and pkg-requires-build to building domain

- @pnpm/build-modules → @pnpm/building.during-install
- @pnpm/exec.pkg-requires-build → @pnpm/building.pkg-requires-build

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: alphabetically sort imports after package renames

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add changeset

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 10:56:13 +01:00
Brandon Cheng
01914345d5 build: enable @typescript-eslint/no-import-type-side-effects (#10630)
* build: enable `@typescript-eslint/no-import-type-side-effects`

* build: disable `@typescript-eslint/consistent-type-imports`

* chore: apply fixes for `no-import-type-side-effects`

pnpm exec eslint "**/src/**/*.ts" "**/test/**/*.ts" --fix
2026-03-08 00:02:48 +01:00
Zoltan Kochan
b7f0f21582 feat: use SQLite for storing package index in the content-addressable store (#10827)
## Summary

Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries.

Closes #10826

## Design

### New package: `@pnpm/store.index`

A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage.

Key design decisions:

- **WAL mode** enables concurrent reads from workers while the main process writes.
- **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access.
- **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`.
- **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`).
- **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits.
- **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space.

### Key format

Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`.

### Shared StoreIndex instance

A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access.

### Worker architecture

Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads.

### SQLite ExperimentalWarning suppression

`node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`.

### No migration from `.mpk` files

Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store).

## Changed packages

121 files changed across these areas:

- **`store/index/`** — New `@pnpm/store.index` package
- **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process
- **`store/package-store/`** — StoreIndex creation and lifecycle management
- **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed)
- **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files
- **`store/plugin-commands-store/`** — `store status` uses StoreIndex
- **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex
- **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading
- **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter
- **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex
- **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex
- **`cache/api/`** — Cache view uses StoreIndex
- **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression

## Test plan

- [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching
- [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle
- [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index
- [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes
- [x] `pnpm --filter @pnpm/headless test` — Headless install
- [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching
- [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index
- [x] `pnpm --filter @pnpm/license-scanner test` — License scanning
- [x] e2e tests pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-03-06 12:59:04 +01:00
Dhruv Bhanderi
ebe166c30c fix(store): include devDependencies in bundled manifest (#10711) 2026-03-01 15:50:01 +01:00
Zoltan Kochan
fd511e4676 feat: isolated global packages (#10697)
**TLDR:** Global packages in pnpm v10 are annoying and slow because they all are installed to a single global package. Instead, we will now use a system that is similar to the one used by "pnpm dlx" (aka "pnpx").

Each globally installed package (or group of packages installed together) now gets its own isolated installation directory with its own `package.json`, `node_modules`, and lockfile. This prevents global packages from interfering with each other through peer dependency conflicts or version resolution shifts.

## Changes

- Add `@pnpm/global-packages` shared utilities package for scanning, hashing, and managing isolated global installs
- `pnpm add -g` creates isolated installs in `{pnpmHomeDir}/global/v11/{hash}/`
- `pnpm remove -g` removes the entire installation group containing the package
- `pnpm update -g` re-installs into new isolated directories and swaps symlinks
- `pnpm list -g` scans isolated directories to show installed global packages
- `pnpm outdated -g` checks each isolated installation for outdated dependencies
- `pnpm store prune` cleans up orphaned global installation directories

## Breaking changes

- `pnpm install -g` (no args) is no longer supported — use `pnpm add -g <pkg>`
- `pnpm link <pkg-name>` no longer resolves packages from the global store — only relative or absolute paths are accepted
- `pnpm link --global` is removed — use `pnpm add -g .` to register a local package's bins globally
2026-03-01 15:49:18 +01:00
Allan Kimmer Jensen
f92ac24c1b feat(sbom): add pnpm sbom command (#10592)
* feat(sbom): add `pnpm sbom` command (#9088)

new command that generates SBOMs from the lockfile + store metadata.
supports CycloneDX 1.6 JSON and SPDX 2.3 JSON via `--sbom-format`.

two new packages following the existing `pnpm licenses` architecture:
- `@pnpm/sbom` — core library (lockfile walking, store reading, serializers)
- `@pnpm/plugin-commands-sbom` — CLI plugin wiring

uses the lockfile walker for dependency traversal and reads package.json
from the CAFS store for license/author/description metadata. `--lockfile-only`
skips the store entirely for faster CI runs where metadata isn't needed.

validated against official CycloneDX 1.6 and SPDX 2.3 JSON schemas.

* chore: add sbom-related words to cspell dictionary

* fix(sbom): address CycloneDX review feedback and bump to 1.7

Implements all 5 items from the CycloneDX maintainer review:
split scoped names into group/name, move hashes to
externalReferences distribution, use license.id for known SPDX
identifiers, switch to modern tools.components structure with
pnpm version, and bump specVersion to 1.7.

Also adds spdx-license-ids for proper license classification and
improves SPDX serializer test coverage.

* fix(sbom): fix CI bundle failure for spdx-license-ids

createRequire doesn't work in the esbuild bundle since it's a runtime
resolve, switched back to regular import which esbuild can inline.

* fix(sbom): use tarball URL for distribution externalReferences

Use actual tarball download URL instead of PURL for CycloneDX
distribution externalReferences, per review feedback.

* feat(sbom): add CycloneDX metadata and improve SBOM quality scores

adds $schema, timestamp, lifecycles (build/pre-build) to CycloneDX output
to match what npm does. also enriches both CycloneDX and SPDX with
metadata.authors, metadata.supplier, component supplier from author,
vcs externalReferences from repository, and root component details
(purl, license, description, author, vcs). SPDX now uses tarball URL
for downloadLocation instead of NOASSERTION.

renames CycloneDxToolInfo to CycloneDxOptions, passes lockfileOnly
through to the serializer for lifecycle phase selection. adds store-dir
to accepted CLI options.

* fix(sbom): address CycloneDX review feedback round 2

switches license classification from spdx-license-ids to
@cyclonedx/cyclonedx-library (SPDX.isSupportedSpdxId) for accurate
CycloneDX license ID validation per jkowalleck's feedback.

removes hardcoded metadata.authors and metadata.supplier — these are
not appropriate for a tool to set. adds --sbom-authors and
--sbom-supplier CLI flags so the SBOM consumer (e.g. ACME Corp) can
declare who they are.

removes supplier from components — supplier is the registry/distributor,
not the package author. also fixes distribution externalReference to
only emit when a real tarball URL exists, no PURL fallback.

* fix(sbom): use sub-path import for CycloneDX library to fix bundle

top-level import from @cyclonedx/cyclonedx-library drags in
validation/serialize layers with optional deps (ajv-formats, libxmljs2,
xmlbuilder2) that esbuild can't resolve during pnpm CLI bundling.

switch to @cyclonedx/cyclonedx-library/SPDX which only pulls in the
SPDX module we actually use — pure JS, no optional deps.

* chore: update manifests

* refactor: extract shared store-reading logic into @pnpm/store.pkg-finder

Both @pnpm/license-scanner and @pnpm/sbom independently implemented
nearly identical logic to read a package's file index from the
content-addressable store. This extracts that into a new shared package
that returns a uniform Map<string, string> (filename → absolute path),
simplifying both consumers.

Close #9088

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
2026-02-25 08:45:21 +01:00
Zoltan Kochan
50fbecae7d refactor(env): pnpm env use now delegates to pnpm add --global (#10666)
This PR overhauls `pnpm env` use to route through pnpm's own install machinery instead of maintaining a parallel code path with manual symlink/shim/hardlink logic.

```
pnpm env use -g <version>
```

now runs:

```
pnpm add --global node@runtime:<version>
```

via `@pnpm/exec.pnpm-cli-runner`. All manual symlink, hardlink, and cmd-shim code in `envUse.ts` is gone (~1000 lines removed across the package).

### Changes

**npm and npx shims on all platforms**

Added `getNodeBinsForCurrentOS(platform)` to `@pnpm/constants`, returning a `Record<string, string>` with the correct relative paths for `node`, `npm`, and `npx` inside a Node.js distribution. `BinaryResolution.bin` is widened from `string` to `string | Record<string, string>` in `@pnpm/resolver-base` and `@pnpm/lockfile.types`, so the node resolver can set all three entries and pnpm's bin-linker creates shims for each automatically.

**Windows npm/npx fix**

`addFilesFromDir` was skipping root-level `node_modules/` (to avoid storing a package's own dependencies), which stripped the bundled `npm` from Node.js Windows zip archives. Added an `includeNodeModules` option and enabled it from the binary fetcher so Windows distributions keep their full contents.

**Removed subcommands**

`pnpm env add` and `pnpm env remove` are removed. `pnpm env use` handles both installing and activating a version. `pnpm env list` now always lists remote versions (the `--remote` flag is no longer required, though it is kept for backwards compatibility).

**musl support**

On Alpine Linux and other musl-based systems, the musl variant of Node.js is automatically downloaded from [unofficial-builds.nodejs.org](https://unofficial-builds.nodejs.org).
2026-02-22 12:06:34 +01:00
Zoltan Kochan
e18a879d72 feat!: drop Node.js 22.12 support 2026-02-18 14:54:09 +01:00
Zoltan Kochan
56a59df674 perf: persist bundled manifest in store index to avoid reading package.json from CAFS (#10473)
close #10461
2026-02-17 12:03:08 +01:00
Zoltan Kochan
1b4df57a01 feat!: drop Node.js 20 and 21 support (#10569) 2026-02-08 19:16:24 +01:00
이종혁
5f5f1db2b8 fix(plugin-commands-store): support relative storeDir in store commands (#10490)
close #10290

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
2026-02-06 16:31:47 +01:00
Zoltan Kochan
57e99b6996 chore: update pnpm-lock.yaml (#10558)
---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-02-06 03:28:39 +01:00
Zoltan Kochan
e2e0a321b3 perf: optimize how the integrities of files in the CAFS are stored (#10504) 2026-01-24 21:41:11 +01:00
Zoltan Kochan
40b107efa7 perf: migrate internal cache and index files to MessagePack serialization (#10500) 2026-01-23 01:31:09 +01:00
Brandon Cheng
d85ea8d817 fix: pass storeDir to createClient to fix resolution skipping (#10502) 2026-01-22 23:51:38 +01:00
Zoltan Kochan
2ea64631eb fix: skip symlinks pointing outside package root in git and file deps (#10493) 2026-01-21 15:45:56 +01:00
Zoltan Kochan
e3b35b6f37 style: update eslint to v9 (#10474) 2026-01-17 12:01:23 +01:00
Zoltan Kochan
caabba44ff fix: normalize Windows backslash path traversal attempts in tarball entry filenames
to prevent security vulnerabilities.
2026-01-15 17:10:17 +01:00
Zoltan Kochan
9f2b622d10 refactor: rename customFetcherHooks to customFetchers 2026-01-15 12:02:06 +01:00
Zoltan Kochan
5beece9615 feat!: remove old API for custom fetchers (#10464) 2026-01-15 11:57:48 +01:00
Zoltan Kochan
a8fe2d5298 feat!: remove the server command (#10463) 2026-01-15 11:32:07 +01:00
Zoltan Kochan
a00f9e515c chore: use typescript-go (#10452) 2026-01-14 01:18:13 +01:00
Zoltan Kochan
da112f7cb2 revert: "perf: use v8 serialize/deserialize instead of JSON (#9971)" (#10420)
close #10409
2026-01-13 15:16:33 +01:00
Zoltan Kochan
8a8a51c394 perf: don't calculate package file paths in the store twice (#10428) 2026-01-12 15:58:25 +01:00
Zoltan Kochan
0bcbaf9994 refactor: move out skip resolution logic from package requester (#10439) 2026-01-12 13:08:50 +01:00
Zoltan Kochan
5a0ed1d450 fix: prevent circular symlinks in projects registry (#10432)
close #10411
2026-01-09 18:04:57 +01:00