mirror of
https://github.com/pnpm/pnpm.git
synced 2026-06-28 09:55:39 -04:00
d91e2edbec23df6ba36ee442d588d366149fdb91
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Victor Sumner
|
61810aa684 |
feat: add --frozen-store for installs against a read-only store (#12190)
## What
Adds an opt-in `frozenStore` / `--frozen-store` setting (default `false`) that lets `pnpm install --offline --frozen-lockfile` run against a package store that lives on a **read-only filesystem** — a Nix store, a read-only bind mount, an OCI layer.
## Why
A normal install fails against such a store **not** because it writes package content, but because it unconditionally:
1. opens the SQLite `index.db` in WAL mode, which needs to create `-shm`/`-wal` sidecars in the store directory; and
2. writes a project-registry entry under the store.
Both fail with `attempt to write a readonly database` / `EROFS` on a read-only store directory, even when the store is complete and the lockfile is frozen. This blocks any deployment that wants an immutable, content-addressed store.
## How
When `frozenStore` is enabled, pnpm opens `index.db` through the SQLite **`immutable=1`** URI — which tells SQLite the file cannot change underneath it, so it bypasses the WAL/`-shm` sidecar machinery entirely and reads the raw file with zero sidecar creation — and suppresses every store-write path. Pair it with `--offline --frozen-lockfile` against a fully-populated store. It is incompatible with two settings that would write into the store, and each throws a clear config-conflict error before any network or store access: **`--force`** (which bypasses the no-write-on-hit skip → `ERR_PNPM_CONFIG_CONFLICT_FROZEN_STORE_WITH_FORCE`) and a configured **pnpr server** (which would fetch and write packages into the store → `ERR_PNPM_FROZEN_STORE_INCOMPATIBLE_WITH_PNPR`).
> Plain `SQLITE_OPEN_READ_ONLY` is **not** sufficient: opening a WAL-mode db read-only still tries to create the `-shm` sidecar, which fails on a read-only *directory*. `immutable=1` is the load-bearing piece.
> **Node.js requirement:** `node:sqlite` only passes `SQLITE_OPEN_URI` to SQLite (so the `immutable=1` query is honored rather than treated as part of a literal filename) starting in **v22.15.0** (22.x line), **v23.11.0**, and every **v24+**. pnpm's `engines` floor is `>=22.13`, so on a runtime older than that the frozen open is detected up front and fails with a clear `ERR_PNPM_FROZEN_STORE_UNSUPPORTED_NODE` instead of SQLite's cryptic "unable to open database file". (pacquet uses rusqlite with an explicit `SQLITE_OPEN_URI` flag, so it has no such floor.)
> The `immutable=1` URI path is also percent-encoded (`%`→`%25`, `?`→`%3f`, `#`→`%23`, in that order, leaving `/` literal) so a store path containing those characters doesn't truncate the path or inject a spurious query parameter — applied identically in both stacks.
### Build backstop under the global virtual store
Under the global virtual store (default), a package's directory lives **inside** the store (`{storeDir}/links/...`). Applying a patch or running an allowlisted lifecycle script writes into that directory — so on a frozen store it would crash mid-build with a raw `EROFS`. A fully-seeded store never reaches the build step (patched/built packages are imported from the side-effects cache and filtered out by the `isBuilt` gate), so any residual build candidate means the seed is **missing that package's build output**.
`buildModules` now refuses up front with `ERR_PNPM_FROZEN_STORE_NEEDS_BUILD` and actionable guidance ("rebuild the seed with their scripts enabled, or remove them from `onlyBuiltDependencies`") instead of failing cryptically once a script starts. The check is gated on the global virtual store — under the isolated linker, slot directories live in the writable project-local store, so builds there are fine. Non-allowlisted scripts never run, so they are not treated as a blocking write.
Bin-linking has its own read-only-store edge under the global virtual store. On a **warm** checkout (the project's `.bin/<name>` already points at the seed target) `linkBin` returns before touching the store, so it is write-free. But on a **fresh** checkout it (re)creates the bin and calls `fixBin`, whose `chmod` targets the bin's **source file inside the store** (`{storeDir}/links/...`) — which is refused with `EPERM`/`EACCES` on a read-only store, even though a complete seed already ships that bin executable (so the `chmod` is redundant). `@pnpm/bins.linker` now wraps that call in `ensureExecutable`: it swallows the refusal when the target is already executable and rethrows otherwise, so bin-linking is write-free against a frozen store on a cold checkout too, while a genuinely non-executable bin (a broken seed) still surfaces as an error.
The blocking predicate distinguishes the two write kinds: a **patch** is applied regardless of `ignoreScripts`, so a patched package is always blocked; a **lifecycle script** is suppressed under `ignoreScripts`, so an allowlisted build-requiring package is *not* blocked when scripts are off (it would write nothing). This avoids falsely rejecting a valid `--ignore-scripts` frozen install. **Optional dependencies are exempt**: a build or patch failure on an optional dependency is non-fatal at runtime, so a seed missing an optional package's build output skips that build (emitting the `skipped-optional-dependency` log) instead of blocking the install — in both stacks.
### Both stacks (parity rule)
**TypeScript pnpm CLI**
- Config plumbing (`@pnpm/config.reader`): `frozen-store` type, config-file key, default, `Config.frozenStore`.
- The read-only open branch (`@pnpm/store.index`): `immutable=1`, read-only statements, throwing mutators.
- Wiring through `@pnpm/store.controller` and `@pnpm/store.connection-manager` to the sole `StoreIndex` construction site.
- Gating the project-registry write (`@pnpm/installing.context`).
- The `--force` / pnpr-server conflict guards (`@pnpm/installing.deps-installer`) and CLI surface (`@pnpm/installing.commands`).
- **After-install rebuild** (`@pnpm/building.after-install`): the post-install rebuild opens its `StoreIndex` immutably under the flag, so re-reading the store for a rebuild never attempts a writable open against the frozen store.
- **Worker fix:** `@pnpm/worker` opens its *own* writable `StoreIndex` on every `readPkgFromCafs` cache hit, so a pure read crashed on a frozen store. `frozenStore` is threaded through to `getStoreIndex` and keyed into its connection cache.
- **Build backstop** (`@pnpm/building.during-install`): `buildModules` throws `ERR_PNPM_FROZEN_STORE_NEEDS_BUILD` for a GVS slot that would build/patch on a frozen store, honoring `ignoreScripts`; threaded from `@pnpm/installing.deps-installer`.
- **Bin-linking on a read-only store** (`@pnpm/bins.linker`): `linkBin` wraps the `fixBin` chmod in `ensureExecutable`, which tolerates `EPERM`/`EACCES` when the bin's store-resident source is already executable (a complete seed) and rethrows otherwise — so a fresh checkout against a frozen store links bins without crashing on the redundant chmod. Catch-on-failure keeps the writable hot path at zero added syscalls.
**Rust pacquet**
- A dedicated `open_immutable` / `shared_immutable_in` opens via `immutable=1`, selected only under the flag. Plain `open_readonly` keeps the ordinary `SQLITE_OPEN_READ_ONLY` open (WAL locking intact) because normal installs read the index while the same process's `StoreIndexWriter` writes it concurrently — an immutable connection skips all locking and change detection, so a concurrent writer would make those reads undefined.
- `--frozen-store` CLI flag + `frozenStore` workspace-yaml setting.
- The store-index writer is replaced with a drain-and-drop stub (`spawn_disabled`) and `init_store_dir_best_effort` is skipped under the flag.
- **Build backstop:** `build_modules` returns `BuildModulesError::FrozenStoreNeedsBuild` (`ERR_PNPM_FROZEN_STORE_NEEDS_BUILD`) under the same GVS + frozen-store condition, threaded from `config.frozen_store`. The gate keys off `should_run_scripts` (which already folds the allow-build policy), so it is correct without an explicit ignore-scripts branch — pacquet has no configurable ignore-scripts mode yet.
pacquet already separated read-only index access (`shared_readonly_in`, or `shared_immutable_in` under the flag) from writes, so it never had the worker-conflation bug; the flag makes the "no writes attempted" contract explicit and gates the remaining best-effort write attempts.
## Testing
- **TS:** `store.index` frozen-mode-on-`0555`-directory test (reads work, writes throw `ERR_PNPM_FROZEN_STORE_WRITE`) plus a path-with-`?` open test — both gated on the runtime's immutable-URI support, with a complementary test asserting `ERR_PNPM_FROZEN_STORE_UNSUPPORTED_NODE` fires where that support is absent (the CI Node 22.13.0 path); `config.reader` round-trip; `deps-installer` `--force` and pnpr-server conflict guards; `worker`/`package-requester` unit tests. End-to-end on a `chmod -R 0555` store: install succeeds, `node_modules` materializes, no `-shm`/`-wal`/`-journal` sidecars; negative control without the flag fails as expected; incomplete store → clean offline error.
- **pacquet:** `open_immutable_reads_wal_db_on_readonly_directory` unit test plus the `immutable_sqlite_uri` encoding test and a path-with-`?` open test; yaml + CLI fold tests; **integration test** `frozen_store_installs_against_a_read_only_store` — primes a store, `chmod 0555` the tree, runs `install --frozen-lockfile --frozen-store --offline`, asserts success + materialized `node_modules` + zero sidecars. Confirmed load-bearing by reverting the `immutable=1` fix (test then fails).
- **Build backstop (both stacks):** `building/during-install` unit tests — approved-build-not-cached and patched-not-cached refuse; cached, non-allowlisted, and non-GVS cases pass through; **approved-build-under-`ignoreScripts` passes through while patched-under-`ignoreScripts` still refuses** — and the matching pacquet `build_modules` tests (`frozen_store_gvs_patch_not_seeded_refuses` + GVS-off / frozen-off controls). Each confirmed load-bearing by disabling the relevant guard and watching the corresponding test fail.
- **Bin-linking (`bins/linker`):** `ensureExecutable` tests with `fixBin` mocked to reject with `EPERM` — an already-executable bin source resolves (and `fixBin` is asserted called, so it isn't the warm skip-guard passing), a non-executable one rethrows `EPERM`. Confirmed end-to-end by running the built `linkBins` against a real `chflags uchg`-immutable store: the executable-seed case resolves and links the bin, the non-executable-seed control throws `EPERM`.
A changeset is included with `"pnpm": minor` and `"@pnpm/bins.linker": patch` (the read-only-store bin-linking fix).
|
||
|
Zoltan Kochan
|
a33c4bfcb0 |
perf: skip resolution when only pnpm-lock.yaml is missing (pnpm + pacquet) (#12004)
* perf: skip resolution when only pnpm-lock.yaml is missing
When pnpm-lock.yaml is absent but node_modules/.pnpm/lock.yaml exists and still
satisfies the manifest, reuse the materialized snapshot to regenerate the
wanted lockfile instead of walking the registry to rebuild it. Closes the
cache+node_modules variation gap in the vlt.sh benchmarks for the pnpm CLI
side; the pacquet port is tracked separately at #11993.
`--frozen-lockfile` still fails when pnpm-lock.yaml is absent: the regenerated
file must be committed, so failing loudly is the correct behavior for CI.
* perf(pacquet): port the cache+node_modules shortcut
When `pnpm-lock.yaml` is absent but `node_modules/.pnpm/lock.yaml` exists
and still satisfies the manifest, synthesize the wanted lockfile from the
materialized snapshot and take the frozen-install path. The install skips
resolution and regenerates `pnpm-lock.yaml` from the synthesized object.
Mirrors the pnpm-side change at
|
||
|
Zoltan Kochan
|
b5d93c6ba9 |
refactor(config): remove rawLocalConfig and force* hoist flags (#11199)
rawLocalConfig detected whether hoist settings were explicitly set. In v11, config values are always authoritative. - Remove rawLocalConfig from ConfigContext, config reader, inheritPickedConfig, UniversalOptions - Remove forceHoistPattern, forcePublicHoistPattern, forceShamefullyHoist — validateModules always checks now - Simplify save-workspace-protocol check - Remove dead rawLocalConfig overrides in deploy/patchCommit |
||
|
Zoltan Kochan
|
0d88df854f |
chore: update all dependencies to latest versions (#11032)
* chore: update all dependencies to latest versions Update all outdated dependencies across the monorepo catalog and fix breaking changes from major version bumps. Notable updates: - ESLint 9 → 10 (fix custom rule API, disable new no-useless-assignment) - @stylistic/eslint-plugin 4 → 5 (auto-fixed indent changes) - @cyclonedx/cyclonedx-library 9 → 10 (adapt to removed SPDX API) - esbuild 0.25 → 0.27 - TypeScript 5.9.2 → 5.9.3 - Various @types packages, test utilities, and build tools Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update unified/remark/mdast imports for v11/v4 API changes Update imports in get-release-text for the new ESM named exports: - mdast-util-to-string: default → { toString } - unified: default → { unified } Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: resolve typecheck errors from dependency updates - isexe v4: use named import { sync } instead of default export - remark-parse/remark-stringify v11: add vfile as packageExtension dependency so TypeScript can resolve type declarations - get-release-text: remove unused @ts-expect-error directives Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert runtime dependency major version bumps Revert major version bumps for runtime dependencies that are bundled into pnpm to fix test failures where pnpm add silently fails: - bin-links: keep ^5.0.0 (was ^6.0.0) - cli-truncate: keep ^4.0.0 (was ^5.2.0) - delay: keep ^6.0.0 (was ^7.0.0) - filenamify: keep ^6.0.0 (was ^7.0.1) - find-up: keep ^7.0.0 (was ^8.0.0) - isexe: keep 2.0.0 (was 4.0.0) - normalize-newline: keep 4.1.0 (was 5.0.0) - p-queue: keep ^8.1.0 (was ^9.1.0) - ps-list: keep ^8.1.1 (was ^9.0.0) - string-length: keep ^6.0.0 (was ^7.0.1) - symlink-dir: keep ^7.0.0 (was ^9.0.0) - terminal-link: keep ^4.0.0 (was ^5.0.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restore runtime dependency major version bumps Re-apply all runtime dependency major version bumps that were previously reverted. All packages maintain their default exports except isexe v4 which needs named imports. Updated runtime deps: - bin-links: ^5.0.0 → ^6.0.0 - cli-truncate: ^4.0.0 → ^5.2.0 - delay: ^6.0.0 → ^7.0.0 - filenamify: ^6.0.0 → ^7.0.1 - find-up: ^7.0.0 → ^8.0.0 - isexe: 2.0.0 → 4.0.0 (fix: use named import { sync }) - normalize-newline: 4.1.0 → 5.0.0 - p-queue: ^8.1.0 → ^9.1.0 - ps-list: ^8.1.1 → ^9.0.0 - string-length: ^6.0.0 → ^7.0.1 - symlink-dir: ^7.0.0 → ^9.0.0 - terminal-link: ^4.0.0 → ^5.0.0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert tempy to 3.0.0 to fix bundle hang tempy 3.2.0 pulls in temp-dir 3.0.0 which uses async fs.realpath() inside its module init. When bundled by esbuild into the __esm lazy init pattern, this causes a deadlock during module initialization, making the pnpm binary hang silently on startup. Keeping tempy at 3.0.0 which uses temp-dir 2.x (sync fs.realpathSync). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add comment explaining why tempy cannot be upgraded Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert nock to 13.3.4 for node-fetch compatibility nock 14 changed its HTTP interception mechanism in a way that doesn't properly intercept node-fetch requests, causing audit tests to hang waiting for responses that are never intercepted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add comment explaining why nock cannot be upgraded Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update symlink-dir imports for v10 ESM named exports symlink-dir v10 removed the default export and switched to named exports: { symlinkDir, symlinkDirSync }. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert @typescript/native-preview to working version Newer tsgo dev builds (>= 20260318) have a regression where @types/node cannot be resolved, breaking all node built-in types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: vulnerabilities * fix: align comment indentation in runLifecycleHook Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: pin msgpackr to 1.11.8 for TypeScript 5.9 compatibility msgpackr 1.11.9 has broken type definitions that use Iterable/Iterator without required type arguments, causing compile errors with TS 5.9. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Zoltan Kochan
|
4a36b9a110 |
refactor: rename internal packages to @pnpm/<domain>.<leaf> convention (#10997)
## Summary Rename all internal packages so their npm names follow the `@pnpm/<domain>.<leaf>` convention, matching their directory structure. Also rename directories to remove redundancy and improve clarity. ### Bulk rename (94 packages) All `@pnpm/` packages now derive their name from their directory path using dot-separated segments. Exceptions: `packages/`, `__utils__/`, and `pnpm/artifacts/` keep leaf names only. ### Directory renames (removing redundant prefixes) - `cli/cli-meta` → `cli/meta`, `cli/cli-utils` → `cli/utils` - `config/config` → `config/reader`, `config/config-writer` → `config/writer` - `fetching/fetching-types` → `fetching/types` - `lockfile/lockfile-to-pnp` → `lockfile/to-pnp` - `store/store-connection-manager` → `store/connection-manager` - `store/store-controller-types` → `store/controller-types` - `store/store-path` → `store/path` ### Targeted renames (clarity improvements) - `deps/dependency-path` → `deps/path` (`@pnpm/deps.path`) - `deps/calc-dep-state` → `deps/graph-hasher` (`@pnpm/deps.graph-hasher`) - `deps/inspection/dependencies-hierarchy` → `deps/inspection/tree-builder` (`@pnpm/deps.inspection.tree-builder`) - `bins/link-bins` → `bins/linker`, `bins/remove-bins` → `bins/remover`, `bins/package-bins` → `bins/resolver` - `installing/get-context` → `installing/context` - `store/package-store` → `store/controller` - `pkg-manifest/manifest-utils` → `pkg-manifest/utils` ### Manifest reader/writer renames - `workspace/read-project-manifest` → `workspace/project-manifest-reader` (`@pnpm/workspace.project-manifest-reader`) - `workspace/write-project-manifest` → `workspace/project-manifest-writer` (`@pnpm/workspace.project-manifest-writer`) - `workspace/read-manifest` → `workspace/workspace-manifest-reader` (`@pnpm/workspace.workspace-manifest-reader`) - `workspace/manifest-writer` → `workspace/workspace-manifest-writer` (`@pnpm/workspace.workspace-manifest-writer`) ### Workspace package renames - `workspace/find-packages` → `workspace/projects-reader` - `workspace/find-workspace-dir` → `workspace/root-finder` - `workspace/resolve-workspace-range` → `workspace/range-resolver` - `workspace/filter-packages-from-dir` merged into `workspace/filter-workspace-packages` → `workspace/projects-filter` ### Domain moves - `pkg-manifest/read-project-manifest` → `workspace/project-manifest-reader` - `pkg-manifest/write-project-manifest` → `workspace/project-manifest-writer` - `pkg-manifest/exportable-manifest` → `releasing/exportable-manifest` ### Scope - 1206 files changed - Updated: package.json names/deps, TypeScript imports, tsconfig references, changeset files, renovate.json, test fixtures, import ordering |