mirror of
https://github.com/pnpm/pnpm.git
synced 2026-07-22 05:32:44 -04:00
native-binary-deps
146 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Zoltan Kochan
|
9dd6fbf4d8 | docs: reword native binary comment to satisfy cspell | ||
|
Zoltan Kochan
|
c8f0dc36cf | style: rename libcs and avoid %2F paths to satisfy cspell | ||
|
Zoltan Kochan
|
16f51b11f7 |
feat(npm-resolver): synthesize native bin dependencies into platform variants
A package that ships a JS launcher shim plus per-platform native binaries as optionalDependencies (e.g. `pacquet`) otherwise installs the wrapper + every optional dep and links a Node launcher shim that re-spawns the real binary. When such a package is a native bin dependency, the npm resolver now returns a `variations` resolution over its platform packages instead of the wrapper tarball, so only the host's binary is fetched and linked directly — no launcher shim, no lifecycle scripts. The whole downstream variants pipeline (selection, BinaryFetcher, bin-linking) is reused. The set of native bin dependencies is the hardcoded default list (`pacquet`, `@pnpm/pacquet`) plus the new `nativeBinDependencies` setting. Adds the `nativeBinDependencies` config setting and the synthesis in `resolveNativeBinVariations`, gated so a wrapper with no platform-tagged optional dependency falls back to its normal tarball resolution. |
||
|
Zoltan Kochan
|
29ab905c21 |
fix: preserve catalog version range policy on update (#12416)
A named catalog whose name parses as a version (e.g. catalog:express4-21) had its range policy overridden by pnpm update because whichVersionIsPinned misread the catalog: reference in the previous specifier as a pinned version. The catalog reference carries no pinning of its own, so the prefix from the catalog entry is now preserved. Closes https://github.com/pnpm/pnpm/issues/10321 |
||
|
Zoltan Kochan
|
681b593eb2 |
fix: support scope-specific registry auth tokens (#12392)
pnpm can now use different auth tokens for different package scopes, even when those scopes use the same registry URL. Previously, auth was selected only by registry URL. If `@org-a` and `@org-b` both used `https://npm.pkg.github.com/`, they had to share the same token. This caused problems for registries that issue tokens per organization or per scope. Configure a scope-specific token by adding the package scope after the registry URL in the auth key: ```ini @org-a:registry=https://npm.pkg.github.com/ @org-b:registry=https://npm.pkg.github.com/ //npm.pkg.github.com/:@org-a:_authToken=${ORG_A_TOKEN} //npm.pkg.github.com/:@org-b:_authToken=${ORG_B_TOKEN} //npm.pkg.github.com/:_authToken=${FALLBACK_TOKEN} ``` `pnpm login --registry=https://npm.pkg.github.com --scope=@org-a` writes the token to the same scope-specific auth key. When installing or publishing `@org-a/*`, pnpm uses `ORG_A_TOKEN`. For `@org-b/*`, pnpm uses `ORG_B_TOKEN`. Packages without a matching scope continue to use the registry-wide fallback token. |
||
|
Zoltan Kochan
|
1310ab53c4 |
perf: close repeat-install and warm-resolve gaps (lazy lockfile, pre-runtime fast path, 304 freshness renewal) (#12364)
* perf(cli): keep the repeat-install fast path off the lockfile parse and thread spawns The "Already up to date" short-circuit decides from manifest mtimes alone (mirroring upstream checkDepsStatus, which never reads the wanted lockfile), yet pacquet parsed pnpm-lock.yaml eagerly in State::init before the check ran — a multi-millisecond YAML parse on every no-op install, scaling with lockfile size (babylon's 720 KB lockfile dominated its repeat-install wall time). pnpm-lock.yaml now loads through a LazyLockfile (OnceLock-backed) that install forces only after the fast path has passed on the run; add / update / remove / outdated / the pnpr path force it up front, keeping their behavior unchanged. The repeat-install regenerate branch probes for the file's existence instead of its parsed contents, so the fast path stays mtime-cheap. The rayon global pool is likewise no longer built eagerly at startup: the worker count is published via RAYON_NUM_THREADS (set in fn main while the process is still single-threaded) so the pool spawns lazily on first parallel use — commands that never reach a parallel phase no longer pay 2×CPUs thread spawns. A corrupt lockfile now surfaces its parse error when the install actually reads the file; an up-to-date project with an unreadable lockfile reports "Already up to date" exactly as pnpm does. * perf(cli): finish up-to-date installs before building the async runtime A plain `pacquet install` that is already up to date now completes on the main thread, before the tokio runtime, the rayon pool, the HTTP client, or any install state exists. The new `install_already_up_to_date` twin of the repeat-install short-circuit reuses the exact same workspace discovery and `check_optimistic_repeat_install` inputs as `Install::run`, and the CLI invokes it from the (now synchronous) `main` after clap parsing. Gates mirror everything that would make the full path behave differently: `--frozen-lockfile` / `--lockfile-only`, a configured pnpr server (that path never runs the optimistic check), `--recursive` / `--filter`, config dependencies, and pnpmfile updateConfig hooks (both can mutate the config the check compares against). Any gate or error falls through to the full install path, which re-runs the check and reproduces failures with their established error shape; the "Already up to date" + summary emissions are byte-identical. Repeat-install instruction count on the vue fixture drops from ~203M to ~41M retired instructions — within a rounding error of the `pacquet --version` floor (~39M). * perf(resolving): renew metadata-mirror freshness on 304 Not Modified The minimumReleaseAge freshness shortcut treats a metadata mirror younger than the cutoff as authoritative and resolves without touching the network. But a 304 revalidation never rewrote the mirror file, so its mtime froze at the last 200 response: once a cached packument grew older than minimumReleaseAge (24h by default), every subsequent install re-validated every package against the registry, forever. A 304 proves the cached packument equals the registry's current document, so the validation clock legitimately restarts at the response: bump the mirror's mtime to now (fire-and-forget — a read-only cache dir only costs the next install another conditional request). Applied to both stacks: pnpm's pickPackage notModified branch and pacquet's fetch_full_metadata_cached 304 path. On a vue-fixture install with a stale cache, the second warm resolve drops from ~2s (520 conditional requests) to ~250ms (zero requests). * style(resolving): use clippy's preferred Duration units in the 304 mtime test CI clippy denies duration_suboptimal_units; from_hours / from_mins replace the hand-multiplied from_secs values. * style(package-manager): use clippy's preferred Duration unit in the sync fast-path test Same duration_suboptimal_units deny as the previous commit, one site CI's clippy surfaced after it stopped at the first failing crate. * fix(lockfile): address review — dir-addressed LazyLockfile, read-open 304 touch LazyLockfile resolved pnpm-lock.yaml against the process cwd while the CLI honours a canonicalized --dir without chdir, so the deferred load and the existence probe could consult a different lockfile than the rest of the install (which derives lockfile_path from the manifest's directory). The lazy handle now carries the manifest's directory and loads via load_wanted_from_dir; lockfile-disabled config gets an explicit disabled() constructor. The pre-runtime fast path builds its handle from the same directory, keeping verdict parity with Install::run. renew_mirror_freshness opened the mirror with append just to bump the mtime; set_modified only needs a file handle plus ownership (futimens semantics), so a plain read-open also covers mirrors whose mode dropped write permission. * test(integrated-benchmark): compare best-of-N samples in the slow-start proxy test The test raced two single wall-clock samples, and a loaded CI runner can inflate the ramped-vs-flat comparison in either direction (macOS runner measured flat 318ms vs ramped 304ms against a ~66ms model). Scheduler stalls only ever inflate a sample, while slow start's ramp overhead is structural and survives in every sample — so the minimum of several runs per side is the noise-resistant estimator. * chore(deps): bump esbuild to 0.28.1 to clear GHSA-gv7w-rqvm-qjhr The new advisory (install-module RCE via NPM_CONFIG_REGISTRY, patched in 0.28.1) fails the audit gate. 0.28.1 was published within the minimumReleaseAge window, so the patched version is excluded from the age gate — the same mechanism pnpm audit --fix uses — including its '@esbuild/*' platform packages, whose versions move in lockstep with the root package. * fix(lockfile): make the unloaded presence probe match the loader's absence rules The loader treats an empty file and an env-only combined document as an absent wanted lockfile (Ok(None)), but is_loaded_or_on_disk probed bare file existence, so the repeat-install path could skip restoring a semantically-missing pnpm-lock.yaml. The probe now reads the file and checks the main document is non-empty (Lockfile::wanted_exists_in_dir) — the loader's exact absence rules, still without the YAML parse. * fix(cli): build the rayon pool after the fast-path gate instead of injecting env Publishing the worker count through RAYON_NUM_THREADS leaked the variable into every child process the install spawns — lifecycle scripts, node probes, git — and pnpm exposes no such variable to scripts. Build the global pool with ThreadPoolBuilder again, but only once the repeat-install fast path has declined: real installs pay exactly the cost they always did, the no-op path still spawns no workers, and the process environment stays untouched (which also drops the unsafe set_var and its single-threaded contract). * fix(lockfile): treat only NotFound as absence in the presence probe A permission or I/O failure reading pnpm-lock.yaml reported the file as absent, which would send the repeat-install path into the regenerate-on-missing branch — overwriting an existing lockfile it merely could not read. Only NotFound counts as absent now; any other read failure reports presence, and the real load surfaces the underlying error when the contents are actually needed. * fix(resolving): open the mirror write-capable for the 304 touch, read-only as fallback Windows' set_modified requires write-attributes access on the handle, so the read-only open silently failed there (caught by the Windows CI run of a_304_renews_the_mirror_mtime). Append-mode open carries that access; the read-only fallback still covers Unix mirrors whose mode dropped write permission, where timestamp syscalls need ownership rather than write access. * fix(package-manager): never short-circuit partial installs as already up to date add and remove mutate the manifest in memory and persist it only after Install::run returns, so the on-disk mtimes the optimistic repeat-install check reads still describe the pre-mutation project. With a fresh workspace state, `pacquet add X` right after a clean install reported "Already up to date", skipped the entire install, and then saved a package.json declaring a dependency that was never resolved, lockfiled, or materialized (self-healing on the next run, which sees the newer manifest mtime). Gate the short-circuit on is_full_install, mirroring upstream installDeps calling checkDepsStatus only for the plain-install mutation, never for installSome / uninstallSome. The new partial_install_disables_optimistic_short_circuit test fails without the gate. The bug predates this PR (the KeepAll gate has carried add since the optimistic path landed) — surfaced by CodeRabbit review on pnpm/pnpm#12364. |
||
|
Victor Sumner
|
61810aa684 |
feat: add --frozen-store for installs against a read-only store (#12190)
## What
Adds an opt-in `frozenStore` / `--frozen-store` setting (default `false`) that lets `pnpm install --offline --frozen-lockfile` run against a package store that lives on a **read-only filesystem** — a Nix store, a read-only bind mount, an OCI layer.
## Why
A normal install fails against such a store **not** because it writes package content, but because it unconditionally:
1. opens the SQLite `index.db` in WAL mode, which needs to create `-shm`/`-wal` sidecars in the store directory; and
2. writes a project-registry entry under the store.
Both fail with `attempt to write a readonly database` / `EROFS` on a read-only store directory, even when the store is complete and the lockfile is frozen. This blocks any deployment that wants an immutable, content-addressed store.
## How
When `frozenStore` is enabled, pnpm opens `index.db` through the SQLite **`immutable=1`** URI — which tells SQLite the file cannot change underneath it, so it bypasses the WAL/`-shm` sidecar machinery entirely and reads the raw file with zero sidecar creation — and suppresses every store-write path. Pair it with `--offline --frozen-lockfile` against a fully-populated store. It is incompatible with two settings that would write into the store, and each throws a clear config-conflict error before any network or store access: **`--force`** (which bypasses the no-write-on-hit skip → `ERR_PNPM_CONFIG_CONFLICT_FROZEN_STORE_WITH_FORCE`) and a configured **pnpr server** (which would fetch and write packages into the store → `ERR_PNPM_FROZEN_STORE_INCOMPATIBLE_WITH_PNPR`).
> Plain `SQLITE_OPEN_READ_ONLY` is **not** sufficient: opening a WAL-mode db read-only still tries to create the `-shm` sidecar, which fails on a read-only *directory*. `immutable=1` is the load-bearing piece.
> **Node.js requirement:** `node:sqlite` only passes `SQLITE_OPEN_URI` to SQLite (so the `immutable=1` query is honored rather than treated as part of a literal filename) starting in **v22.15.0** (22.x line), **v23.11.0**, and every **v24+**. pnpm's `engines` floor is `>=22.13`, so on a runtime older than that the frozen open is detected up front and fails with a clear `ERR_PNPM_FROZEN_STORE_UNSUPPORTED_NODE` instead of SQLite's cryptic "unable to open database file". (pacquet uses rusqlite with an explicit `SQLITE_OPEN_URI` flag, so it has no such floor.)
> The `immutable=1` URI path is also percent-encoded (`%`→`%25`, `?`→`%3f`, `#`→`%23`, in that order, leaving `/` literal) so a store path containing those characters doesn't truncate the path or inject a spurious query parameter — applied identically in both stacks.
### Build backstop under the global virtual store
Under the global virtual store (default), a package's directory lives **inside** the store (`{storeDir}/links/...`). Applying a patch or running an allowlisted lifecycle script writes into that directory — so on a frozen store it would crash mid-build with a raw `EROFS`. A fully-seeded store never reaches the build step (patched/built packages are imported from the side-effects cache and filtered out by the `isBuilt` gate), so any residual build candidate means the seed is **missing that package's build output**.
`buildModules` now refuses up front with `ERR_PNPM_FROZEN_STORE_NEEDS_BUILD` and actionable guidance ("rebuild the seed with their scripts enabled, or remove them from `onlyBuiltDependencies`") instead of failing cryptically once a script starts. The check is gated on the global virtual store — under the isolated linker, slot directories live in the writable project-local store, so builds there are fine. Non-allowlisted scripts never run, so they are not treated as a blocking write.
Bin-linking has its own read-only-store edge under the global virtual store. On a **warm** checkout (the project's `.bin/<name>` already points at the seed target) `linkBin` returns before touching the store, so it is write-free. But on a **fresh** checkout it (re)creates the bin and calls `fixBin`, whose `chmod` targets the bin's **source file inside the store** (`{storeDir}/links/...`) — which is refused with `EPERM`/`EACCES` on a read-only store, even though a complete seed already ships that bin executable (so the `chmod` is redundant). `@pnpm/bins.linker` now wraps that call in `ensureExecutable`: it swallows the refusal when the target is already executable and rethrows otherwise, so bin-linking is write-free against a frozen store on a cold checkout too, while a genuinely non-executable bin (a broken seed) still surfaces as an error.
The blocking predicate distinguishes the two write kinds: a **patch** is applied regardless of `ignoreScripts`, so a patched package is always blocked; a **lifecycle script** is suppressed under `ignoreScripts`, so an allowlisted build-requiring package is *not* blocked when scripts are off (it would write nothing). This avoids falsely rejecting a valid `--ignore-scripts` frozen install. **Optional dependencies are exempt**: a build or patch failure on an optional dependency is non-fatal at runtime, so a seed missing an optional package's build output skips that build (emitting the `skipped-optional-dependency` log) instead of blocking the install — in both stacks.
### Both stacks (parity rule)
**TypeScript pnpm CLI**
- Config plumbing (`@pnpm/config.reader`): `frozen-store` type, config-file key, default, `Config.frozenStore`.
- The read-only open branch (`@pnpm/store.index`): `immutable=1`, read-only statements, throwing mutators.
- Wiring through `@pnpm/store.controller` and `@pnpm/store.connection-manager` to the sole `StoreIndex` construction site.
- Gating the project-registry write (`@pnpm/installing.context`).
- The `--force` / pnpr-server conflict guards (`@pnpm/installing.deps-installer`) and CLI surface (`@pnpm/installing.commands`).
- **After-install rebuild** (`@pnpm/building.after-install`): the post-install rebuild opens its `StoreIndex` immutably under the flag, so re-reading the store for a rebuild never attempts a writable open against the frozen store.
- **Worker fix:** `@pnpm/worker` opens its *own* writable `StoreIndex` on every `readPkgFromCafs` cache hit, so a pure read crashed on a frozen store. `frozenStore` is threaded through to `getStoreIndex` and keyed into its connection cache.
- **Build backstop** (`@pnpm/building.during-install`): `buildModules` throws `ERR_PNPM_FROZEN_STORE_NEEDS_BUILD` for a GVS slot that would build/patch on a frozen store, honoring `ignoreScripts`; threaded from `@pnpm/installing.deps-installer`.
- **Bin-linking on a read-only store** (`@pnpm/bins.linker`): `linkBin` wraps the `fixBin` chmod in `ensureExecutable`, which tolerates `EPERM`/`EACCES` when the bin's store-resident source is already executable (a complete seed) and rethrows otherwise — so a fresh checkout against a frozen store links bins without crashing on the redundant chmod. Catch-on-failure keeps the writable hot path at zero added syscalls.
**Rust pacquet**
- A dedicated `open_immutable` / `shared_immutable_in` opens via `immutable=1`, selected only under the flag. Plain `open_readonly` keeps the ordinary `SQLITE_OPEN_READ_ONLY` open (WAL locking intact) because normal installs read the index while the same process's `StoreIndexWriter` writes it concurrently — an immutable connection skips all locking and change detection, so a concurrent writer would make those reads undefined.
- `--frozen-store` CLI flag + `frozenStore` workspace-yaml setting.
- The store-index writer is replaced with a drain-and-drop stub (`spawn_disabled`) and `init_store_dir_best_effort` is skipped under the flag.
- **Build backstop:** `build_modules` returns `BuildModulesError::FrozenStoreNeedsBuild` (`ERR_PNPM_FROZEN_STORE_NEEDS_BUILD`) under the same GVS + frozen-store condition, threaded from `config.frozen_store`. The gate keys off `should_run_scripts` (which already folds the allow-build policy), so it is correct without an explicit ignore-scripts branch — pacquet has no configurable ignore-scripts mode yet.
pacquet already separated read-only index access (`shared_readonly_in`, or `shared_immutable_in` under the flag) from writes, so it never had the worker-conflation bug; the flag makes the "no writes attempted" contract explicit and gates the remaining best-effort write attempts.
## Testing
- **TS:** `store.index` frozen-mode-on-`0555`-directory test (reads work, writes throw `ERR_PNPM_FROZEN_STORE_WRITE`) plus a path-with-`?` open test — both gated on the runtime's immutable-URI support, with a complementary test asserting `ERR_PNPM_FROZEN_STORE_UNSUPPORTED_NODE` fires where that support is absent (the CI Node 22.13.0 path); `config.reader` round-trip; `deps-installer` `--force` and pnpr-server conflict guards; `worker`/`package-requester` unit tests. End-to-end on a `chmod -R 0555` store: install succeeds, `node_modules` materializes, no `-shm`/`-wal`/`-journal` sidecars; negative control without the flag fails as expected; incomplete store → clean offline error.
- **pacquet:** `open_immutable_reads_wal_db_on_readonly_directory` unit test plus the `immutable_sqlite_uri` encoding test and a path-with-`?` open test; yaml + CLI fold tests; **integration test** `frozen_store_installs_against_a_read_only_store` — primes a store, `chmod 0555` the tree, runs `install --frozen-lockfile --frozen-store --offline`, asserts success + materialized `node_modules` + zero sidecars. Confirmed load-bearing by reverting the `immutable=1` fix (test then fails).
- **Build backstop (both stacks):** `building/during-install` unit tests — approved-build-not-cached and patched-not-cached refuse; cached, non-allowlisted, and non-GVS cases pass through; **approved-build-under-`ignoreScripts` passes through while patched-under-`ignoreScripts` still refuses** — and the matching pacquet `build_modules` tests (`frozen_store_gvs_patch_not_seeded_refuses` + GVS-off / frozen-off controls). Each confirmed load-bearing by disabling the relevant guard and watching the corresponding test fail.
- **Bin-linking (`bins/linker`):** `ensureExecutable` tests with `fixBin` mocked to reject with `EPERM` — an already-executable bin source resolves (and `fixBin` is asserted called, so it isn't the warm skip-guard passing), a non-executable one rethrows `EPERM`. Confirmed end-to-end by running the built `linkBins` against a real `chflags uchg`-immutable store: the executable-seed case resolves and links the bin, the non-executable-seed control throws `EPERM`.
A changeset is included with `"pnpm": minor` and `"@pnpm/bins.linker": patch` (the read-only-store bin-linking fix).
|
||
|
Zoltan Kochan
|
6d17b669b4 |
fix: verify lockfile tarball URL matches registry metadata (#12134)
## What The lockfile resolution verifier now confirms that a registry entry pinning an explicit `tarball` URL points at the artifact the registry's own metadata lists for that `name@version`. A mismatch — or any entry that can't be confirmed against the registry — is rejected with `ERR_PNPM_TARBALL_URL_MISMATCH`. ## Why Follow-up to the design discussion on #12122. The verifier checked the age/trust of `name@version` against the registry packument but never bound the lockfile's `tarball` URL to it. For the non-standard entries pnpm preserves a tarball URL for (npm Enterprise, GitHub Packages — see `toLockfileResolution`), pnpm fetches straight from that URL. So a **tampered lockfile could pair a trusted `name@version` with an attacker-chosen tarball URL** (plus a matching integrity for the attacker's bytes); verification passed against the legitimate version while the install fetched the attacker's bytes. Defending a checked-in lockfile is explicitly in this feature's threat model. ## How - For a registry-keyed entry that pins an explicit `tarball`, fetch the packument and assert the URL equals `versions[v].dist.tarball`. The comparison canonicalizes away benign differences — http/https scheme, default ports (`:443`/`:80`), and `%2f` scope-separator encoding (case-insensitive) — so only real mismatches are flagged. The packument is fetched from the user's configured registry (the lockfile's tarball host can't redirect it), and named-registry routing uses the same canonicalization so a scheme/`%2f`-only difference doesn't route to the wrong packument. - **The binding is unconditional.** It runs regardless of `minimumReleaseAge`/`trustPolicy` and is **not** narrowed by their exclude lists, because it guards *integrity*, not *maturity/trust*. Disabling the age/trust policies must not silently disable anti-tamper. (`createNpmResolutionVerifier` therefore always returns a verifier.) - **It is fail-closed.** An entry passes only when the registry metadata affirmatively lists the version with a matching tarball URL. If the metadata can't be fetched, doesn't list the version, or omits `dist.tarball`, the entry is rejected — otherwise a tampered lockfile could smuggle a malicious URL past the check by pointing it at a `name@version` the registry can't vouch for. - **Behavior change:** as a result, an install that re-verifies a lockfile (its content changed since the last verified run, so the verification cache no longer short-circuits) now requires the configured registry to be reachable. `trustLockfile` is the opt-out for environments that treat the on-disk lockfile as already trusted. - **Verification cache.** The policy snapshot records a `tarballUrlBinding` marker and `canTrustPastCheck` requires it, so a cache record written before this rule existed is re-verified rather than trusted (closing an upgrade-time bypass). - Entries with no explicit `tarball` reconstruct the URL from name+version+registry and are inherently bound (no check). `file:`/git-hosted resolutions stay out of scope (#12122). - Threads `nonSemverVersion` to the verifier so URL-keyed tarball deps (a remote `https:` tarball that carries a semver `version` copied from its manifest) are recognized as deliberate non-registry deps and skipped — also fixing a latent release-age over-match on them. The candidate dedupe key includes `nonSemverVersion` so a registry snapshot and a URL-keyed snapshot sharing a `name@version` and serialized resolution stay distinct. Mirrored in pacquet (`create_npm_resolution_verifier`). The dedupe-key change is TS-only: pacquet's candidate `version` comes from the lockfile key suffix, so the two shapes never share a key there. ## Tests - TS: confirmed mismatch → violation; non-standard URL matching metadata → pass; default-port/scheme difference → pass; URL-keyed dep → skipped; URL binding runs (and fails closed) with no age/trust policy configured; `canTrustPastCheck` rejects a cache record lacking the binding marker. Regression-verified (the mismatch test fails when the check is disabled). - pacquet: mirror tests + the no-policy / `minimumReleaseAge: 0` / `trustPolicy: off` cases, default-port/scheme equivalence, and the missing-`tarballUrlBinding` cache rejection. A few install-dispatch / resolution-reuse tests that pin a deliberately bogus tarball URL (or run against an unreachable registry to prove resolution reuse) now set `trustLockfile`, since the always-on fail-closed tarball-URL check would otherwise flag the fixture before the path under test runs. - `clippy --deny warnings`, `fmt`, and `dylint` clean. |
||
|
btea
|
722b9cda24 |
fix: skip lockfile minimumReleaseAge/trustPOlicy verification for non-registry tarball (#12122)
|
||
|
James Garbutt
|
1e9ab2935f |
feat: support staged publishes in trust scale (#12056)
Fixes #11887. Staged publishes now have a signal in the packument: `approver`. If this is set, the package is more trustworthy than a "trusted publisher" package, since it requires 2FA publish approvals. ## Changes **pnpm (TypeScript)** - `getTrustEvidence` recognizes `_npmUser.approver` and classifies it as a new `stagedPublish` trust evidence, ranked above `trustedPublisher` and `provenance`. - Trust-downgrade detection treats `stagedPublish` as the strongest rank, and the resolution verifier's PII-minimizing metadata projection retains the approver *signal* (without keeping the approver's name/email). **pacquet (Rust port)** - Ported the same staged-publish support: an `Approver` registry type, a `StagedPublish` trust evidence (rank 3 — above `TrustedPublisher`/`Provenance`), detection, pretty-printing, and the PII-stripping trust-meta projection. - Wired `trustPolicy='no-downgrade'` enforcement into the **resolver-time** path, not just the lockfile verifier. Previously pacquet only re-checked entries already in `pnpm-lock.yaml`; fresh resolutions weren't gated. The npm resolver now runs `fail_if_trust_downgraded` on each freshly picked version (full metadata is already forced under this policy), mirroring pnpm's resolver-time `failIfTrustDowngraded` call. - Ported the matching `trustChecks` tests for full parity with the TypeScript suite (staged-publish classification/downgrade, plus previously-unported `trustedPublisher → none`, no-evidence-anywhere, and exclude + missing-time cases). --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Silas Rech
|
623542873a |
fix(npm-resolver): revalidate excluded packages instead of trusting mtime cache (#12010)
Skip the publishedBy file-mtime fast path for fully excluded packages so stale abbreviated metadata cannot pin older versions, and add matching regression tests. |
||
|
Zoltan Kochan
|
aa6149df65 |
fix: fail by default when a tarball does not match the locked integrity (#11968)
`pnpm install` (non-frozen) used to react to `ERR_PNPM_TARBALL_INTEGRITY` by logging the error, silently re-resolving from the registry, and overwriting the locked integrity. The lockfile's integrity was effectively advisory by default — a compromised registry, proxy, or republished version could substitute attacker-controlled content on a clean machine even though the project shipped a committed `pnpm-lock.yaml`. Integrity mismatches against the lockfile now fail by default. The **only** opt-in is **`pnpm install --update-checksums`** — a new flag, narrowly scoped to refreshing the locked integrity values. Mirrors yarn's flag of the same name. A warning still prints when the bypass takes effect so the rewrite stays auditable. `--force` and `pnpm update` deliberately do **not** bypass the integrity check. They are routine refresh operations; silently overwriting a locked integrity in those flows would erase the protection a committed lockfile is supposed to provide. `--frozen-lockfile` behavior is unchanged. `--fix-lockfile` keeps its documented purpose (filling in missing lockfile entries) and is also not a bypass. Combining `--frozen-lockfile` with `--update-checksums` errors out — frozen mode refuses to rewrite the lockfile, which is exactly what `--update-checksums` is for. `--update-checksums` also bypasses the resolver's on-disk metadata cache fast path (`pickPackage.ts:271`, `pick_package.rs:531`). Without that, a stale on-disk packument that already contained the pinned version would short-circuit the registry entirely and the flag would silently no-op on dev machines. With the gate, every first-encounter goes through a conditional GET; the in-memory cache is left alone so second-and-onward references within the same install still hit cached fresh data (one network round-trip per *unique* package, not per reference). ## Reported by Reported privately via the security channel. The reproduction: 1. Publish `example-package@1.0.0` with content `v1` and install with pnpm; lockfile records the `v1` integrity. 2. Replace the registry's tarball+metadata for the same `1.0.0` with content `v2`. 3. On a clean store/cache, run `pnpm install`. Before this fix, pnpm logged `ERR_PNPM_TARBALL_INTEGRITY` but exited 0 with `v2` installed and the lockfile rewritten to the new integrity. After this fix, the same install exits non-zero. ## Prior art - **npm** ([sebhastian](https://sebhastian.com/npm-err-code-eintegrity/)): hard-fails with `EINTEGRITY`. No dedicated override flag — recovery is `npm cache clean --force`, manually editing the lockfile, or deleting it. - **yarn** ([Sean C Davis](https://www.seancdavis.com/posts/fix-yarn-integrity-check-failed/)): hard-fails with "Integrity check failed". Has a dedicated **`yarn install --update-checksums`** flag — pnpm now adopts the same name. ## Pacquet parity Pacquet was already fail-hard on integrity mismatch by default (no auto-repair path to remove). This PR brings the rest of the surface into line so `pnpm install --update-checksums` keeps working when pacquet is the materialization target, and `pacquet install --update-checksums` behaves identically standalone: - New `--update-checksums` flag on `pacquet install` (`crates/cli/src/cli_args/install.rs`), plumbed through `Install` and `InstallWithFreshLockfile` into the resolver. - When the flag is set, pacquet skips the frozen-lockfile fast path and routes through the fresh-resolve path so locked integrity values get rewritten from the registry. - `--frozen-lockfile + --update-checksums` errors with `pacquet_package_manager::frozen_lockfile_with_outdated_lockfile`, mirroring pnpm's `ERR_PNPM_FROZEN_LOCKFILE_WITH_OUTDATED_LOCKFILE`. - `pacquet_tarball::verify_checksum_error` now carries a help hint pointing at `--update-checksums` and calling out the supply-chain implication, matching the updated pnpm `TarballIntegrityError`. - The disk fast-path gate is mirrored in `crates/resolving-npm-resolver/src/pick_package.rs:531`, with the flag threaded from `ResolveOptions` → `PickPackageOptions`. |
||
|
mehmet turac
|
0721d64188 |
fix: require provenance for trusted publisher evidence (#11911)
* fix: require provenance for trusted publisher evidence
* test: align provenance fixtures with registry types
* chore: include pnpm CLI in changeset
The repo guideline requires every changeset that touches a published
package to list the pnpm CLI explicitly so the fix appears in the CLI's
release notes.
* fix(resolving-npm-resolver): require provenance for trusted publisher evidence
Ports pnpm's
|
||
|
Zoltan Kochan
|
212315de16 |
fix: cap lockfile verification memory and add trustLockfile opt-out (#11878)
* fix: cap lockfile verification memory and add trustLockfile opt-out Verifying a multi-thousand-entry lockfile against `minimumReleaseAge` or `trustPolicy: no-downgrade` retained every fetched packument in a per-install cache for the entire install. On large workspaces this OOM'd CI runners with a 2GB heap cap. Project both caches down to just the fields each check reads (per-version trust evidence + the `time` map for trust; package-level `modified` + version-name set for the abbreviated shortcut) so the bulk packument is GC'd as soon as the fetch returns. Also adds a `trustLockfile` setting (default `false`) that skips the verification pass entirely for environments where the lockfile is already part of the trusted base. Mirrored in pacquet. Closes #11860. * perf: share resolver packument cache with the lockfile verifier The verifier kept its own per-install dedup Maps and re-fetched every packument the resolver had already pulled during the same install. Plumb the resolver's per-install `PackageMetaCache` through to the verifier (via `createNpmResolutionVerifier` / `build_resolution_verifiers`) so a name already in the resolver's LRU short-circuits the verifier's disk/network round-trip — fast path only, the cached document is projected for the trust check so the verifier's memory footprint stays bounded. In pnpm, `installing/client` now constructs one LRU and hands it to both `createResolver` and `createResolutionVerifiers`. In pacquet, the `InMemoryPackageMetaCache` is lifted to `Install::dispatch` and passed to both `build_resolution_verifiers` and `InstallWithFreshLockfile`. |
||
|
Zoltan Kochan
|
3a54205178 | feat(pacquet): resolver scaffold + npm version picking (#11755) | ||
|
Zoltan Kochan
|
1627943d2a |
feat(outdated): include node, deno, and bun runtimes (#11739)
`pnpm outdated` and `pnpm update --interactive` previously skipped runtime dependencies (`node`/`deno`/`bun` installed via the `runtime:` protocol). Both commands go through `outdatedDepsOfProjects` → `outdated()`, and the inner loop bailed out for anything `parseBareSpecifier` couldn't parse — which is everything `runtime:`-shaped. A runtime was only ever reported if the current install differed from the wanted lockfile entry, so the latest available version was never surfaced. The same gap silently affected `jsr:` and named-registry deps too. Commits, smallest fix first → progressively cleaner architecture: 1. **`feat(outdated)`** — minimal fix: special-case runtime deps in `outdated.ts` so they appear in the table and the interactive update picker. 2. **`refactor(outdated)`** — per-resolver dispatch. Each protocol resolver gets its own "what's the latest?" function; `@pnpm/resolving.default-resolver` composes them. 3. **`refactor(outdated)`** — rename to `resolveLatest` (the function returns info regardless of whether the dep is outdated; "outdated" described a state, not an action). 4. **`refactor(outdated)`** — let the local-resolver own the `link:`/`file:` skip, drop the matching short-circuit in `outdated.ts`. 5. **`refactor(outdated)`** — slim `LatestQuery` / `LatestInfo` to the bare essentials; move `pickRegistryForPackage` into the npm-resolver where it belongs; derive `current`/`wanted` display from `pkgSnapshot.version` in `outdated.ts`. 6. **`chore(outdated)`** — drop stale tsconfig project reference left behind by #5. 7. **`refactor(outdated)`** — drop `wantedRef` from the query; resolvers detect protocol from `bareSpecifier` alone. ## Final architecture `@pnpm/resolving.resolver-base` defines a single tiny protocol: ```ts interface LatestQuery { wantedDependency: WantedDependency compatible?: boolean } interface LatestInfo { latestManifest?: PackageManifest } type ResolveLatestFunction = (query: LatestQuery, opts: ResolveOptions) => Promise<LatestInfo | undefined> ``` - `undefined` from a resolver means "I don't claim this dep — try the next one." - `{}` means "I claim it, but I can't tell you what's latest" (policy-blocked, network unavailable, or a protocol with no concept of latest — git/tarball). - `{ latestManifest }` is the happy path. Each protocol resolver (npm/jsr/named-registry, git, tarball, local, node/bun/deno runtimes) exports its own `resolveLatest*` function alongside its `resolve*`. `@pnpm/resolving.default-resolver` composes them into a single first-match dispatcher, surfaced through `@pnpm/installing.client` as `createResolver(...).resolveLatest`. `outdated.ts` is protocol-agnostic: dispatches, then derives `current`/`wanted` display from `pkgSnapshot.version` (falling back to the raw ref for URL-shaped refs where the URL is the only diff signal between commits), uses raw `wantedRef !== currentRef` for the lockfile-shifted check, and pulls `packageName` from `dp.parse(relativeDepPath).name` so aliased deps still report under the real package name. Per-resolver responsibilities: - **npm-resolver** (`resolveLatestFromNpm` / `resolveLatestFromJsr` / `resolveLatestFromNamedRegistry`): match their respective spec shapes, call the matching `resolveFromX` with `'latest'` (or the original spec under `--compatible`), handle `MINIMUM_RELEASE_AGE_VIOLATION` and `ERR_PNPM_NO_MATCHING_VERSION` so policy-blocked deps don't surface as available updates. Picks the per-package registry internally via its ctx. - **node/bun/deno runtime resolvers**: claim deps via `bareSpecifier.startsWith('runtime:')` + alias match, query their release sources for the latest version (only the version — no asset-hash fetches), return `{ latestManifest }`. - **git / tarball resolvers**: claim deps via spec shape, return `{}` (no concept of "latest"); the caller still surfaces a ref-mismatch report if the lockfile shifted to a different commit/URL. - **local-resolver**: returns `undefined` so `link:`/`file:`/`workspace:` deps fall through and get silently skipped. |
||
|
Zoltan Kochan
|
4195766f10 |
feat: tighten minimumReleaseAge — auto-exclude, lockfile verification, and interactive prompt (#11705)
Three coordinated changes that close the silent-bypass gap in loose `minimumReleaseAge` mode AND the discover-by-loop UX problem in strict mode (#10488), plus a parallel hardening of the lockfile verifier: 1. **Auto-collect into `minimumReleaseAgeExclude` (loose mode)** — fresh resolutions that fall back to a version newer than the cutoff are auto-recorded into the workspace manifest's `minimumReleaseAgeExclude`. A single info message lists what was persisted. The workspace manifest writer dedupes against existing entries. 2. **Lockfile verifier runs in loose mode too** — `createNpmResolutionVerifier` no longer gates on `minimumReleaseAgeStrict`. With auto-collect keeping the exclude list explicit, every accepted-immature pin must be on the list — same contract strict mode enforces. Lockfiles produced under a weaker (or absent) policy that still hold immature entries are rejected the same way strict mode would. 3. **Strict mode prompts on the aggregate set instead of throwing on the first** — the resolver always collects every immature direct and transitive in one pass; the install command's `handleResolutionPolicyViolations` checkpoint decides what to do with the set. Interactive (TTY) prompts the user once with the full list (default = No) and asks whether to add them all to `minimumReleaseAgeExclude` and proceed. Approve → install continues, persisted at the end. Decline → resolution aborts before the lockfile, package.json, or modules dir is touched. Non-interactive (CI) keeps `ERR_PNPM_NO_MATURE_MATCHING_VERSION` as the exit code but lists every offending entry instead of just the first one the resolver happened to hit. 4. **The lockfile verifier now also covers `trustPolicy: 'no-downgrade'`.** The same post-resolution gate that re-checks `minimumReleaseAge` on lockfile entries now re-runs `failIfTrustDowngraded` for every npm-registry entry whose name isn't on `trustPolicyExclude`. The two checks share a single full-metadata fetch per package, so the extra coverage doesn't cost an extra round trip when both policies are active. Resolver-time trust checks still run as before — this just closes the gap when an entry bypasses resolution (peek path, `--frozen-lockfile`, restored CI cache). The steady-state flows: - **Loose mode, `pnpm add foo@immature`**: lockfile clean, verifier no-op, resolver picks via lowest-version fallback, `foo@immature` lands in `minimumReleaseAgeExclude`, install succeeds. Subsequent `pnpm install --frozen-lockfile` in CI verifies against the populated list and succeeds. - **Strict mode (interactive), security bump to `next@15.5.9`**: resolver collects `next@15.5.9` AND every immature `@next/swc-*@15.5.9` shim. pnpm prompts once with the full list. User approves → install completes, all entries persisted in `pnpm-workspace.yaml`. CI then runs the populated config cleanly. - **Strict mode (non-interactive / CI)**: aborts with `ERR_PNPM_NO_MATURE_MATCHING_VERSION` listing every immature entry's `name@version` and publish time — no more discover-by-loop dance. - **Teammate commits a poisoned lockfile**: single-policy batches reject with `ERR_PNPM_MINIMUM_RELEASE_AGE_VIOLATION` (or `ERR_PNPM_TRUST_DOWNGRADE`); a batch that trips both policies escalates to the generic `ERR_PNPM_LOCKFILE_RESOLUTION_VERIFICATION` and lists each entry's per-policy code in the breakdown. ### Implementation - The npm resolver always falls back to the lowest matching version when no mature version satisfies the range, and flags the result with `ResolveResult.policyViolation` instead of throwing `NO_MATURE_MATCHING_VERSION`. `deferImmatureDecision` and `strictPublishedByCheck` are gone — every caller (install, dlx, outdated, self-update) inspects the violation and decides what to do. - `policyViolation` flows from `ResolveResult` → `PackageResponse.body.policyViolation` → a shared accumulator in `ResolutionContext` → the `resolutionPolicyViolations` field on `resolveDependencyTree`'s return → out through `mutateModules` / `addDependenciesToPackage` to the install command. - The violation type lives in `@pnpm/resolving.resolver-base` as `ResolutionPolicyViolation`; the npm resolver exports the two built-in codes (`MINIMUM_RELEASE_AGE_VIOLATION_CODE`, `TRUST_DOWNGRADE_VIOLATION_CODE`) as constants so consumers reference one source of truth. - `handleResolutionPolicyViolations` runs between `resolveDependencyTree` and `resolvePeers` — the resolver-agnostic checkpoint where the install command's plan prompts (TTY) or aborts (no-TTY) with the full violation list. - `setupPolicyHandlers` (in `installing/commands/src/policyHandlers.ts`) composes per-policy handlers behind a uniform plan interface: each handler has its own `handleResolutionPolicyViolations` (filter by code, decide what to do) and `pickManifestUpdates` (return a typed `WorkspaceManifestPolicyUpdates` patch the install command spreads into `updateWorkspaceManifest`). Today the only registered handler is `createMinimumReleaseAgeHandler` — strict + TTY prompts via `enquirer`, strict no-TTY throws `ERR_PNPM_NO_MATURE_MATCHING_VERSION` with every entry listed, loose mode auto-persists at the tail. Strict + `--no-save` is rejected up-front via `ERR_PNPM_STRICT_MIN_RELEASE_AGE_REQUIRES_SAVE`. Future policies plug in via a sibling factory + push into the handlers list, with no changes to `installDeps.ts` / `recursive.ts`. - `installDeps` / `recursive` drain `pickManifestUpdates` after install and spread the patch into `updateWorkspaceManifest`. Plain `pnpm install` (no `--update`, no params) now still updates the workspace manifest when any handler contributes a patch. The `install` command's CLI schema gained `save: Boolean` so `--no-save` actually flows through to `opts.save = false` instead of being silently dropped by nopt. - `makeResolutionStrict` (in `installing/client`) wraps a `ResolveFunction` and rethrows any `policyViolation` as a `PnpmError`. Used by `dlx` and `self-update` under strict `minimumReleaseAge` OR `trustPolicy: 'no-downgrade'`, since one-shot callers have nowhere to defer a violation to. Violation-code → error-code mapping lives in one place so future violation kinds get consistent UX. - `createNpmResolutionVerifier` extends its check to `trustPolicy: 'no-downgrade'` — same per-entry fan-out, same cache key, sharing the full-metadata fetch with the maturity check. Trust-fetch errors now propagate up so the violation reason carries the underlying message (network code, 404 detail) instead of a generic "metadata is unavailable". - `verifyLockfileResolutions`'s aggregate throw uses the per-policy code when every violation in the batch shares it, and escalates to a generic `LOCKFILE_RESOLUTION_VERIFICATION` (with per-entry codes in the breakdown) for mixed batches. - The pnpm agent path refuses installs under `trustPolicy: 'no-downgrade'` (`ERR_PNPM_TRUST_POLICY_INCOMPATIBLE_WITH_AGENT`) — the agent has no server-side counterpart to that check yet, so silently allowing it would land a lockfile the local verifier would later reject. `minimumReleaseAge` is forwarded to the agent and enforced server-side, so that combination is fine. ### Pacquet parity Pacquet only carries a stub reference to `minimumReleaseAgeExclude` (see `pacquet/crates/package-manager/src/version_policy.rs`); the broader `minimumReleaseAge` and `trustPolicy` policies aren't ported yet, so this feature is outside pacquet's current surface area. It'll come along when pacquet ports the policies. ### Closes - Closes #10488 (resolves the discover-by-loop dance for security bumps without needing `withTransitives`). |
||
|
Zoltan Kochan
|
963861cac1 |
perf(npm-resolver): layer abbreviated meta + attestation before full metadata in the minimumReleaseAge gate (#11704)
Follow-up to #11691 — item 2 from #11687, plus a related shortcut. ## What When the `minimumReleaseAge` lockfile verification gate needs to know when a version was published, it used to fetch a multi-MB full metadata document per package just to read one timestamp. This PR replaces that single-step path with a four-layer lookup that pays the cheapest viable source first: 1. **Abbreviated metadata's `modified` field** — the resolver already fetches this for resolution. If the package as a whole hasn't been modified within the policy cutoff, every version it contains is at least that old; return `modified` as a conservative upper-bound and skip the rest of the chain. 2. **Local `FULL_META_DIR` mirror** — exact per-version times if a previous verification populated it. 3. **npm attestation endpoint** (`/-/npm/v1/attestations/<name>@<version>`) — a tens-of-KB Sigstore bundle whose Rekor inclusion time stands in for publish time. Wins on cold cache when the package was published with provenance. 4. **Full metadata fetch** — last resort. ## Why The verification cache from #11691 made repeat installs against an unchanged lockfile effectively free. The remaining cost is the *first* verification on a fresh CI runner with no restored cache — particularly `pnpm install --frozen-lockfile`, where every locked package's publish timestamp has to be confirmed. Fetching the full metadata document for each package is wasteful when: - The resolver has usually already cached abbreviated metadata, whose `modified` field alone is enough to clear stable packages (the common case). - For recently-modified packages, the per-version attestation endpoint is orders of magnitude smaller than full metadata. ## How ### Abbreviated `modified` shortcut `fetchFullMetadataCached` is refactored to share an internal helper with a new `fetchAbbreviatedMetadataCached`. Both do conditional GETs against their respective on-disk mirrors. On a non-frozen install the abbreviated mirror is already populated by the resolver, so the shortcut hits the local cache at headers-only cost. On `--frozen-lockfile` the fetch is still cheaper than full metadata. If `Date.parse(modified) < cutoff`, return `modified` — it's an upper bound on every version's publish time in this package, and the verifier's `published < cutoff` check passes trivially. ### Attestation endpoint `fetchAttestationPublishedAt` (new module) hits `/-/npm/v1/attestations/<name>@<version>`, parses the response, and reads the earliest `bundle.verificationMaterial.tlogEntries[].integratedTime` across the attestation bundles. That's the Rekor inclusion time — a couple of seconds after publish, well within tolerance for a policy that operates in minutes/hours/days. Returns `undefined` on 404 / network error / malformed body so the caller falls back. ### Per-install dedup The lookup carries a `PublishedAtLookupContext` with four memo maps: abbreviated meta per (registry, name), local full meta per (registry, name), full meta network fetch per (registry, name), final published-at per (registry, name, version). Verifying many versions of the same package only pays the disk/network costs once. ## Trust model **No Sigstore signature verification on the attestation path.** The trust model is identical to reading the registry's `time` field on the full metadata document — we already trust the registry to serve correct publish timestamps for the gate's purpose. The win is purely bandwidth. Full Sigstore verification (Fulcio cert chain + Rekor inclusion proof) would harden the timestamp against a compromised registry. It pulls in the `sigstore` npm package and the TUF root — a separate dependency-surface discussion, parked as future work. ## Tests - **13 unit tests** in `resolving/npm-resolver/test/fetchAttestationPublishedAt.test.ts`: ISO timestamp extraction, URL construction (scoped + unscoped), 404 / 5xx / network error / malformed JSON / missing fields → undefined, earliest-of-multiple-attestations, defensive number-as-integratedTime, auth header forwarding, trailing-slash normalization. - Existing `minimumReleaseAge` + `verifyLockfileResolutions` integration suites (45 tests) still pass — the fallback chain preserves end-to-end behavior when the new shortcuts don't apply. |
||
|
Zoltan Kochan
|
fcf95c7faa |
perf: cache the post-resolution lockfile verification gate (#11691)
Closes #11687. ## What Cache the result of the post-resolution lockfile verification gate (#11583) so repeat installs against an unchanged lockfile skip the per-package registry round trips entirely. Persisted as JSON Lines at `<cacheDir>/lockfile-verified.jsonl`. The cache layer is policy-neutral. Today there's one verifier (`minimumReleaseAge`); future resolver-side verifiers (jsr trust, attestation, …) plug in by declaring their own `policy` slot and `canTrustPastCheck` comparator — no install-side changes. ## Why #11583 re-hits the registry on every install for every locked (name, version) pair. On warm/repeat installs where the lockfile hasn't moved, that's a stack of per-package round trips with nothing to show for them. This change makes the steady-state case effectively free without weakening the protection — the gate still runs in full whenever the lockfile changes, any verifier's policy tightens, or no record exists. ## How ### Cache lookup, in order The cache is **indexed by content hash** so git worktrees with identical lockfile bytes share a cache entry. A secondary path-keyed index drives the same-machine stat shortcut. 1. **`stat()` shortcut** — when a previous record for this exact `lockfilePath` matches today's `size + mtime + inode`, trust the cached hash without reading anything. Zero I/O beyond the stat. Microseconds. 2. **Content lookup** — hash the in-memory lockfile (not the file bytes — we already have the parsed object) and look up by content hash. Catches worktrees (same content, different path) and CI checkouts (same content, reset stat). On hit, append a refreshed path/stat entry so the next install at this path takes the stat shortcut. 3. **Any active verifier rejects the cached `policy`** — run the full gate. 4. **No record** — run the full gate. The in-memory object is hashed with `hashObject` from `@pnpm/crypto.object-hasher` (streaming, key-order-stable). ### Record shape ```json { "lockfile": { "hash": "<sha256 base64>", "path": "/abs/path/to/pnpm-lock.yaml", "size": 154, "mtimeNs": "1736245123000000000", "inode": "12345" }, "verifiedAt": "2026-05-17T...", "policy": { "minimumReleaseAge": 1440 } } ``` `policy` is the union of every active verifier's `policy` contribution. Verifiers checking the same logical policy (e.g. `minimumReleaseAge` honored by multiple registries) name it the same and share the slot — no resolver namespacing. ### File semantics - **Sync fs throughout** — the cache is consulted once before verification fan-out and recorded once after. No concurrent install work to overlap with; keeping the call sites straight-line. - **JSONL appends are atomic** on POSIX/NTFS, so parallel pnpm processes (monorepo installs, CI matrices sharing a cache) write without coordination. Latest record per `(path, hash)` tuple wins on read. - **Bounded file** — capped at ~1000 entries; compaction is triggered by a single `stat()` of the cache file (1.5 MiB byte budget) so we never parse the file on the steady-state path. When triggered, the tail is rewritten via tempfile + rename. - **No record on rejection** — a failing verification deliberately doesn't write a record; the next install must rerun the gate. - **Single hash per install** — the in-memory hash is computed lazily and reused: `tryLockfileVerificationCache` returns the precomputed stat+hash to `recordVerification` on a miss, and the stat-shortcut hit forwards the cached record's hash unchanged. ## Plumbing The verifier contract changed alongside the cache to make this composable without install-side knowledge of each policy: - **`@pnpm/resolving.resolver-base`** — `ResolutionVerifier` is now `{ verify, policy, canTrustPastCheck }` (was a bare function in #11583). Each resolver-side verifier owns its policy snapshot and the comparator that decides whether a cached policy is still trustworthy. - **`@pnpm/resolving.npm-resolver`** — `createNpmResolutionVerifier` returns the new shape: `policy: { minimumReleaseAge }`, `canTrustPastCheck` reads `minimumReleaseAge` from the merged cached bag. - **`@pnpm/resolving.default-resolver`** — `createResolutionVerifier` (singular, returning a combined function) → `createResolutionVerifiers` (plural, returning a `ResolutionVerifier[]`). No combinator; each verifier handles its own protocol short-circuit inside `verify`, so dispatch happens naturally at the install side. - **`@pnpm/installing.client`** — `Client.verifyResolution?` → `Client.resolutionVerifiers: ResolutionVerifier[]`. Same rename propagates through `@pnpm/store.connection-manager`, `@pnpm/testing.temp-store`, and `StrictInstallOptions`. - **`@pnpm/installing.deps-installer`** — new `verifyLockfileResolutionsCache.ts` (`tryLockfileVerificationCache` + `recordVerification`). `verifyLockfileResolutions` takes the verifier list plus `cacheDir` + `lockfilePath` as flat options; the cache fires when both are present, otherwise the gate runs without memoization. The dedup key for in-flight candidates includes a serialization of `resolution` so two entries sharing a (name, version) but pinned via different protocols don't collapse. Breaking but safe — `@pnpm/resolving.npm-resolver` hasn't been released since #11583 introduced the verifier abstraction, so no downstream consumer is on the old shape. ## Tests - **17 unit tests** in `verifyLockfileResolutionsCache.ts`: cache miss/hit, stat shortcut, size mismatch falling through to hash lookup, hash-fallback on reset stat, content change with matching size, stricter/weaker policy, missing-field policy rejection, multi-verifier policy merge (shared field stored once), worktree case (same content, different path), JSONL append semantics, malformed-line tolerance. - **12 integration tests** in `verifyLockfileResolutions.ts`: dedup of peer/patch-suffix variants, distinct-resolution dedup at the same (name, version), stable violation ordering, the 20-entry cap, multi-verifier fan-out (first failure wins), cache short-circuit on a passing run, no cache write on a rejecting run, empty-verifier-list passthrough. - **1 e2e test** in `pnpm/test/install/minimumReleaseAge.ts`: bundled CLI plumbing — install once to seed the lockfile, enable `minimumReleaseAge` + `cacheDir`, install again, assert the cache file lands at `<cacheDir>/lockfile-verified.jsonl` with the documented record shape. - Existing `minimumReleaseAge` (13) and `frozenLockfile` (12) suites still pass. |
||
|
Ryo Matsukawa
|
31538bf8d2 |
fix: enforce minimumReleaseAge on existing lockfile entries (#11583)
Closes #10438. ## What Re-verify every entry in `pnpm-lock.yaml` against the policies the resolver chain was configured with — today: `minimumReleaseAge` in strict mode — right after the lockfile is loaded from disk and before any tarball is fetched. A locked version that fails the policy aborts the install with `ERR_PNPM_MINIMUM_RELEASE_AGE_VIOLATION`; `minimumReleaseAgeExclude` is honored. ## Why The policy only fires while pnpm is *choosing* a version. Once a version is pinned in the lockfile — e.g. a developer disabled the policy locally and committed a fresh dependency, or a CI cache restored a stale lockfile — every later `pnpm install` (including `--frozen-lockfile` and `pnpm fetch`) installs it without re-checking, which defeats the supply-chain protection the setting is supposed to provide. The threat model is **a lockfile someone else resolved**, not local resolution: local resolution is already covered by the resolver's own per-version filter. bun fixed the same shape of bug in [oven-sh/bun#30526](https://github.com/oven-sh/bun/pull/30526); this PR is the pnpm side. ## How The fix introduces a generic `ResolutionVerifier` abstraction in the resolver chain — each resolver factory can ship a sibling verifier factory, exactly the way each resolver ships a `resolve` function. Today there's one verifier (npm); the shape leaves room for future ones (jsr, attestation-based, etc.) without changing the install-side interface. - **`@pnpm/resolving.resolver-base`** exports the `ResolutionVerifier` / `ResolutionVerification` types — the shared contract. - **`@pnpm/resolving.npm-resolver`** exports `createNpmResolutionVerifier`. Returns `undefined` when no policy is active, so callers can cheaply decide whether to iterate at all. When active, it inspects each lockfile entry, handles `minimumReleaseAgeExclude`, routes through named-registry prefixes (built-ins like `gh:` merged in), and uses `fetchFullMetadataCached` to fetch full registry metadata — decoupled from the resolver pipeline so neither `peekManifestFromStore` nor abbreviated metadata can hide the publish timestamp. - **`@pnpm/resolving.default-resolver`** exports `createResolutionVerifier`, a combinator that asks each underlying verifier (today: npm) if it has work and returns `undefined` when none does. Designed so that adding more verifiers later doesn't change the install side. - **`@pnpm/installing.client`** exposes `verifyResolution` on `Client`, built from the same `fetchFromRegistry` / `getAuthHeader` the resolver chain already uses — **no second fetcher is constructed**. - **`@pnpm/store.connection-manager`** and **`@pnpm/testing.temp-store`** surface `verifyResolution` alongside the store controller they hand back, so it reaches `mutateModules` through the existing plumbing. - **`@pnpm/installing.deps-installer`** gains one option on `StrictInstallOptions`: `verifyResolution?: ResolutionVerifier`. `mutateModules` invokes `verifyLockfileResolutions(ctx.wantedLockfile, opts.verifyResolution)` **once**, right after `getContext` returns the on-disk lockfile and before any path branches. When the verifier is `undefined`, the call is a no-op. The iteration is policy-neutral: dedupes by `(name, version)`, applies `pLimit(16)`, sorts violations stably, caps the printed list at 20 with an `…and N more` summary, throws a `PnpmError` carrying the verifier-supplied error code. The error includes a recovery hint that points at `pnpm clean --lockfile` followed by `pnpm install` — the safe way to throw away a poisoned lockfile and rebuild from fresh resolution. ## Tests - **9 unit tests** for `verifyLockfileResolutions` against a mock `ResolutionVerifier` — dedup, aggregation, stable ordering, the 20-entry cap, no-op behavior, the verifier-supplied error code surfacing in `PnpmError`. - **13 integration tests** in `installing/deps-installer/test/install/minimumReleaseAge.ts` via the real `install()` entry — `testDefaults()` wires `verifyResolution` from `createTempStore` → `createClient`, so the npm verifier runs end-to-end at the install boundary. Covers the rejection scenario, `minimumReleaseAgeExclude`, the strict-mode toggle, the existing `minimumReleaseAge` resolver-side suite, and a `pnpm add` scenario where a pre-existing entry would otherwise survive resolution. - **3 e2e tests** in `pnpm/test/install/minimumReleaseAge.ts` against the bundled CLI: rejection path with the right `ERR_PNPM_*` code and `pnpm clean --lockfile` hint in output, `minimumReleaseAgeExclude` honored, and the strict-off path (which now requires an explicit `minimumReleaseAgeStrict: false` since the config reader auto-enables strict mode when `minimumReleaseAge` is set). - Existing `frozenLockfile` suite (12 tests) and npm-resolver suite (179 tests) still pass. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Katerina Skroumpelou
|
e526f89650 |
fix(npm-resolver): minimumReleaseAge handling for cached abbreviated metadata (#11622)
* fix(npm-resolver): dont rethrow ERR_PNPM_MISSING_TIME from version-spec cache * fix(npm-resolver): upgrade cached abbreviated metadata on 304 for minimumReleaseAge * fix(npm-resolver): expand abbreviated-meta upgrade to in-memory cache and preferOffline paths * fix(npm-resolver): address Copilot review feedback on pickPackage - Extract `persistUpgradedMeta` helper and call it from all three sites (in-memory cache hit, preferOffline disk-cache hit, 304 path) so a fresh process doesn't repeat the upgrade fetch. - Forward `etag`/`modified` to the upgrade fetch in `maybeUpgradeAbbreviatedMetaForReleaseAge` so the registry can answer 304. - Extract `shouldRethrowFromFastPathCache` so the two fast-path catch sites can't drift on the MISSING_TIME-vs-strict invariant. - Document the deliberate choice to upgrade-fetch when `meta.modified` is absent or unparseable (correctness over saving a network call). - Add a companion test that exercises the catch fix with the default `ignoreMissingTimeField` so the invariant holds regardless of that flag. - Fix the existing `bareSpecifier: '3.1.0'` test setup: 3.1.0 was published 2016-01-11, after the test's `publishedBy` of 2015-08-17, so strict mode correctly rejected it. Switch to 3.0.0 (released 2015-07-10). * chore(npm-resolver): replace 'unparseable' with 'malformed' for cspell * style(npm-resolver): declare pickPackage helpers after their caller --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Peter Goldberg
|
c2c289094f | fix: time-based resolution loses publishedAt on fast path (#11618) | ||
|
Zoltan Kochan
|
50b33c1e6b |
fix: address open CodeQL findings (#11609)
Resolves the 15 open alerts on https://github.com/pnpm/pnpm/security/code-scanning by addressing all four categories that CodeQL flagged. ### Prototype-polluting assignment (3 alerts, product code) - `pkg-manifest/utils/src/convertEnginesRuntimeToDependencies.ts`: the inner write now dispatches over a literal `switch` on `runtimeName`, so the assignment is always keyed by `'node' | 'deno' | 'bun'`. - `pkg-manifest/utils/src/updateProjectManifestObject.ts`: added an `isProtoPollutionKey` barrier at the top of the loop so `packageSpec.alias` can never reach the dynamic property write with `__proto__` / `constructor` / `prototype`. - `installing/deps-installer/src/uninstall/removeDeps.ts`: the package list is filtered through `isProtoPollutionKey` once up front, and the dependency record is captured into a local before the loop. ### Polynomial ReDoS (2 alerts) - `deps/inspection/list/src/renderDependentsTree.ts`: `replace(/\n+$/, '')` swapped for a constant-time `charCodeAt` trim. - `resolving/npm-resolver/src/fetch.ts`: removed the super-linear-backtracking `semverRegex` and replaced it with an O(n) `stripTrailingSemverSuffix` that splits on the rightmost `@` and `semver.valid`s, with a digit-block fallback so `foo1.0.0`-style names still produce the existing "Did you mean foo?" hint. ### Bad code sanitization (8 alerts, test infrastructure) - `__utils__/test-ipc-server/src/TestIpcServer.ts`: the `JSON.stringify(...).slice(1, -1)` smell at the source of all 8 test-file alerts is gone. Both `sendLineScript` and `generateSendStdinScript` now build the JS source with plain `JSON.stringify` and delegate shell wrapping to a new `wrapNodeEval` helper that escapes `\\` and `"` for the outer double-quoted shell argument. ### Incomplete sanitization (2 alerts, test file) - `releasing/commands/test/publish/oidcProvenance.test.ts`: `.replace('/', '%2f')` → `.replaceAll(...)` on both flagged lines. |
||
|
btea
|
8543b89fb2 |
feat: view command display published time (#11532)
* **New Features**
* `pnpm view` now shows publish age (e.g., "published 2 hours ago") and, when available, publisher attribution ("by …"); invalid or future timestamps fall back to "just now".
* Console styling for package metadata (name/version, license, counts, links, dependencies, maintainers) was improved for readability.
* **Tests**
* Added tests verifying the published timestamp and publisher attribution appear in `pnpm view` output.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Zoltan Kochan
|
38bdfefdf9 |
fix: upgrade @pnpm/semver-diff and @pnpm/colorize-semver-diff to v2 (#11555)
- Upgrade `@pnpm/semver-diff` and `@pnpm/colorize-semver-diff` to v2, which expose the helpers as named exports. - Update the call sites in `@pnpm/deps.inspection.commands` and `@pnpm/installing.commands` from `semverDiff.default(...)` / `colorizeSemverDiff.default(...)` to plain `semverDiff(...)` / `colorizeSemverDiff(...)`. - Refactor `buildPkgChoice` in `getUpdateChoices.ts` to build the row as a `string[]`. Previously the row was an object whose values relied on `nextVersion` being inferred as `any` (a side effect of the broken `.default` access poisoning the type) — that masked `outdatedPkg.current` and `outdatedPkg.workspace` being `string | undefined`. With the v2 named imports the types tighten up, and `Object.values(lineParts)` would no longer assign cleanly to `string[]`. The previous v1 packages exported their helpers as `module.exports.default = fn`, so `.default(...)` only worked through the legacy CJS interop — and it broke under Node.js ESM (which is what the Jest runner uses with `--experimental-vm-modules`). Most of the `deps/inspection/commands` outdated tests had been silently failing on `main` with `TypeError: semverDiff.default is not a function`; this change brings them back. |
||
|
Igor Savin
|
b61e268d57 |
feat: add support for github prefix and named registries (#11324)
This is consistent with #9358, but implements support for the GitHub Packages npm registry and, more broadly, for vlt-style https://docs.vlt.sh/cli/registries for any registry. This PR adds a built-in gh: specifier that resolves against the GitHub Packages npm registry, plus a namedRegistries config key so a project can map its own aliases to arbitrary registries. A project can mix public npm packages and private GitHub Packages (or self-hosted) ones without applying a scope-wide registry override to every @scope/* package. - pnpm add gh:@acme/private writes "@acme/private": "gh:^1.0.0" and resolves from https://npm.pkg.github.com/. - pnpm add gh:@acme/private@^1.0.0 (with or without an alias) is also supported. Aliased form writes "my-alias": "gh:@acme/private@^1.0.0". - Auth comes from the existing per-URL .npmrc mechanism, e.g. //npm.pkg.github.com/:_authToken=${GITHUB_TOKEN}. No new auth surface. - @github is intentionally not defaulted to https://npm.pkg.github.com/ - hardcoding that would hijack installs of the public @github/* packages on npmjs.org (e.g. @github/relative-time-element) for users without a scope-wide override. Use gh: to install from GitHub Packages, or configure @github:registry=... yourself if that's really what you want. - Additional named registries (a self-hosted proxy, GitHub Enterprise Server, etc.) can be configured in pnpm-workspace.yaml: ```yml namedRegistries: gh: https://npm.pkg.github.example.com/ # optional: overrides the built-in `gh` alias for GHES work: https://npm.work.example.com/ ``` - Then work:@corp/lib@^2.0.0 resolves against https://npm.work.example.com/, and the built-in gh alias can be redirected to a GHES host. - Env-var substitution (${VAR}) is supported in namedRegistries values, mirroring the .npmrc convention. - Reserved alias names (npm, jsr, github, workspace, catalog, file, git, http, https, link, patch, and related git host shorthands) cannot be redefined as user-named registries - the resolver throws ERR_PNPM_RESERVED_NAMED_REGISTRY_ALIAS at startup rather than silently shadowing another protocol. Malformed URLs throw ERR_PNPM_INVALID_NAMED_REGISTRY_URL at startup too, instead of failing as a confusing 404 during resolution. - On publish, createExportableManifest strips any named-registry prefix (both the built-in gh: and any user-configured alias) so npm and yarn consumers can still resolve the dependency via their own scope-registry configuration - mirroring the user-facing requirement when installing such a dep without the prefix. The prefix is gh: rather than github: because github: is reserved by npm-package-arg / hosted-git-info as a git host shorthand (e.g. github:owner/repo) - reusing it would be a deviation from the specs used by the npm CLI. gh: is shorter, matches vlt's convention, and cannot collide with any existing npm scheme. Unlike jsr:, gh: (and any other named-registry alias) does not rewrite the package name - gh:@acme/foo resolves @acme/foo from the GitHub Packages registry as-is. This also means npm/yarn consumers see the original name after the prefix is stripped on publish. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
ccc606ed15 |
feat: pnpm agent — server-side resolution for faster installs (#11251)
## Summary
Adds an opt-in **pnpm agent** server that resolves dependencies server-side and streams only the files missing from the client's content-addressable store.
- **`@pnpm/agent.server`** — multi-process HTTP server (Node.js `cluster`) with SQLite-backed metadata and file caches
- **`@pnpm/agent.client`** — streams an NDJSON response, dispatches worker threads to fetch files while the server is still resolving
- **New config**: `agent` in `pnpm-workspace.yaml` (opt-in)
## How it works
1. Client reads integrity hashes from its local store index
2. Sends `POST /v1/install` with dependencies + store integrities
3. Server resolves the dependency tree using pnpm's `install({ lockfileOnly: true })`, with a SQLite-backed `PackageMetaCache` for fast repeat resolution
4. As each package resolves, a wrapped `storeController.requestPackage` looks up its files and immediately streams digests the client is missing (NDJSON `D` lines)
5. Client reads the stream line by line; digest batches fill up and dispatch worker threads to `POST /v1/files` — file downloads overlap with server-side resolution
6. After resolution, server sends index entries (`I` lines) and lockfile (`L` line)
7. Client writes index entries to store, then runs headless install with a wrapped `fetchPackage` that calls `readPkgFromCafs` with `verifyStoreIntegrity: false` (files are trusted from the agent)
8. `/v1/files` response is gzip-streamed (274MB → ~80MB) — server pipes through `createGzip`, worker pipes through `createGunzip`, parsing and writing files to CAFS as data arrives
## Performance
1351-package project, cold local store, warm server (localhost):
| Scenario | Time |
|----------|------|
| Vanilla pnpm install (cold OS cache) | ~48s |
| Vanilla pnpm install (warm OS cache) | ~34s |
| With pnpm agent (consistent) | **~33s** |
### Key optimizations
1. **SQLite metadata cache** — server-side resolution drops from ~3.4s to ~0.9s
2. **SQLite file store** — consistent read performance regardless of OS file cache state
3. **Streaming `/v1/install`** — file digests stream during resolution, downloads start before resolution finishes
4. **Gzip-streamed `/v1/files`** — whole-stream gzip (274MB → ~80MB), significant savings on remote servers
5. **Worker-thread streaming HTTP** — workers pipe gzip → parse → write to CAFS as data arrives, no buffering
6. **No rehashing** — server-provided digests used directly, skipping 33K SHA-512 computations
7. **No re-verification** — wrapped `fetchPackage` calls `readPkgFromCafs` with `verifyStoreIntegrity: false`
8. **Direct `writeFileSync` with `wx`** — no stat + temp + rename
9. **Pre-packed msgpack** — server sends raw store index buffers, client writes directly to SQLite
10. **WAL checkpoint** — ensures store index entries written by agent are visible to headless install's worker threads
## Usage
Start the server:
```bash
node agent/server/lib/bin.js
```
Configure in `pnpm-workspace.yaml`:
```yaml
agent: http://localhost:4873
```
|
||
|
Zoltan Kochan
|
9e0833c3cc |
feat: add minimumReleaseAgeIgnoreMissingTime setting (#11293)
Skips the minimumReleaseAge maturity check when the registry metadata lacks the "time" field, instead of throwing ERR_PNPM_MISSING_TIME. Defaults to true, and prints a warning once per affected package. |
||
|
Zoltan Kochan
|
2554264fdd |
perf: use NDJSON format for metadata cache (#11188)
The metadata cache files now use a two-line NDJSON format: - Line 1: cache headers (etag, modified, cachedAt) ~100 bytes - Line 2: raw registry metadata JSON (unchanged) This allows loadMetaHeaders to read only the first 1 KB of the file to extract conditional-request headers (etag, modified), avoiding the cost of reading and parsing multi-MB metadata files when the registry returns 200 and the old metadata would be discarded. Also moves cache directories to v11/ namespace (v11/metadata, v11/metadata-full, v11/metadata-full-filtered) since the format is not backwards compatible. |
||
|
Zoltan Kochan
|
968724fc0b |
perf: use abbreviated metadata for minimumReleaseAge (#11160)
* perf: use abbreviated metadata for minimumReleaseAge when possible Instead of always fetching full package metadata when minimumReleaseAge is set, fetch the smaller abbreviated document first and check the top-level `modified` field. If the package was last modified before the release age cutoff, all versions are mature and no per-version time filtering is needed. Only re-fetch full metadata for the rare case of recently-modified packages. Also uses fs.stat() to check cache file mtime instead of reading and parsing the JSON to check cachedAt, avoiding unnecessary I/O. * fix: validate modified date and handle abbreviated metadata edge cases - Validate meta.modified date to prevent invalid dates from bypassing minimumReleaseAge filtering - Skip full metadata refetch for packages excluded by publishedByExclude - Allow ERR_PNPM_MISSING_TIME from cached abbreviated metadata to fall through to the network fetch path instead of throwing * fix: cache abbreviated metadata before re-fetching full metadata Save the abbreviated metadata to disk before re-fetching full metadata so subsequent runs benefit from the mtime cache fast-path. * fix: resolve type narrowing for conditional metadata fetch result |
||
|
Zoltan Kochan
|
421d120972 |
perf: use If-Modified-Since for conditional metadata fetches (#11161)
Before fetching package metadata from the registry, stat the local cache file and send its mtime as an If-Modified-Since header. If the registry returns 304 Not Modified, read the local cache instead of downloading the full response body. This saves bandwidth and latency for packages whose metadata hasn't changed since the last fetch. Registries that don't support If-Modified-Since simply return 200 as before, so there is no behavior change for unsupported registries. |
||
|
Zoltan Kochan
|
6c480a4375 |
perf: replace node-fetch with undici (#10537)
Replace node-fetch with native undici for HTTP requests throughout pnpm. Key changes: - Replace node-fetch with undici's fetch() and dispatcher system - Replace @pnpm/network.agent with a new dispatcher module in @pnpm/network.fetch - Cache dispatchers via LRU cache keyed by connection parameters - Handle proxies via undici ProxyAgent instead of http/https-proxy-agent - Convert test mocking from nock to undici MockAgent where applicable - Add minimatch@9 override to fix ESM incompatibility with brace-expansion |
||
|
Alessio Attilio
|
d3d6938414 |
feat: add native view/info/show/v command (#11064)
* feat: add native view/info command * test: add unit tests for native view command * fix(view): support ranges, aliases, and tags * chore: update lockfile and tsconfig * refactor(view): reuse pickPackageFromMeta from npm-resolver - Share version resolution logic with the npm-resolver instead of reimplementing tag/range/version matching in the view command. - Export pickPackageFromMeta and pickVersionByVersionRange from @pnpm/resolving.npm-resolver. - Remove redundant double HTTP fetch (metadata already contains all version data). - Remove duplicate author/repository fields from PackageInRegistry (already inherited from BaseManifest). - Consolidate four changesets into one. - Revert unrelated .gitignore change. - Drop direct semver dependency from deps.inspection.commands. * refactor(view): reuse fetchMetadataFromFromRegistry from npm-resolver Use the npm-resolver's fetchMetadataFromFromRegistry instead of hand-rolled fetch logic. This fixes: - Broken URL encoding for scoped packages (@scope/pkg) - Missing auth header, proxy, SSL, and retry config - Duplicated fetch + error handling code Also pass proper Config options (rawConfig, userAgent, SSL, proxy, retry, timeout) through to createFetchFromRegistry and createGetAuthHeaderByURI so the view command works with private registries and corporate proxies. * test(view): improve test coverage for view command Add tests for: - non-registry spec rejection (git URLs) - no matching version error - version range resolution (^1.0.0) - dist-tag resolution (latest) - nested field selection (dist.shasum) - field selection with --json - text output format (header, dist section, dist-tags) - scoped package lookup (@pnpm.e2e/pkg-with-1-dep) - deps count / deps: none in header - object field rendering as JSON * revert: undo rename of @pnpm/resolving.registry.types The rename from @pnpm/resolving.registry.types to @pnpm/registry.types (and the move from resolving/registry/types/ to registry/types/) is a separate refactoring concern unrelated to the view command. Revert all rename-related changes. Keep the legitimate type additions to PackageInRegistry: maintainers, contributors, and dist.unpackedSize. * revert: restore pnpm-workspace.yaml (remove registry/* glob) * fix(view): handle edge cases in formatBytes and unpackedSize - Use explicit null check for unpackedSize so 0 B is still rendered - Add TB/PB units and clamp index to prevent undefined output --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Brandon Cheng
|
6557dc09f9 |
fix: clearCache function in @pnpm/resolving.npm-resolver (#11050)
* test: add test for `clearCache` function in `@pnpm/resolving.npm-resolver` * fix: clear pMemoize when clearing NPM resolver `clearCache` function |
||
|
Brandon Cheng
|
831f574330 |
fix: propagate error cause when throwing PnpmError in @pnpm/npm-resolver (#10990)
* fix: show error cause when failing to read metadata * fix: correct changeset package name and add cause assertion tests - Fix changeset to reference @pnpm/resolving.npm-resolver (not @pnpm/npm-resolver) - Add PnpmError cause unit tests in @pnpm/error - Fix npm-resolver tests to actually verify cause on thrown errors (.toThrow() only checks message, not cause/hint/code properties) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Zoltan Kochan <z@kochan.io> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Zoltan Kochan
|
0d88df854f |
chore: update all dependencies to latest versions (#11032)
* chore: update all dependencies to latest versions Update all outdated dependencies across the monorepo catalog and fix breaking changes from major version bumps. Notable updates: - ESLint 9 → 10 (fix custom rule API, disable new no-useless-assignment) - @stylistic/eslint-plugin 4 → 5 (auto-fixed indent changes) - @cyclonedx/cyclonedx-library 9 → 10 (adapt to removed SPDX API) - esbuild 0.25 → 0.27 - TypeScript 5.9.2 → 5.9.3 - Various @types packages, test utilities, and build tools Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update unified/remark/mdast imports for v11/v4 API changes Update imports in get-release-text for the new ESM named exports: - mdast-util-to-string: default → { toString } - unified: default → { unified } Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: resolve typecheck errors from dependency updates - isexe v4: use named import { sync } instead of default export - remark-parse/remark-stringify v11: add vfile as packageExtension dependency so TypeScript can resolve type declarations - get-release-text: remove unused @ts-expect-error directives Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert runtime dependency major version bumps Revert major version bumps for runtime dependencies that are bundled into pnpm to fix test failures where pnpm add silently fails: - bin-links: keep ^5.0.0 (was ^6.0.0) - cli-truncate: keep ^4.0.0 (was ^5.2.0) - delay: keep ^6.0.0 (was ^7.0.0) - filenamify: keep ^6.0.0 (was ^7.0.1) - find-up: keep ^7.0.0 (was ^8.0.0) - isexe: keep 2.0.0 (was 4.0.0) - normalize-newline: keep 4.1.0 (was 5.0.0) - p-queue: keep ^8.1.0 (was ^9.1.0) - ps-list: keep ^8.1.1 (was ^9.0.0) - string-length: keep ^6.0.0 (was ^7.0.1) - symlink-dir: keep ^7.0.0 (was ^9.0.0) - terminal-link: keep ^4.0.0 (was ^5.0.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restore runtime dependency major version bumps Re-apply all runtime dependency major version bumps that were previously reverted. All packages maintain their default exports except isexe v4 which needs named imports. Updated runtime deps: - bin-links: ^5.0.0 → ^6.0.0 - cli-truncate: ^4.0.0 → ^5.2.0 - delay: ^6.0.0 → ^7.0.0 - filenamify: ^6.0.0 → ^7.0.1 - find-up: ^7.0.0 → ^8.0.0 - isexe: 2.0.0 → 4.0.0 (fix: use named import { sync }) - normalize-newline: 4.1.0 → 5.0.0 - p-queue: ^8.1.0 → ^9.1.0 - ps-list: ^8.1.1 → ^9.0.0 - string-length: ^6.0.0 → ^7.0.1 - symlink-dir: ^7.0.0 → ^9.0.0 - terminal-link: ^4.0.0 → ^5.0.0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert tempy to 3.0.0 to fix bundle hang tempy 3.2.0 pulls in temp-dir 3.0.0 which uses async fs.realpath() inside its module init. When bundled by esbuild into the __esm lazy init pattern, this causes a deadlock during module initialization, making the pnpm binary hang silently on startup. Keeping tempy at 3.0.0 which uses temp-dir 2.x (sync fs.realpathSync). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add comment explaining why tempy cannot be upgraded Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert nock to 13.3.4 for node-fetch compatibility nock 14 changed its HTTP interception mechanism in a way that doesn't properly intercept node-fetch requests, causing audit tests to hang waiting for responses that are never intercepted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add comment explaining why nock cannot be upgraded Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update symlink-dir imports for v10 ESM named exports symlink-dir v10 removed the default export and switched to named exports: { symlinkDir, symlinkDirSync }. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert @typescript/native-preview to working version Newer tsgo dev builds (>= 20260318) have a regression where @types/node cannot be resolved, breaking all node built-in types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: vulnerabilities * fix: align comment indentation in runLifecycleHook Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: pin msgpackr to 1.11.8 for TypeScript 5.9 compatibility msgpackr 1.11.9 has broken type definitions that use Iterable/Iterator without required type arguments, causing compile errors with TS 5.9. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Zoltan Kochan
|
4a36b9a110 |
refactor: rename internal packages to @pnpm/<domain>.<leaf> convention (#10997)
## Summary Rename all internal packages so their npm names follow the `@pnpm/<domain>.<leaf>` convention, matching their directory structure. Also rename directories to remove redundancy and improve clarity. ### Bulk rename (94 packages) All `@pnpm/` packages now derive their name from their directory path using dot-separated segments. Exceptions: `packages/`, `__utils__/`, and `pnpm/artifacts/` keep leaf names only. ### Directory renames (removing redundant prefixes) - `cli/cli-meta` → `cli/meta`, `cli/cli-utils` → `cli/utils` - `config/config` → `config/reader`, `config/config-writer` → `config/writer` - `fetching/fetching-types` → `fetching/types` - `lockfile/lockfile-to-pnp` → `lockfile/to-pnp` - `store/store-connection-manager` → `store/connection-manager` - `store/store-controller-types` → `store/controller-types` - `store/store-path` → `store/path` ### Targeted renames (clarity improvements) - `deps/dependency-path` → `deps/path` (`@pnpm/deps.path`) - `deps/calc-dep-state` → `deps/graph-hasher` (`@pnpm/deps.graph-hasher`) - `deps/inspection/dependencies-hierarchy` → `deps/inspection/tree-builder` (`@pnpm/deps.inspection.tree-builder`) - `bins/link-bins` → `bins/linker`, `bins/remove-bins` → `bins/remover`, `bins/package-bins` → `bins/resolver` - `installing/get-context` → `installing/context` - `store/package-store` → `store/controller` - `pkg-manifest/manifest-utils` → `pkg-manifest/utils` ### Manifest reader/writer renames - `workspace/read-project-manifest` → `workspace/project-manifest-reader` (`@pnpm/workspace.project-manifest-reader`) - `workspace/write-project-manifest` → `workspace/project-manifest-writer` (`@pnpm/workspace.project-manifest-writer`) - `workspace/read-manifest` → `workspace/workspace-manifest-reader` (`@pnpm/workspace.workspace-manifest-reader`) - `workspace/manifest-writer` → `workspace/workspace-manifest-writer` (`@pnpm/workspace.workspace-manifest-writer`) ### Workspace package renames - `workspace/find-packages` → `workspace/projects-reader` - `workspace/find-workspace-dir` → `workspace/root-finder` - `workspace/resolve-workspace-range` → `workspace/range-resolver` - `workspace/filter-packages-from-dir` merged into `workspace/filter-workspace-packages` → `workspace/projects-filter` ### Domain moves - `pkg-manifest/read-project-manifest` → `workspace/project-manifest-reader` - `pkg-manifest/write-project-manifest` → `workspace/project-manifest-writer` - `pkg-manifest/exportable-manifest` → `releasing/exportable-manifest` ### Scope - 1206 files changed - Updated: package.json names/deps, TypeScript imports, tsconfig references, changeset files, renovate.json, test fixtures, import ordering |
||
|
Zoltan Kochan
|
5d5818e44f |
style: enforce node: protocol for builtin imports (#10951)
Add n/prefer-node-protocol rule and autofix all bare builtin imports to use the node: prefix. Simplify the simple-import-sort builtins pattern to just ^node: since all imports now use the prefix. |
||
|
Zoltan Kochan
|
1c8c4e49f5 |
style: add eslint-plugin-simple-import-sort (#10947)
Add eslint-plugin-simple-import-sort to enforce consistent import ordering: - Node.js builtins first - External packages second - Relative imports last - Named imports sorted alphabetically within each statement |
||
|
Dami Oyeniyi
|
61cad0cdbc | fix: treat HTTP 400 responses as errors in npm resolver fetch (#10945) | ||
|
Jason Paulos
|
15549a9445 |
feat(audit): add fix update mode (#10341)
* feat(audit): add fix update mode Add the ability to fix vulnerabilities by updating packages in the lockfile instead of adding overrides. * revert: remove audit-registry parameter * fix: properly invoke audit command recursively on workspace * fix: negative weight version priority & top-level pinned dep updating * refactor: apply packageVulnerabilityAudit version preferences earlier * chore: update changeset * fix: vulnerability penalties are greater than direct dep weight * test: use nock on mock registry directly * fix: exit with 1 if it can't resolve all vulnerabilities to match npm * fix: properly update workspace top-level pinned vulnerable dependencies * fix: update lockfile * fix: update vulnerabilities in catalogs * chore: sync pnpm-lock.yaml with main |
||
|
Zoltan Kochan
|
bb177242df |
feat: support devEngines.packageManager for pnpm version management (#10932)
## Summary - Support specifying the pnpm version via `devEngines.packageManager` in `package.json`, as an alternative to the `packageManager` field - Unlike `packageManager`, `devEngines.packageManager` supports semver ranges — the resolved version is stored in `pnpm-lock.env.yaml` and reused if it still satisfies the range - The `onFail` field determines behavior: `download` (auto-download), `error` (default), `warn`, or `ignore` - `devEngines.packageManager` takes precedence over `packageManager` when both are present (with a warning) - For array notation, default `onFail` is `ignore` for non-last elements and `error` for the last - For the legacy `packageManager` field, `onFail` is derived from existing config settings (`managePackageManagerVersions`, `packageManagerStrict`, `packageManagerStrictVersion`), so `main.ts` uses `onFail` as the single source of truth - Reuses `EngineDependency` type from `@pnpm/types` instead of a custom `WantedPackageManager` type ## Test plan - [x] 10 tests in `switchingVersions.test.ts` — version switching with `packageManager` field, `devEngines.packageManager` with `onFail=download` (exact + range), env lockfile reuse, corrupt binary - [x] 15 tests in `packageManagerCheck.test.ts` — version checks with `engines.pnpm`, `packageManager` field, `devEngines.packageManager` with all `onFail` values, array notation, range matching, precedence close https://github.com/pnpm/pnpm/issues/8153 |
||
|
Zoltan Kochan
|
3a5bfaa94f |
chore: update zkochan packages to latest versions (#10930)
Update all packages from zkochan/packages to their latest major versions and exclude them from minimumReleaseAge requirement. This includes updating catalog entries, adapting to breaking API changes (default exports replaced with named exports, sync functions renamed with Sync suffix), and updating type declarations. |
||
|
Brandon Cheng
|
01914345d5 |
build: enable @typescript-eslint/no-import-type-side-effects (#10630)
* build: enable `@typescript-eslint/no-import-type-side-effects` * build: disable `@typescript-eslint/consistent-type-imports` * chore: apply fixes for `no-import-type-side-effects` pnpm exec eslint "**/src/**/*.ts" "**/test/**/*.ts" --fix |
||
|
Zoltan Kochan
|
5a15a32d84 |
feat: use JSON for npm registry metadata cache instead of msgpack (#10886)
* feat: use JSON for npm registry metadata cache instead of msgpack Switch the on-disk package metadata cache from msgpack (.mpk) to JSON (.json). When metadata is not filtered, the raw JSON response from the registry is written directly to disk with cachedAt injected, avoiding a parse-then-serialize round-trip. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update lockfileOnly test to use .json metadata extension Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update resolving/npm-resolver/src/pickPackage.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Zoltan Kochan
|
b7f0f21582 |
feat: use SQLite for storing package index in the content-addressable store (#10827)
## Summary Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries. Closes #10826 ## Design ### New package: `@pnpm/store.index` A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage. Key design decisions: - **WAL mode** enables concurrent reads from workers while the main process writes. - **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access. - **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`. - **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`). - **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits. - **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space. ### Key format Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`. ### Shared StoreIndex instance A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access. ### Worker architecture Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads. ### SQLite ExperimentalWarning suppression `node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`. ### No migration from `.mpk` files Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store). ## Changed packages 121 files changed across these areas: - **`store/index/`** — New `@pnpm/store.index` package - **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process - **`store/package-store/`** — StoreIndex creation and lifecycle management - **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed) - **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files - **`store/plugin-commands-store/`** — `store status` uses StoreIndex - **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex - **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading - **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter - **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex - **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex - **`cache/api/`** — Cache view uses StoreIndex - **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression ## Test plan - [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching - [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle - [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index - [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes - [x] `pnpm --filter @pnpm/headless test` — Headless install - [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching - [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index - [x] `pnpm --filter @pnpm/license-scanner test` — License scanning - [x] e2e tests pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
roysandrew
|
143ca78d09 |
fix(npm-resolver): respect version constraints when falling back to workspace packages (#10704)
* fix(npm-resolver): respect version constraints when falling back to workspace packages When link-workspace-packages=true, the fallback resolution paths (registry 404 and no matching registry version) pass update: Boolean(opts.update) to tryResolveFromWorkspacePackages. On fresh installs without a lockfile entry, opts.update is 'compatible' (truthy), which overrides the version spec to '*' and matches any workspace package regardless of version. Change both fallback call sites to pass update: false so version constraints are always respected for non-workspace-protocol dependencies. The workspace: protocol path returns before these blocks and correctly continues to use opts.update. Close #10173 * test: clarify npm-resolver test names for workspace version mismatch scenarios --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
56a59df674 |
perf: persist bundled manifest in store index to avoid reading package.json from CAFS (#10473)
close #10461 |
||
|
Brandon Cheng
|
0e9c559068 |
fix: remove misleading maxAge argument to pMemoize (#10620)
|
||
|
Shunta Takemoto
|
0625e20442 |
feat: treat bare workspace: protocol as workspace:* (#10436)
* feat: treat bare `workspace:` protocol as `workspace:*` * chore: add chageset * test(exportable-manifest): add test for `workspace` with explicit versions * test: add tests and update changesets --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |