`extract_main_document` / `extract_env_document` split the combined
`pnpm-lock.yaml` on literal `---\n` and `\n---\n`. A lockfile pacquet
did not write may carry CRLF line endings — a `core.autocrlf` checkout
on Windows — or a UTF-8 BOM, and neither marker then matches. The main
extractor fell through to returning the whole file, which serde rejected
as `multiple YAML documents detected`, so the lockfile was discarded and
every dependency re-resolved from the registry.
Normalize the whole file (strip BOM, CRLF to LF) inside both extractors,
which covers every caller. `save_value_to_path` already normalized the
same way inline and now shares the helper.
The TypeScript stack needs no counterpart change:
`pnpm11/lockfile/fs/src/yamlDocuments.ts` already normalizes CRLF in
`extractMainDocument` / `extractEnvDocument` and strips the BOM at each
read site.
Closes https://github.com/pnpm/pnpm/issues/13606
pacquet's pnpm login / pnpm adduser honored scope only as the --scope
command-line flag, ignoring it when configured. The TypeScript pnpm CLI
resolves scope through the config chain, so pacquet was missing the
pnpm-workspace.yaml, global config.yaml, and PNPM_CONFIG_SCOPE sources.
Add a scope field to Config, populated from pnpm-workspace.yaml and the
global config.yaml (via WorkspaceSettings) and from PNPM_CONFIG_SCOPE (via
from_pnpm_config_env). The login CLI adapter falls back to the resolved
config.scope when --scope is absent, with the flag still taking
precedence.
.npmrc is deliberately excluded: pnpm's readAndFilterNpmrc keeps only
auth/registry keys (isNpmrcReadableKey), and scope is in neither
NPM_AUTH_SETTINGS nor NETWORK_INI_KEYS, so pnpm drops a `scope=` line
there. A test pins that pacquet ignores it too.
Fixes https://github.com/pnpm/pnpm/issues/12893
Version tags were created in CI with `git tag "$tag" "$SHA"` — a lightweight tag,
which has no tag object to carry a signature. Since v11.18.0 that made
`git verify-tag` fail with "cannot verify a non-tag object of type commit",
breaking downstream packagers who verify the tag before building.
Signing those tags in CI would restore verify-tag but not the property that makes
it worth verifying: the key would live in Actions secrets, inside the same trust
boundary the signature is meant to attest independently of. Cut tags from a
maintainer machine instead, and have release.yml verify the tag before any
publish job runs.
verify-release-tag imports the public keys committed under .github/release-keys
into a throwaway GNUPGHOME and runs `git verify-tag` on the pushed tag. Every
publish path descends from `plan` and `plan` descends from that job, so one edge
gates all three products. Verified against a signed tag, a lightweight tag, an
annotated unsigned tag, and a tag signed by a key outside the directory: only the
first proceeds.
The keyring is isolated so verification cannot succeed against a key already on
the runner, and checkout sets fetch-tags because a tag push checks out the commit
while the signature lives on the tag object. The non-tag verify_ts_release
dispatch exits from inside the step rather than a job-level `if`, since a skipped
job skips everything that needs it.
This guards against unsigned tags, not against an actor who can push arbitrary
ones: a tag push runs the workflow file and checks out the tree from the pushed
tag. Restricting creation of v* and pnpr@* tags is a repository ruleset concern.
RELEASING.md documents the manual procedure: tag an explicit commit, verify every
tag before pushing, and retag at the fixed commit to retry a failed release.
Related to pnpm/pnpm#13578
Treat empty proxy values as unset.
Every source that can configure a proxy (`.npmrc` keys, env vars, CLI
flags, workspace YAML) can produce an empty string; previously only
`parse_proxy_url` rejected it, surfacing as ERR_PNPM_INVALID_PROXY on
every install.
Filtering empty values only where the resolved ProxyConfig is parsed
into the HTTP client (`for_installs_with_redirect`) stops the abort but
leaves an empty value shadowing every lower-priority source. Empty
proxy settings are now dropped as the .npmrc, pnpm-workspace.yaml, and
CLI layers are applied, with the filter at the client-build site kept as
the final normalization. An empty env var keeps shadowing the env vars
below it, and what is left resolves to no proxy.
The legacy `.npmrc` `proxy=` key was the one setting where the two
config-reader paths disagreed: the trusted packageManagerNetworkConfig
path ran it through `getProxyValue` and fell through on empty, while the
main path kept `''` and let a project .npmrc suppress an
environment-configured proxy. Both paths now use `getProxyValue`, and
pacquet's cascade mirrors it.
Also fix the workspace-yaml `noProxy` / `noproxy` aliases: an empty
primary key consumed the alias's turn before being filtered, discarding
a valid bypass list.
Supersedes the branch of pnpm/pnpm#13582, which had maintainer edits
disabled.
Fixespnpm/pnpm#13533.
From v12 the unscoped `pnpm` package is itself the native executable, so
`@pnpm/exe` is no longer published alongside it. The env lockfile now pins
only `pnpm` for those versions, in both stacks, instead of resolving a
package that does not exist.
The native-binary leg of the engine identity check followed `@pnpm/exe`
by name, so it silently verified nothing once that package is absent. It
now follows whichever package carries the host's platform binary as an
optional dependency: `@pnpm/exe` for the majors that publish it, the
unscoped `pnpm` from v12.
On the TypeScript side the wanted version can arrive as a range or a
dist-tag (`pnpm with latest`), so `pnpm` is resolved first and the second
resolution only happens for the versions that do ship `@pnpm/exe`.
GHSA-c59q-g84q-2gj5 reported that the isolated linker joined the
lockfile-derived package name onto the virtual store without validation.
That site, and the PnP `packageLocation`, were already contained by
51300fd41c; the one recommended site that commit missed is the rebuild
phase, which resolves its package root the same way.
`_rebuild` reads the current lockfile from `node_modules` and rebuilds
`pkgRoot` as `path.join(pkgModulesDir(depPath), pkgInfo.name)`, where
`pkgInfo.name` comes from `dp.parse` of the depPath key. `dp.parse` is a
raw substring and validates nothing, so a `../../../escaped@1.0.0` key
pointed the lifecycle-script runner and the post-build bin linking at a
directory outside the virtual store. The dep graph that phase walks comes
from `@pnpm/deps.graph-hasher`, which deliberately defers the name check
to its sink -- and this sink had none.
Route both joins through `safeJoinModulesDir`, so a crafted name fails
with ERR_PNPM_INVALID_DEPENDENCY_NAME before any script runs.
Reachability is narrower than the reported install path: the poisoned
lockfile has to already be in `node_modules`, which means a shipped or
restored `node_modules` rather than a committed `pnpm-lock.yaml`.
No pacquet change: its `rebuild` drives the frozen-install pipeline, which
runs `verify_lockfile_dependency_names` unconditionally over snapshot
package names, and it reads the project lockfile rather than a copy in
`node_modules`, so it has no equivalent unguarded sink.
Two installs of one workspace, same inputs and same binary, emitted
different lockfiles: a peer suffix would bind `postcss@8.4.19` in one
run and `8.5.25` in the next. https://github.com/pnpm/pnpm/pull/13584
removed most of it — 196 differing lines where there had been 10,390 —
but not the last of it.
Bisecting what varies between runs: the resolved package set and every
package's children are identical, hash for hash. The occurrence nodes
are not, ranging from 58,972 to 60,664 on a 114-importer workspace.
Importers race for a package's children-ownership claim, and a walk
holding the claim only transiently still leaves its occurrence nodes in
the shared tree; how many it leaves depends on the interleaving.
Occurrence identity feeds peer-variant computation, so that is enough
to move a binding.
Resolve the initial waves one importer at a time, boxing each so the
enclosing install future does not grow by a wave's frame. Eight
consecutive replays of that workspace now emit byte-identical
lockfiles, where before almost every pair differed, and the result
matches one of the two lockfiles the concurrent path was alternating
between — no resolution changes, the choice simply stops depending on
scheduling.
It is also faster wherever it was measured, because the races cost
redundant subtree walks: that workspace resolves in ~9.3s against
~10.3s, a cold install over a 50ms link takes 2.64s against 2.72s, and
the resolution fixtures are unchanged. The waves overlapped resolver
waits, but pacquet dedups those through the workspace-wide wanted-dep
cache, so there was less to overlap than the fan-out assumed.
Closes https://github.com/pnpm/pnpm/issues/13567.
The built-in aliases and the user's `namedRegistries` were combined
independently at each place that needed them: `resolveNamedRegistries` in
constants, a second call inside the same verifier factory to build the URL
prefix list, and in pacquet a hand-rolled `chain` next to a separate
`build_named_registry_prefixes`. The Rust pair had already drifted — the
hand-rolled merge skipped the URL validation the other applied.
That set feeds two different consumers: alias lookup for an entry that names
its registry in the dep path, and the tarball-URL prefix list that decides
which registry an entry carrying only a recorded URL is verified against.
Splitting the merge across call sites is what made adding the `npmjs`
built-in look local while it silently redirected verification traffic for
lockfiles that record a registry.npmjs.org URL and name no alias at all.
That consequence was caught by an unrelated normalization test that happened
to use such a URL, not by anything at the edit site.
`KnownRegistries` now owns the merge and exposes both views, so a change to
either input reaches both consumers or neither. `createKnownRegistries`
computes the prefixes lazily because alias lookup runs per package while
prefix matching runs only for entries with a recorded URL.
Both stacks now pin the default prefix list in a test, so a new built-in
cannot land without stating where it sends verification traffic.
Writing that test surfaced a latent bug: the two built-in URLs are both 27
characters, and sorting by length alone left their relative order to hash-map
iteration, which varies between runs. The sort now tie-breaks
lexicographically in both stacks. No routing outcome changes — two prefixes
of equal length cannot both match one URL unless they are identical — but the
list is now stable enough to assert on.
Closes https://github.com/pnpm/pnpm/issues/13590
pnpm 11 rewrites `--prefix` to `--dir` and `--store` to `--store-dir`
before parsing argv (`RENAMED_OPTIONS` in `parseCliArgs.ts`), so npm
habits and existing CI scripts keep working. pacquet had no equivalent
and clap rejected the unknown long flag outright, breaking invocations
like `pnpm --prefix ../ run test:ci`.
Both spellings are added as clap aliases on the global options rather
than as an argv-rewriting pass: the pre-clap passes derive their arity
tables from the clap `Command` and already fold in every alias, so
`--prefix` relocates and steps over its value like `--dir` does. The
aliases are hidden so the help output stays identical to pnpm 11's,
which does not document either spelling.
pnpm additionally drops the alias when a command line spells the same
option both ways, so `--prefix foo --dir bar` and `--dir bar --prefix
foo` both resolve to `bar`. Clap keeps the last occurrence instead, so
`renamed_options` removes the shadowed alias tokens ahead of the parse,
alongside the other pre-clap passes and using the same clap-derived
arity tables.
The `--version` pre-scan reads `--dir` directly to find the project
whose pin decides the delegated version. It now honors `--prefix` there
too, and takes option arity from the clap grammar instead of a
hand-maintained list of value-taking options: the list did not know
about `--store-dir`, so `pnpm --store-dir /s --version` took `/s` for
the command name and stopped scanning.
The clap grammar is built once per process rather than per caller.
Constructing it walks every subcommand, and every pass that runs before
the parse needs the same view of it, so the boundary scan no longer
rebuilds one on each call.
Closespnpm/pnpm#13583
reqwest's `rustls-platform-verifier` backend hard-errors with
"No CA certificates were loaded from the system" when the platform
store yields nothing, and the install client turned that into a
`Main thread panicked` through an `.expect` on `ClientBuilder::build`.
Any install on such a machine — a nixpkgs sandbox, a scratch
container — died, offline installs included, even though pnpm 11 works
there because Node ships its own Mozilla root store.
Keep the platform verifier as the primary trust source and fall back to
the Mozilla roots bundled via `webpki-root-certs` only when the client
cannot be built with it at all. A user who deliberately distrusts a root
system-wide keeps that decision as long as their store loads, and a
build failure for any other reason now surfaces as
`ForInstallsError::ClientBuild` — carrying both that error and the
retry's own — instead of a panic.
pacquet-only: the TypeScript CLI is unaffected, since Node's bundled
root store is what makes pnpm 11 work on these systems.
Closespnpm/pnpm#13588
The cycle graph the final depPath pass consults is over package names,
of which a workspace has orders of magnitude fewer than the occurrences
contributing to it — on the reference workspace, ~5,000 packages against
224,946 occurrences with external peers. It was built by deriving the
name from every one of those occurrences, and `pkg_name_version`
allocates both the name and a version this caller discards, so the graph
cost two allocations per occurrence and its keys and edges cost another
`String` clone each.
Derive the name once per package id and borrow throughout: the graph,
the peer-name list that seeds its isolated nodes, and the Tarjan pass
over it all hold `&str` now. Only the returned cyclic set owns its
names, because it outlives the graph.
Same graph, same cycles. The effect is ~1-2% on the full-size
resolution fixture and below this workspace's run-to-run noise on the
reference workspace; what it removes is a per-occurrence term in a pass
whose result depends only on the package set.
Related to https://github.com/pnpm/pnpm/issues/13505.
Packages resolved from a named registry are now keyed
`<name>@<registryName>:<version>` (e.g. `foo@work:1.0.0`) instead of
`<name>@<version>`. Without the qualifier the same name+version served by two
different registries collapsed onto one `packages:` entry, so whichever
resolved first won and the other consumer silently got the wrong tarball.
The lockfile also carried no registry marker at all: entry lookup keys on
`refToRelative(ref, name)`, so a `work:foo` spec could be satisfied by an
entry that was really resolved from npmjs, and vice versa.
The alias goes in the version slot because the dep-path grammar already
admits colon-bearing non-semver values there (`foo@file:...`,
`node@runtime:...`), so every suffix mechanism composes unchanged:
`foo@work:1.0.0(react@18.0.0)` parses with the existing balanced-paren scan,
`depPathToFilename` escapes the colon to `+`, and `refToRelative` already
reconstructs `foo@work:1.0.0` from a `work:1.0.0` snapshot ref with no
change. The alias is recorded rather than the registry URL so keys stay
readable and independent of registry configuration; the concrete URL is
still pinned by the resolution when it is not derivable.
Tarball URLs now apply the pre-existing "omit whatever is reconstructible"
rule against the named registry instead of the scope-routed default, so a
canonical named-registry tarball serializes as bare `{integrity}` and is
recomputed from the alias on read. Registries whose URLs cannot be rebuilt
from name, version, and base URL (GitHub Packages hashed `/download/` URLs)
keep the explicit field, exactly as before.
The lockfile format version is unchanged. Registry-qualified keys are
additive: they appear only for packages resolved through `namedRegistries`,
the tarball-omission rule only fires on those entries, and the alias
validation is config-time, so a project that configures no named registry
produces a byte-identical lockfile. A format bump would not have been
adoptable incrementally — readers gate on the major version alone and reject
an unknown one outright rather than degrading, including pnpm engines
embedded in other tools, which cannot be patched after the fact.
Named-registry aliases that would shadow a reserved specifier prefix
(file, link, workspace, runtime, npm, jsr, and so on) are now rejected at
resolver construction with ERR_PNPM_RESERVED_NAMED_REGISTRY_NAME. They were
previously accepted and silently shadowed by that prefix's own resolver in
the chain, which was harmless while the alias only appeared in specifiers;
in the version slot it would make the dep path genuinely ambiguous.
Deciding whether a missing-peer issue is covered by another importer's
walk scans the issue's ancestor chain for a package that importer owns.
The scan ran per issue, and issues of one peer come from occurrences
spread through the tree whose chains share long suffixes, so the same
suffix was rescanned once per issue: O(issues x depth) per importer,
with the issue count growing with the workspace.
SharedChain already shares those suffixes structurally, so the answer
for a link covers that link and everything above it, for every chain
built on it. `any_memoized` records that answer per link, taking the
total to the number of distinct links; a suffix that already matched
spares each branch's own tip as well.
The post-round scope filter memoizes, per peer name; the walk's own
suppression check does not, because a memo outliving its links cannot
be keyed on their addresses and the walk drops chains throughout. Each
entry holds an `Arc` to the link it is keyed on, so an address cannot
be recycled underneath it.
Output is unchanged. On the full-size workspace resolution fixture,
whose chains are 250 deep, resolution goes from ~1.56s to ~1.19s, and
the CI-sized fixture moves 3%. It is neutral on the 114-importer
bit.cloud workspace, whose chains are too shallow for the rescan to
dominate: the term this removes scales with the issue count, but only
in proportion to chain depth.
Related to https://github.com/pnpm/pnpm/issues/13505.
pnpm v12 shipped no node-gyp, so any package that shells out to it during
an install script died with `spawn node-gyp ENOENT`: `node-gyp-build` with
no matching prebuild, `node-pre-gyp`, a plain `"install": "node-gyp
rebuild"`, and the `node-gyp rebuild` script pnpm synthesizes for a package
that ships a `binding.gyp` without one. Embedders worked around it by
putting their own node-gyp on the lifecycle PATH.
pnpm 11 solves this without a JS bundle: node-gyp is declared `external`,
and `pnpm deploy` resolves its whole tree from this repo's lockfile at
release time into a `dist/node_modules/node-gyp` directory that sits beside
the entry point, with `dist/node-gyp-bin` holding the wrappers that go on a
lifecycle script's PATH. Those are ordinary files next to the executable,
so the native binary can ship them the same way — nothing about the
mechanism depended on there being a bundle to hide node-gyp inside.
Resolving the tree at release time rather than at install time is the point,
not an implementation detail: the dependency tree is frozen and reviewed per
pnpm release instead of being fetched from the registry at the moment an
install script is about to execute arbitrary code.
scripts/bundle-node-gyp.mjs builds the payload, reusing the TypeScript CLI's
wrapper scripts so the two stacks cannot drift on how a script's node-gyp is
dispatched, and refusing to produce a partial dist/ — a PATH entry that
resolves nothing is worse than shipping none at all. Both wrappers ship it,
since each places the binary next to itself.
node-gyp moves to a catalog entry so both stacks pin one version; the
Node.js-floor hold-back note moves with it.
At runtime bundled_node_gyp_bin finds the wrapper dir relative to the
executable and returns None when nothing was shipped, which is what a
checkout build sees — lifecycle scripts then fall back to whatever node-gyp
the environment provides, exactly as before. The wrapper itself honors
npm_config_node_gyp and only falls back to the shipped copy when it is
unset, so a user-supplied node-gyp, a workspace one, and a package's own
dependency all keep winning.
Refs https://github.com/pnpm/pnpm/issues/13580
Canonicalize the child specs extracted from resolved manifests before the
dependency walk. Manifest fetches run concurrently and equivalent JSON maps
can arrive with different insertion orders. Letting that order reach lazy
child discovery changes internal node-ID assignment, which can alter the peer
providers selected later in large workspaces.
Add a regression test proving that reversed dependency property order yields
the same child sequence. Also correct an existing changeset wording issue
found by the repository pre-push spellcheck.
Closespnpm/pnpm#13567.
Two per-importer rescans of the whole workspace came out of the
peer-hoist loop, which made a hoist wave cost the square of the
workspace rather than the workspace.
Every round of every importer cloned `first_importer_by_pkg` and
`first_walk_missing_by_pkg` whole to build its `HoistMissingScope`.
Each map now carries the `Arc` snapshot last projected from it and
hands that out until a write invalidates it; the two writers that
usually change nothing — an ownership claim re-recording the importer
that already owns the package, and a later round re-recording a
missing-peer report the same owner generation already wrote — test
before writing, so a round that changes nothing reuses the projection.
Snapshots already issued keep what they were built from.
Each discovery pass also folded its subtree summaries into an owned
`pkg id -> missing peer names` map, cloning every descendant's package
id and name set into it, for one consumer that read the map and
dropped it: on the benchmark below, 1.6M entries built and 1.1s spent
freeing them. The summaries are already shared bottom-up by `Arc`, so
a pass now reports its roots and the reader indexes them in place. The
index borrows the names, which makes it free to drop, and the union
several occurrences of one package require is materialized only when a
second summary reports that package.
The hoist-heavy `workspace_full_resolution` scenario (331 importers,
5,030 packages) drops from ~6.7s to ~1.50s; the two scenarios that
never enter the hoist loop are unchanged. On the 114-importer
bit.cloud workspace from the issue the gain is ~0.2s of a 10.4s
resolve, since its importers mostly have their peers provided and its
per-round maps are small.
Related to https://github.com/pnpm/pnpm/issues/13505.
The hoist-heavy resolution fixture is the only benchmark that measures
the peer-hoist loop — the importer barrier, the per-round context syncs,
and the per-importer work that scales with the workspace. Nothing ran it
in CI: it lived as a harness-free bench in the resolver crate, and the
scenarios the integrated benchmark does run converge in one hoist round,
so none of them moves when that loop's cost changes. An A/B of the
40-importer integrated-benchmark fixture measures 820.6ms against
819.0ms across a change that makes the loop 4.4x faster.
Move the fixture into the micro-benchmark task, which already builds
both sides of a pull request and posts their comparison, and give it two
sizes. The CI size — 120 importers over 1,200 packages — runs under
criterion on every PR at ~0.2s a resolve; the full size, which models the
331-importer workspace the original regression came from, keeps its
single timed run behind `--full-workspace-resolution`, since a regressed
run there can take minutes and several GiB and has no business in a
statistical harness.
Only the hoist-heavy shape is benched. The plain and peer-heavy shapes
re-measure the tree walk that other scenarios already cover, and with
every peer provided up front their hoist loop converges in one round.
The CI size keeps the signal: across the same change, the group moves
577ms -> 219ms (-62%, p = 0.00).
Related to https://github.com/pnpm/pnpm/issues/13505.
Winning the children-ownership claim is what licenses an occurrence to
walk a package's manifest children. Concurrent occurrences of one
package take turns winning it — each turn re-walked the whole subtree
and left its occurrence nodes in the shared tree — so a chain of
peer-carrying packages expanded exponentially with its depth. One
project with one such dependency, over a graph of ~500 packages, could
not finish inside 6 GB; at depth 22 it peaked at 4.1 GB. The TypeScript
CLI resolves the same input in 1.06s and 263 MB.
Winning the claim only means this occurrence outranks the one that
recorded the children, not that the children differ. When the displaced
owner resolved them under conditions this occurrence would reproduce —
the same peer-shadowed set, the same update policy, the same
prior-lockfile key — expand from the recorded children instead, exactly
as a losing occurrence does. Each term of that test is load-bearing: an
update-active occurrence re-resolves what a keep-all one reused, and an
occurrence pinned by a different snapshot key resolves different
children. Packages whose child specifiers pass through the importer's
catalogs are excluded as well, being a property of the resolving
importer rather than of the package id.
On the generated chain from the issue, one project and one dependency,
depth 15 falls from 198,023 occurrence nodes / 278 MB / 1.00s to 3,211
nodes / 62 MB / 0.10s, and depths 24 and 26 — which previously did not
finish inside 6 GB — complete in 78 MB and 0.2s. The emitted lockfile is
byte-identical before and after, and byte-identical to the TypeScript
CLI's for the same input.
Closes https://github.com/pnpm/pnpm/issues/13574.
The peer-hoist discovery engine keeps one view of the workspace context
across every hoist round, refreshing it before each pass. That refresh
walked all three shared maps end to end — `children_by_id`, `packages`,
`dependencies_tree` — so every round cost one scan of the whole
workspace even when the round had added a handful of nodes. A resolve of
the 114-project Bit workspace from
https://github.com/pnpm/pnpm/issues/13505 runs 172 of those syncs in the
initial rounds alone, and each `extend_tree` bumps the revision that
triggers them.
Have the context record which keys of those maps each write touched, and
sync by visiting the keys written since the view's cursor. Only keys are
recorded, never values: the sync reads each key's current value, so a key
written several times, or written by concurrently-walking importers in
either order, converges on the same view. A view discarded after an
unmergeable change (a children-owner handover that re-split a package's
children and peers) refills itself by scanning the shared maps once,
rather than replaying the log, which would copy every key twice.
Neither existing scenario of the `workspace_full_resolution` benchmark
runs the auto-install-peers loop — the peer-heavy one provides every
importer's peers up front, so it converges in one round with a single
sync and measures identically here — so add a hoist-heavy scenario where
the peers are missing and the importers' direct sets differ. That
scenario runs 7.68/7.32/7.31s on main against 7.33/6.78/6.67s here.
On that workspace the initial rounds' sync time drops from 683ms to
215ms — the remainder is the three rebuilds, which replay the log — and
the hoist loop's from 30ms to 0.3ms, taking the two phases from 3.78s +
1.28s to 2.79s + 1.03s.
Related to https://github.com/pnpm/pnpm/issues/13505.
The optional- and required-peer hoists read the same run-resolved
candidate set but pick differently, so the transient-walk regression
scenario from pnpm/pnpm#13567 is now asserted through both pickers:
the required-peer variant pins that an unreachable transiently-walked
version must not win the dedupe-onto-highest-satisfying pick. Verified
to fail against the pre-pnpm/pnpm#13570 resolver with the exact
symptom (host suffixed with the unreachable pin). The shared fixture
is extracted into a resolve_with_transient_shared_walk driver.
Related to pnpm/pnpm#13567.
The peer-hoist pickers biased toward preferred_versions_from_run, an
arrival-ordered workspace-wide fold written by every walk. Concurrent
importer waves can transiently win a children-ownership claim, walk a
subtree, and lose the context to a deterministically better-placed
occurrence; whether that transient walk (and its version folds) happens
depends on thread interleaving, so the fold's membership — and the
hoisted peer picks, and every peer suffix downstream — varied run to
run, rewriting pnpm-lock.yaml on every install of identical inputs.
Derive the pickers' candidates from the settled reachable tree instead:
WorkspaceTreeCtx::run_preferred_versions walks children_by_id (which
only ever holds the deterministic children owner's records) from the
recorded importer-level direct deps and folds each reachable package's
identity. The closure is cached, grown incrementally per revision, and
rebuilt when a children-ownership rewrite restructures existing
subtrees. workspace:-surfaced project versions keep their fold via a
per-package identity record, also reachability-gated. This matches the
TS resolver, where allPreferredVersions ends up equal to the set
reachable through its arrival-ordered dedup because it is
single-threaded. The concurrent init waves stay concurrent.
Closespnpm/pnpm#13567
The peer-resolution passes sort `NodeId`s so their output does not depend
on hash iteration order. The comparison key was a freshly formatted
`String` per element — `"0:{value:020}"` for a counter, `"1:{id}"` for a
leaf — and `sort_by_key` re-invokes the key function on every comparison,
so each sort allocated `O(n log n)` strings. `build_final_dep_paths` and
the Tarjan pass under it run those sorts over every node that resolved a
peer, plus one sorted successor list per node.
Derive `Ord` on `NodeId` instead and sort the values directly. The
derived order is the same total order the rendered keys encoded: the
variant order puts counters before leaves, a fixed-width zero-padded
u64 compares like the number it renders, and leaf ids compare as their
`str` either way. The resolved graph is unchanged.
On the 114-project Bit workspace from
https://github.com/pnpm/pnpm/issues/13505, resolution drops from ~16.0s
to ~13.4s, with the final depPath pass falling from 2.50s to 0.76s. The
peer-heavy `workspace_full_resolution` benchmark goes from ~1.01s to
~0.85s.
record_walked_node called previously_resolved_children twice with identical
arguments — once to seed the graph children, once to seed the record edges.
Each call rescans the ancestor pkg-id chain and, on a match, re-realizes every
matching occurrence's children, so the second one was pure repeat work in the
peer-resolution hot path.
Build the record edges first and iterate them by reference for the graph
children. The clone count is unchanged — the per-entry clones in the loop
replace the ones the second call made while rebuilding the map — while one
chain scan and one realize_children pass go away, and the common case (the
node's package is not in its own ancestor chain, so the map is empty) now does
strictly less work.
Raised by CodeRabbit on pnpm/pnpm#13564. Its suggested fix cloned the map into
the first loop instead, which trades one cost for another; building the record
edges first removes the call without adding a clone.
resolve_dependency_tree.rs carried the public options/error types, the shared
workspace context, the per-importer tree context, the recursive walk, node
reuse and children-ownership bookkeeping, lockfile-snapshot reuse checks,
catalog specifier resolution, and manifest extraction in one 4,037-line file.
Keep resolve_dependency_tree.rs as the facade holding the options, error, and
notification types plus the two entry points, and move the implementation into
resolve_dependency_tree/: workspace_ctx.rs (WorkspaceTreeCtx, the shared
per-pkgIdWithPatchHash dedup maps, and the children-ownership claim/handover),
tree_ctx.rs (TreeCtx and the per-edge ResolveOptions derivations), walk.rs (the
fresh-resolve walk — seed, children walk, per-wanted resolve cache, level
folds), reuse.rs (ReuseSource, the update scope, direct-dep change bookkeeping,
subtree-reuse predicates, and the snapshot-driven walk), manifest.rs (what the
walk reads off a resolved package: pkg id, child specs, peers, leaf
classification, deprecation), and catalogs.rs (catalog: specifier resolution).
TreeCtx and WorkspaceTreeCtx stay one struct each; their inherent impls are
divided across the modules with pub(super) visibility, so every cross-module
item stays private to resolve_dependency_tree and the crate's public API is
unchanged. The pub(super) surface was derived mechanically: every marker was
stripped and the compiler re-added only the ones the crate cannot compile
without. The unit tests that drive one module's internals move next to it, with
the shared result builder lifted into a test_support module; the end-to-end
tree-walk tests stay at the facade.
Pure code movement: no behavior, allocation, cloning, dispatch, or traversal
change. After normalizing away imports, module declarations, visibility
markers, and reference-style doc links, the multiset of source lines before and
after differs only by the eleven signatures rustfmt re-wrapped because
pub(super) pushed them past the width.
Closespnpm/pnpm#13554.
`remap_link_node_id` compared `lockfile_dir` against the link's
`directory` verbatim, but a workspace link records that directory
relative to the importer that declares it, so the lexical containment
check never matched and every internal link was remapped to
`link:<modules_dir>/<alias>` — feeding peer resolution a node id that
should only stand in for links living outside the workspace. The
lockfile then recorded the peer as `link:node_modules/<alias>`, a path
relative to the lockfile root rather than to the importer that holds
the symlink.
Resolve the target against `project_dir` before the containment check,
and thread each importer's root into `ResolvePeersOptions::project_dir`
in the multi-importer walk (it already threads `modules_dir`).
The same check also caught injected (`file:`) workspace dependencies,
whose directory is relative to the lockfile dir. Upstream only remaps
linked dependencies (`isLocal`, i.e. a non-`file:` directory
resolution), so gate the remap on a `link:` id.
The TypeScript CLI is unaffected — its directory resolutions for links
are absolute — so it only gains the mirror test.
Closespnpm/pnpm#13556
Walker::resolve_node was 549 lines — by a wide margin the largest function in
the crate (next is resolve_node_seed at 321, the median is 13). Nesting was
never the problem: real block depth peaks at 5, in the two-pass child loop that
is legitimately nested. Length was, because eleven sequential phases shared one
large local context.
Lift the three phases that have clean seams into named helpers:
- The graph-entry / NodeRecord capture moves to finalize.rs as
Walker::record_walked_node, next to the post-walk passes that consume those
records, taking a borrowed WalkedNode context.
- Building the ParentRefs map descendants see — locked-peer scoping, provider
children, locked pins — becomes Walker::build_child_parent_refs, returning
the shared map, the node's own contribution, and whether anything changed.
- The node's own peer resolution, its fold with the children's, and the depPath
it renders to become Walker::resolve_node_peers.
The early-return short-circuits (context-free depPath, cycle re-entry, fast
cache hit, peersCache hit) stay inline: as Option<NodeOutput>-returning helpers
they would make the control flow harder to follow, not easier. resolve_node is
now 378 lines of linear short-circuits plus the child walk.
No allocation, cloning, or traversal change. The extracted phases take borrowed
context structs, following the CacheHitContext / DeferredChildContext
precedent, so no borrow conflict is resolved with a clone. The one behavioral
identity worth naming: resolve_node_peers folds the node's own resolved peers
into auto_install_resolved_peers after its loop instead of during it — the two
maps agree because resolve_one_peer only ever inserts under the peer name it
was called for.
Also fold in the adjacent micro-cleanup the issue called out:
Walker::deferred_child_resolution did two pure_pkgs hash lookups where one
if-let chain does.
Closespnpm/pnpm#13553.
The two final-graph tests that exercise peer-edge suffixes left
`node_dep_paths` empty, so `build_final_graph` skipped its `min_depth`
pass entirely and every node's depth came from the fallback
`record.depth`. Each test now registers a shallower `find_hit` revisit —
walked, so it has a `node_dep_paths` entry, but short-circuited before a
`NodeRecord` was created — and asserts the merged node takes that
shallower depth. Both new assertions fail when the `min_depth` lookup is
replaced by `record.depth`.
The bare `assert!` that no trimmed provider variant is fabricated now
prints the final graph keys, as does its sibling in the neighbouring
test, so a failure shows which variants were built.
Closes https://github.com/pnpm/pnpm/issues/13555
The peer-heavy resolver fixture has a wide run-to-run spread — a local 15-run
pass measured a ~15% coefficient of variation on an idle machine, with the slow
samples scattered through the sequence rather than clustered at the start, so
the noise is runner contention rather than a cold cache. Three CI runs against
that spread produced a 2.83s outlier on pnpm/pnpm#13551 that lifted the pacquet
mean to 2.15s while its fastest run sat at 1.54s, inside main's range.
Compare the cross-engine gate on each target's fastest run instead of its mean.
Contention only ever adds time, so the minimum is the sample least perturbed by
a noisy neighbour; this is already the statistic the workflow reports to
Bencher, for the same reason. Raise the scenario to nine runs so that minimum
has a good chance of landing on an uncontended run, and surface both mean and
min in the diagnostics table.
The gate is a correctness check, not a measurement, and it was costing us the
measurement: asserting mid-job aborted before the Bencher upload and dropped
every scenario's data point for the whole run. That is why this benchmark has
almost no history on bencher.dev — the two main pushes after it was added both
failed on the since-recalibrated 10x floor, so only one point was ever
published. Record the verdict, let the upload proceed, and re-raise the failure
afterwards.
Pacquet-only benchmark-harness and CI change; no user-visible surface, no
TypeScript counterpart, and no changeset.
resolve_peers.rs carried the public entry points, workspace orchestration,
peer-hoist discovery, the walker, cache matching, child realization, final
dep-path construction, cycle analysis, graph construction, and the standalone
peer-context utilities in one 3,934-line file, with a 2,434-line test module
beside it.
Keep resolve_peers.rs as the facade holding the options/result types and the
two public entry points, and move the implementation into resolve_peers/:
discovery.rs (the hoist-loop discovery engine and its persistent caches),
walker.rs (Walker, its state, walk, resolve_node, and peer resolution),
cache.rs (purePkgs and peersCache matching, deferred child resolution, child
realization, retention decisions), finalize.rs (pending-edge repair, SCC cycle
analysis, final depPath recomputation, graph construction), and context.rs
(parent references, shared ancestor chains, link remapping, peer-suffix
parsing, range matching).
Walker stays one struct; its inherent impl is divided across the modules with
`pub(super)` visibility, so every cross-module item stays private to
resolve_peers and the crate's public API is unchanged. The unit tests that
drive one module's internals move next to it, with the shared tree builders
lifted into a test_support module; the end-to-end peer-resolution tests stay
at the facade.
Pure code movement: no behavior, allocation, cloning, dispatch, or traversal
change, and release builds use `lto = "fat"` with `codegen-units = 1`, so the
module boundaries cannot shift inlining either.
Closespnpm/pnpm#13548.
The peer-heavy scenario asserted that pacquet resolves the fixture at least
10x faster than the TypeScript CLI. That headroom was measured against a base
that still re-filtered the packument and re-parsed semver ranges once per
dependency edge on the offline hot-cache path. pnpm/pnpm#13504 removed that
per-edge work, so the same 401-package DAG now resolves in ~2.9s instead of
~34.4s and the ratio fell to ~1.7x, failing the benchmark on main.
Tie the floor to what the assertion is actually for: a peer-resolution blowup
of the kind pnpm/pnpm#13540 tracked runs slower than the TypeScript CLI (7.0s
versus 2.9s, ~0.4x), so 1.25x clears the run-to-run noise while still catching
it, and no longer breaks whenever the TypeScript resolver gets faster.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
web_auth_fake! emitted its whole surface — FakeHost with its eight capability
impls, both reporters, every set_* setter, and the message queries — into every
test and silenced the unused ones with allow(dead_code), noise that can also
mask a genuinely dead helper.
Keep only the thread-local statics and reset() as the always-present core and
generate each optional helper from a named macro argument, with a catch-all
compile_error! arm for a mistyped name. Each test names exactly the helpers it
drives, so nothing is unused and no dead_code allow is needed. Call sites in
network-web-auth, auth-commands, cli, and publish name their helpers, and
CODE_STYLE_GUIDE.md documents the convention with login_fake! as the reference.
Pacquet-only test-infrastructure and guide change; no user-visible surface, no
TypeScript counterpart, and no changeset.
Stream lazy child resolution through caches and share ancestry, missing-peer summaries, package
data, and dependency paths across repeated peer contexts. Retain only discovery nodes needed as
peer providers instead of occurrence state for every cache-served visit.
Index peer-provider children once per evolving workspace graph so cached peer lookup and lazy
provider previews avoid repeatedly scanning every dependency edge.
Recursively hoist nested conflict roots so the hoisted linker does not expand a small dependency
graph into deeply repeated node_modules paths.
Add a deterministic integrated benchmark that compares the peer-heavy resolver against main and
TypeScript pnpm while enforcing byte-identical lockfiles.
Closespnpm/pnpm#13540.
Resolution re-derived the same data once per dependency edge instead of
once per packument:
- filterPkgMetadataByPublishDate ran on every pick (every edge) with
minimumReleaseAge active — the default since it is 1440 min — parsing
a Date per version and rebuilding the versions map each time. It is
now memoized per packument object in a WeakMap, keyed by cutoff and
trusted versions so one packument served under different policies
still filters per policy. Sharing the returned object is safe: callers
already shared the inner version manifests, and the packument objects
the resolver passes in are themselves cached and treated as read-only.
- pickPackageFromMeta re-parsed version strings and ranges on every
satisfies/maxSatisfying call. Parsed SemVer and Range instances are
now cached in capped Maps (cleared at 50k entries so long-lived
processes cannot grow them without bound), and maxSatisfying/
minSatisfying are replaced with loops that reuse the caches. The
dist-tag repopulation loop in the publish-date filter likewise
compares already-parsed instances instead of calling semver.gt on
strings.
- Concurrent offline picks of one package all miss the pre-queue
in-memory cache check, then each re-read and re-parsed the mirror
behind the per-package limiter (176 of 1311 loads on the benchmark
fixture were duplicates). The limiter callback now re-checks the
cache; offline entries are always disk-sourced, so this is equivalent
to arriving after the first caller cached it.
- clearMeta condensed every version object through ramda's curried
pick; a direct field loop does the same with no per-field dispatch.
The presence check via undefined-comparison matches the previous
behavior because version objects come from JSON, which cannot encode
undefined.
Resolution of the 1246-package benchmark fixture is 12.5% faster on
top of current main (4.27s -> 3.73s, hyperfine, offline hot cache);
the resulting lockfile is byte-identical. Attribution: filter
memoization -11%, semver caches -4%, the rest ~-1%.
Every projected numeric network value carries the engine default when
unset, and Bit merges the projection over its own global network
config - without knowing which values are explicit it either forwards
defaults as overrides or forwards nothing.
A registry version that carried no `dist.integrity` was recorded in the
lockfile with a bare tarball URL and no hash, and `--lockfile-only` never
computed one — so the very first `pnpm install --frozen-lockfile` over
that lockfile failed with `ERR_PNPM_MISSING_TARBALL_INTEGRITY`. Two
independent gaps produced it, and both are closed here.
First, `dist.shasum` was ignored. pnpm's `getIntegrity` promotes the
legacy hex digest to a `sha1-` SRI string when `dist.integrity` is
absent, which is what pins every version published before subresource
integrity — the whole of `https://node-registry.bit.cloud/`, for one.
pacquet parsed the field and dropped it, leaving those versions unpinned
and diverging from the lockfile the TypeScript CLI writes for the same
graph. `dist_integrity` now mirrors `getIntegrity`, including the
`ERR_PNPM_INVALID_TARBALL_INTEGRITY` failure for a shasum that is not a
hex digest. It also restores the stronger of the two behaviours: a
registry-published hash pins the bytes, where hashing whatever arrived
pins nothing.
Second, `--lockfile-only` skipped the `PrefetchingResolver` wholesale,
and that wrapper owns `populate_missing_integrity` — the only path that
hashes a tarball no registry hash covers. Background prefetching and
integrity completion are now separate concerns: the wrapper always sits
in the resolver chain, and `prefetch_downloads` turns off only the
speculative download. A run that materializes nothing (`--lockfile-only`)
or only part of the graph (a filtered workspace selection) still fetches
the few tarballs whose hash the lockfile cannot be written without, which
is what pnpm does with `resolutionNeedsFetch` under `dryRun`.
Closespnpm/pnpm#13547
The lockfile pinned `@types/node@26.1.2` as the hoisted optional peer of
every `@inquirer/prompts` consumer, next to the `22.20.1` the workspace
catalog pins with `^22.19.19`. That is the bug 71a4934d3f fixed:
`getHoistableOptionalPeers` used to maximize over every version in the
graph instead of bounding its candidates by the workspace root's own
specifier, so an optional peer declared as `*` pulled in the newest
version any importer had resolved.
The entries came in with 7f81da30b8, six days after that fix landed,
because the install that wrote them ran pnpm 12.0.0-alpha.21 — published
before the fix — rather than the pinned 12.0.0-beta.2. Replaying that
commit's inputs with alpha.21 reproduces its lockfile byte for byte,
while beta.1, beta.2, and the TypeScript CLI at 11.18.0 all produce
`22.20.1`.
So every full re-resolution rewrites those 80 lines, and unrelated pull
requests pick the correction up as soon as one of their manifest changes
forces one — in pnpm/pnpm#13535 it accounts for most of the lockfile
diff. Re-resolving here keeps it out of the next PR that touches a
manifest. No product code changes: the resolver fix is already in.
Both utilities lived in the pnpm/components Bit workspace, where their
component names collide with same-named components elsewhere in the Bit
registry. Move the code here and publish it under new names:
`@pnpm/util.lex-comparator` -> `@pnpm/text.ordinal-comparator`
`@pnpm/config.nerf-dart` -> `@pnpm/config.registry-auth-key`
The new names describe what the utilities do: the comparator is ordinal
rather than locale-aware, and the mapped URL is the key that registry
settings are stored under in `.npmrc`. The exported functions keep their
names, so consumers only change their import specifiers.
Implementations and tests are carried over unchanged. The rationale from
the components' docs pages moves into doc comments: why `localeCompare`
cannot be used for values compared across machines, and where `nerfDart`
originates.
No pacquet counterpart is needed. The Rust stack has its own
implementations of both and no user-visible behavior changes.
The napi install result populated depsRequiringBuild from the ignored-
builds accumulation only, so an embedder that allows dependency builds
to run (Bit) always received an empty list and wiped its recorded
bit.depsRequiringBuild lockfile block on every install.
Mirror the TypeScript engine's returnListOfDepsRequiringBuild option:
when set, the fresh-resolve path reports every non-skipped package
whose files carry install scripts, regardless of the allow-build
policy, through a sink threaded into the install pipeline. Installs
served from the frozen-lockfile path leave the field undefined so the
embedder keeps its previously recorded list, matching the TS engine,
which only computes the list from a fresh resolve's fetch results.
Every import was type-only, so the declared dependency shipped the
whole config-reader tree to reporter consumers for nothing — Bit's
mirror-constrained installs were pulling config.env-replace@4.x through
it. The reporter now declares ReporterPnpmConfig, the structural slice
of the config it actually reads; the pnpm CLI's full Config satisfies
it unchanged.
Picks up pnpm/update#4: the pin bump now happens first, so the
dependency update and the refreshed lockfile are produced by the pnpm
version the update PR moves the repository onto.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Zoltan Kochan