* chore(bench): add Cargo repeat-install scenario to integrated-benchmark
Add the cargo.repeat-install.hot-cache.hot-store scenario to
pnpm-integrated-benchmark and the CI integrated-benchmark workflow
to guard Cargo crate slot reuse (pnpm/pnpm#17105).
* fix(bench): isolate cargo repeat-install benchmark from parent workspace
* fix(bench): connect generated crates in Cargo dependency graph and verify prewarm slots
None of the integrated benchmark's scenarios resolved against a lockfile
the walk could mostly reuse, installed store-only (`enableModulesDir:
false`), or read the per-file store index. The install regressions fixed
in pnpm/pnpm#16886, pnpm/pnpm#16888 and pnpm/pnpm#16890 all sat in that shape; the
benchmark would have missed each of them, and would miss their reversal.
The new `store-only.unlocked-direct-deps.cold-cache.hot-store` scenario
seeds the static fixture's lockfile with the direct dependencies whose
locked version carries a peer suffix left out of the importer, while
`packages` and `snapshots` keep the whole graph. That is `pnpm install`
after dependencies were added to `package.json` by hand: the missing
direct dependencies resolve from the registry, land on versions the
graph holds, and the walk reuses their subtrees. The choice of the
peered ones is deliberate: a dependency without peers is taken from the
lockfile's package list without a registry request, and a dependency
whose specifier merely changed is reused as long as the recorded version
satisfies the new range, so neither exercises the fresh pick that the
subtree reuse then follows. The metadata cache is wiped per iteration,
so every package the walk does resolve costs a round trip over the
emulated registry link; with a hot cache on local disk the whole
speculative warm-up costs about 25 ms and sits inside the noise. The
lockfile is trusted (`trustLockfile: true`): its supply-chain
verification reads every entry's packument and left nothing for the
resolution to fetch, hiding the difference entirely. With
`enableModulesDir: false` nothing is linked, so the rest of the timed
work is the check that the store holds every package, read through
`storeIndexFormat: files`.
`--no-frozen-lockfile` is passed explicitly because the runners set
`CI`, where a plain `install` is frozen and would reject the lockfile.
Only `cache-dir` is removed and the lockfile restored between
iterations; `node_modules` is never created and the warm store stays.
The scenario runs direct targets only, like the workspace fixtures: a
pnpr server would take over the resolution being measured. The
comparison script and its matrix-coverage test, the Bencher upload and
the summary list it alongside the others. Every `cleanup` arm now goes
through a `Cleanup` constructor, which keeps the method within the
function-length limit with the new arm.
Measured locally (hyperfine, 5 runs, main vs the commit before pnpm/pnpm#16886):
main 1.008 s +- 0.025 s (0.991 .. 1.052 s), 513 packuments
before pnpm/pnpm#16886 1.148 s +- 0.026 s (1.131 .. 1.194 s), 840 packuments
Avoid replacing an injected directory when its regular files are still
hardlinked to the current source and its entries match. Keep the existing
refresh path for mismatches, links, forced imports and build isolation.
Preserve import reporting and add a frozen-repeat scenario to the existing
integrated benchmark and CI comparison.
Closespnpm/tasks#121.
Confirm suspected integrated benchmark regressions once with target order
reversed. Same-run measurements can still be biased by sustained runner drift:
a control run on identical Rust sources produced a 29% apparent pnpr slowdown.
Keep the 5% tolerance with a 2 ms floor and the retained-sample noise guard.
Require the regression to reproduce in the confirmation, preserve both sets
of measurements in the artifact and report, and keep the initial timings for
historical Bencher uploads. Missing required confirmations fail the check.
The wrapper preserves the existing harness arguments and assertions. It does
not retry command failures, and confirmation failures propagate. Main-only
runs and scenarios without suspected regressions execute once.
Related to pnpm/pnpm#15228.
Compare integrated benchmark samples against main measured on the same runner.
Keep absolute minimum timings in Bencher for historical tracking without letting
cross-machine historical alerts overwrite the PR regression check.
Require at least nine successful samples per target. Fail when the median
slowdown and retained sample separation both exceed 5% of main's median or
2 ms, whichever is larger. Trim up to 10% from each tail. Report other median
slowdowns above that tolerance as inconclusive. This conservative noise guard
is not a statistical confidence interval; smaller or noisy regressions can pass,
and sequential measurements remain vulnerable to runner drift.
Run the comparison from trusted default-branch code in the privileged comment
workflow, publish the report before enforcing the result, and preserve the
existing check name on the triggering PR commit. Keep the peer-heavy
Rust-versus-TypeScript assertion unchanged.
Cover regressions, shared machine slowdowns, outliers, invalid data, all scenario
and engine combinations, command names, CLI exit codes, and the duration-dependent tolerance with Node tests.
Keep the explanation in the existing benchmark documentation.
Zoltan Kochan