mirror of https://github.com/pnpm/pnpm.git synced 2026-05-24 16:46:06 -04:00

Files

Zoltan Kochan 4088de0433 ci(bencher): record benchmark results to Bencher (#11875 )

* ci(bencher): record benchmark results to Bencher

Tracks both stacks in one Bencher project (`pnpm`), under separate
testbeds (`pnpm`, `pacquet`).

- `benchmarks/bench.sh` emits a hyperfine-shaped `bencher-results.json`
  combining the six pnpm scenarios.
- `benchmark.yml` adds `push: branches: [main]` so each merge updates
  the `pnpm` baseline, then uploads via the Bencher CLI.
- `pacquet-integrated-benchmark.yml` adds the same main-push baseline
  for `pacquet`, combines the four scenario JSONs into a Bencher
  report, and stages it into the existing artifact.
- `pacquet-integrated-benchmark-comment.yml` uploads PR results from
  the trusted `workflow_run` context so fork PRs are covered too.

Requires `BENCHER_API_TOKEN` in repo secrets; workflows no-op with a
`::notice::` if it's missing.

* ci(bencher): allow workflow_dispatch to upload pacquet results

The inline Bencher upload was gated to `event_name == 'push'`, which
meant manual dispatch from a feature branch ran the bench but skipped
the upload. Both push and workflow_dispatch execute in the base-repo
privilege context, so it's safe to upload from both — the fork-safety
gate only needs to keep `pull_request` runs out.

On dispatch from a non-main branch we record into the ref name with
`--start-point main --start-point-reset`, matching the pnpm bench's
branch policy.

* ci(bencher): treat workflow_dispatch on main as the main baseline

Two small fixes from PR review:

- `benchmark.yml`: manual dispatch from `main` was falling into the
  non-main branch arm, recording `--branch main --start-point main
  --start-point-reset` — equivalent to forking main from itself.
  Treat it like a `push` event so a manual run on main updates the
  baseline directly. Matches the pacquet workflow's branch policy.
- `bench.sh`: emit a stderr warning when `jq` is missing instead of
  silently skipping `bencher-results.json`. Keeps behaviour optional
  for local users but makes the skip discoverable.

* bench: rename scenarios with explicit state axes

Replaces the hyperfine-leaking names (`clean-install`, `frozen-lockfile`,
`peek`, `gvs-warm`, …) with a consistent grid that spells out every
state the benchmark depends on:

- "Fresh" — node_modules wiped at start (future variants will start
  with a populated node_modules).
- "Install" vs "Restore" vs "Add new dep" — the work being measured.
- "hot/cold cache + hot/cold store" — both pnpm directories,
  spelled out separately because they're distinct on disk.
- "isolated linker" — nodeLinker mode (future variants will cover
  `hoisted` and `pnp`).

The slugs map directly from the clap-derived kebab-case names, so
`--scenario=fresh-restore-cold-cache-cold-store-isolated` is the new
CLI surface. Updates land across the Rust orchestrator
(`BenchmarkScenario`), `benchmarks/bench.sh`, the pacquet workflow,
`benchmarks/README.md`, and `pacquet/CONTRIBUTING.md` so the names
agree end-to-end.

Adds a justified `#[allow(clippy::enum_variant_names)]` on the enum
because every variant currently shares the `Fresh` prefix; the lint
will stop firing once `Filled*`/`Resynced*` counterparts land.

Bencher's stored history for the old benchmark names will become
orphaned and can be archived in the UI.

* bench: linker-first slug shape with dot-separated axes

Reshapes the scenario identifiers so the linker mode is the leading
group: `<linker>.<action>.<cache state>.<store state>`. Dots separate
the four axes the bench varies, and `isolated-linker.*` /
`gvs-linker.*` sort together in any dashboard that groups by prefix.
Future buckets (`hoisted-linker.*`, `pnp-linker.*`) will slot in
without disturbing the existing names.

GVS is its own top-level bucket rather than a sub-variant of
isolated — its perf profile differs enough to chart separately.

Renames:

- `clean-install` → `isolated-linker.fresh-install.cold-cache.cold-store`
- `full-resolution` → `isolated-linker.fresh-install.hot-cache.hot-store`
- `frozen-lockfile` → `isolated-linker.fresh-restore.cold-cache.cold-store`
- `frozen-lockfile-hot-cache` → `isolated-linker.fresh-restore.hot-cache.hot-store`
- `peek` → `isolated-linker.fresh-add-dep.hot-cache.hot-store`
- `gvs-warm` → `gvs-linker.fresh-restore.hot-cache.hot-store`

Each Rust variant now carries `#[value(name = "…")]` so clap accepts
the dotted CLI form (`--scenario=isolated-linker.fresh-install.cold-cache.cold-store`).

Display labels follow the slug structure: `Isolated linker: fresh
install, cold cache + cold store` and `GVS linker: fresh restore,
hot cache + hot store`.

The `#[allow(clippy::enum_variant_names)]` is renewed; 5 of 6 variants
share the `Isolated` prefix today. Once `Hoisted*` / `Pnp*` buckets
land the lint will stop firing on its own.

* style: apply rustfmt after scenario rename

The longer match-arm pattern produced by the linker-first rename
exceeded the rustfmt width budget. Auto-format breaks the
`&["install", "--frozen-lockfile"]` body onto its own line so the
arm stays within the limit.

2026-05-23 15:19:49 +02:00

fixture

ci(benchmark): consolidate (#11741 )

2026-05-21 21:12:29 +02:00

bench.sh

ci(bencher): record benchmark results to Bencher (#11875 )

2026-05-23 15:19:49 +02:00

README.md

ci(bencher): record benchmark results to Bencher (#11875 )

2026-05-23 15:19:49 +02:00

README.md

pnpm Benchmarks

Compares pnpm install performance between the current branch (HEAD) and main, across the six scenarios listed below.

This wrapper builds both pnpm revisions and runs hyperfine through the shared Rust orchestrator at pacquet/tasks/integrated-benchmark, so scenario / fixture / workspace / install-script / report generation stay consistent with the pacquet benchmark.

Prerequisites

cargo (install Rust via rustup if you don't have it).
hyperfine, pnpm, node, git on $PATH.

Usage

./benchmarks/bench.sh

The script:

Builds the integrated-benchmark binary in release mode.
Clones the current repo into the temp work-env once per revision (HEAD and main) and runs pnpm install && pnpm run compile-only in each to produce pnpm/dist/pnpm.mjs. compile-only skips the update-manifests pass that the root compile script does — it would rewrite tracked files and trigger a second install per revision, neither of which the bench needs.
Runs hyperfine on each scenario with --registry=npm (hits registry.npmjs.org directly, no proxy — same as before).
Writes a per-scenario BENCHMARK_REPORT.md / .json and a consolidated results.md into the temp work-env. The path is printed at the end of the run.
Emits bencher-results.json — a hyperfine-shaped file with one result per scenario (the @HEAD revision only, command renamed to the scenario name) that the Benchmarks GitHub Actions workflow uploads to Bencher for continuous tracking.

Scenarios

Slugs follow <linker>.<action>.<cache state>.<store state> so the leading segment groups runs by linker mode. Today there are two groups (isolated-linker.* and gvs-linker.*); future scenarios will add hoisted-linker.* and pnp-linker.*.

Every current scenario starts with node_modules wiped — "fresh" names that target state; future variants that begin with a populated node_modules will use a different action prefix.

#	Slug	Lockfile	Cache	Store	Description
1	`isolated-linker.fresh-restore.hot-cache.hot-store`	✔ frozen	hot	hot	Restore from lockfile with both directories hot (repeat-headless shape)
2	`isolated-linker.fresh-add-dep.hot-cache.hot-store`	✔ + add dep	hot	hot	`pnpm add <dep>` against an existing lockfile
3	`isolated-linker.fresh-install.hot-cache.hot-store`	✗	hot	hot	Resolve from scratch with both directories hot
4	`isolated-linker.fresh-restore.cold-cache.cold-store`	✔ frozen	cold	cold	Restore from lockfile with cold disks (typical CI shape)
5	`isolated-linker.fresh-install.cold-cache.cold-store`	✗	cold	cold	True cold start — no lockfile, nothing cached
6	`gvs-linker.fresh-restore.hot-cache.hot-store`	✔ frozen	hot	hot + GVS	Frozen-lockfile restore with `enableGlobalVirtualStore: true`, pre-warmed GVS

All scenarios use --ignore-scripts and isolated store/cache directories per revision.

Fixture

The fixture lives at fixture/ — a synthetic package.json with ~80 typical front-end dependencies, plus a committed pnpm-lock.yaml (generated once with pnpm install --lockfile-only). The lockfile is checked in so every CI run starts from the same resolution graph regardless of registry drift.

Configuration

Environment variables read by bench.sh:

WARMUP — number of warmup runs before timing (default: 1)
RUNS — number of timed runs per benchmark (default: 10)