mirror of
https://github.com/pnpm/pnpm.git
synced 2026-06-28 09:55:39 -04:00
4088de0433
* ci(bencher): record benchmark results to Bencher Tracks both stacks in one Bencher project (`pnpm`), under separate testbeds (`pnpm`, `pacquet`). - `benchmarks/bench.sh` emits a hyperfine-shaped `bencher-results.json` combining the six pnpm scenarios. - `benchmark.yml` adds `push: branches: [main]` so each merge updates the `pnpm` baseline, then uploads via the Bencher CLI. - `pacquet-integrated-benchmark.yml` adds the same main-push baseline for `pacquet`, combines the four scenario JSONs into a Bencher report, and stages it into the existing artifact. - `pacquet-integrated-benchmark-comment.yml` uploads PR results from the trusted `workflow_run` context so fork PRs are covered too. Requires `BENCHER_API_TOKEN` in repo secrets; workflows no-op with a `::notice::` if it's missing. * ci(bencher): allow workflow_dispatch to upload pacquet results The inline Bencher upload was gated to `event_name == 'push'`, which meant manual dispatch from a feature branch ran the bench but skipped the upload. Both push and workflow_dispatch execute in the base-repo privilege context, so it's safe to upload from both — the fork-safety gate only needs to keep `pull_request` runs out. On dispatch from a non-main branch we record into the ref name with `--start-point main --start-point-reset`, matching the pnpm bench's branch policy. * ci(bencher): treat workflow_dispatch on main as the main baseline Two small fixes from PR review: - `benchmark.yml`: manual dispatch from `main` was falling into the non-main branch arm, recording `--branch main --start-point main --start-point-reset` — equivalent to forking main from itself. Treat it like a `push` event so a manual run on main updates the baseline directly. Matches the pacquet workflow's branch policy. - `bench.sh`: emit a stderr warning when `jq` is missing instead of silently skipping `bencher-results.json`. Keeps behaviour optional for local users but makes the skip discoverable. * bench: rename scenarios with explicit state axes Replaces the hyperfine-leaking names (`clean-install`, `frozen-lockfile`, `peek`, `gvs-warm`, …) with a consistent grid that spells out every state the benchmark depends on: - "Fresh" — node_modules wiped at start (future variants will start with a populated node_modules). - "Install" vs "Restore" vs "Add new dep" — the work being measured. - "hot/cold cache + hot/cold store" — both pnpm directories, spelled out separately because they're distinct on disk. - "isolated linker" — nodeLinker mode (future variants will cover `hoisted` and `pnp`). The slugs map directly from the clap-derived kebab-case names, so `--scenario=fresh-restore-cold-cache-cold-store-isolated` is the new CLI surface. Updates land across the Rust orchestrator (`BenchmarkScenario`), `benchmarks/bench.sh`, the pacquet workflow, `benchmarks/README.md`, and `pacquet/CONTRIBUTING.md` so the names agree end-to-end. Adds a justified `#[allow(clippy::enum_variant_names)]` on the enum because every variant currently shares the `Fresh` prefix; the lint will stop firing once `Filled*`/`Resynced*` counterparts land. Bencher's stored history for the old benchmark names will become orphaned and can be archived in the UI. * bench: linker-first slug shape with dot-separated axes Reshapes the scenario identifiers so the linker mode is the leading group: `<linker>.<action>.<cache state>.<store state>`. Dots separate the four axes the bench varies, and `isolated-linker.*` / `gvs-linker.*` sort together in any dashboard that groups by prefix. Future buckets (`hoisted-linker.*`, `pnp-linker.*`) will slot in without disturbing the existing names. GVS is its own top-level bucket rather than a sub-variant of isolated — its perf profile differs enough to chart separately. Renames: - `clean-install` → `isolated-linker.fresh-install.cold-cache.cold-store` - `full-resolution` → `isolated-linker.fresh-install.hot-cache.hot-store` - `frozen-lockfile` → `isolated-linker.fresh-restore.cold-cache.cold-store` - `frozen-lockfile-hot-cache` → `isolated-linker.fresh-restore.hot-cache.hot-store` - `peek` → `isolated-linker.fresh-add-dep.hot-cache.hot-store` - `gvs-warm` → `gvs-linker.fresh-restore.hot-cache.hot-store` Each Rust variant now carries `#[value(name = "…")]` so clap accepts the dotted CLI form (`--scenario=isolated-linker.fresh-install.cold-cache.cold-store`). Display labels follow the slug structure: `Isolated linker: fresh install, cold cache + cold store` and `GVS linker: fresh restore, hot cache + hot store`. The `#[allow(clippy::enum_variant_names)]` is renewed; 5 of 6 variants share the `Isolated` prefix today. Once `Hoisted*` / `Pnp*` buckets land the lint will stop firing on its own. * style: apply rustfmt after scenario rename The longer match-arm pattern produced by the linker-first rename exceeded the rustfmt width budget. Auto-format breaks the `&["install", "--frozen-lockfile"]` body onto its own line so the arm stays within the limit.
78 lines
3.6 KiB
Markdown
78 lines
3.6 KiB
Markdown
# pnpm Benchmarks
|
|
|
|
Compares `pnpm install` performance between the current branch (`HEAD`) and
|
|
`main`, across the six scenarios listed below.
|
|
|
|
This wrapper builds both pnpm revisions and runs hyperfine through the
|
|
shared Rust orchestrator at
|
|
[`pacquet/tasks/integrated-benchmark`](../pacquet/tasks/integrated-benchmark/),
|
|
so scenario / fixture / workspace / install-script / report generation
|
|
stay consistent with the pacquet benchmark.
|
|
|
|
## Prerequisites
|
|
|
|
- `cargo` (install Rust via [rustup](https://rustup.rs) if you don't have it).
|
|
- `hyperfine`, `pnpm`, `node`, `git` on `$PATH`.
|
|
|
|
## Usage
|
|
|
|
```sh
|
|
./benchmarks/bench.sh
|
|
```
|
|
|
|
The script:
|
|
|
|
1. Builds the `integrated-benchmark` binary in release mode.
|
|
2. Clones the current repo into the temp work-env once per revision
|
|
(`HEAD` and `main`) and runs `pnpm install && pnpm run compile-only`
|
|
in each to produce `pnpm/dist/pnpm.mjs`. `compile-only` skips the
|
|
`update-manifests` pass that the root `compile` script does — it
|
|
would rewrite tracked files and trigger a second install per
|
|
revision, neither of which the bench needs.
|
|
3. Runs hyperfine on each scenario with `--registry=npm` (hits
|
|
`registry.npmjs.org` directly, no proxy — same as before).
|
|
4. Writes a per-scenario `BENCHMARK_REPORT.md` / `.json` and a
|
|
consolidated `results.md` into the temp work-env. The path is printed
|
|
at the end of the run.
|
|
5. Emits `bencher-results.json` — a hyperfine-shaped file with one
|
|
result per scenario (the `@HEAD` revision only, `command` renamed to
|
|
the scenario name) that the `Benchmarks` GitHub Actions workflow
|
|
uploads to [Bencher](https://bencher.dev) for continuous tracking.
|
|
|
|
## Scenarios
|
|
|
|
Slugs follow `<linker>.<action>.<cache state>.<store state>` so the
|
|
leading segment groups runs by linker mode. Today there are two
|
|
groups (`isolated-linker.*` and `gvs-linker.*`); future scenarios
|
|
will add `hoisted-linker.*` and `pnp-linker.*`.
|
|
|
|
Every current scenario starts with `node_modules` wiped — "fresh"
|
|
names that target state; future variants that begin with a populated
|
|
`node_modules` will use a different action prefix.
|
|
|
|
| # | Slug | Lockfile | Cache | Store | Description |
|
|
|---|---|---|---|---|---|
|
|
| 1 | `isolated-linker.fresh-restore.hot-cache.hot-store` | ✔ frozen | hot | hot | Restore from lockfile with both directories hot (repeat-headless shape) |
|
|
| 2 | `isolated-linker.fresh-add-dep.hot-cache.hot-store` | ✔ + add dep | hot | hot | `pnpm add <dep>` against an existing lockfile |
|
|
| 3 | `isolated-linker.fresh-install.hot-cache.hot-store` | ✗ | hot | hot | Resolve from scratch with both directories hot |
|
|
| 4 | `isolated-linker.fresh-restore.cold-cache.cold-store` | ✔ frozen | cold | cold | Restore from lockfile with cold disks (typical CI shape) |
|
|
| 5 | `isolated-linker.fresh-install.cold-cache.cold-store` | ✗ | cold | cold | True cold start — no lockfile, nothing cached |
|
|
| 6 | `gvs-linker.fresh-restore.hot-cache.hot-store` | ✔ frozen | hot | hot + GVS | Frozen-lockfile restore with `enableGlobalVirtualStore: true`, pre-warmed GVS |
|
|
|
|
All scenarios use `--ignore-scripts` and isolated store/cache directories per revision.
|
|
|
|
## Fixture
|
|
|
|
The fixture lives at [`fixture/`](./fixture/) — a synthetic
|
|
`package.json` with ~80 typical front-end dependencies, plus a committed
|
|
`pnpm-lock.yaml` (generated once with `pnpm install --lockfile-only`).
|
|
The lockfile is checked in so every CI run starts from the same
|
|
resolution graph regardless of registry drift.
|
|
|
|
## Configuration
|
|
|
|
Environment variables read by `bench.sh`:
|
|
|
|
- `WARMUP` — number of warmup runs before timing (default: 1)
|
|
- `RUNS` — number of timed runs per benchmark (default: 10)
|