Commit Graph

3 Commits

Author SHA1 Message Date
Zoltan Kochan
4088de0433 ci(bencher): record benchmark results to Bencher (#11875)
* ci(bencher): record benchmark results to Bencher

Tracks both stacks in one Bencher project (`pnpm`), under separate
testbeds (`pnpm`, `pacquet`).

- `benchmarks/bench.sh` emits a hyperfine-shaped `bencher-results.json`
  combining the six pnpm scenarios.
- `benchmark.yml` adds `push: branches: [main]` so each merge updates
  the `pnpm` baseline, then uploads via the Bencher CLI.
- `pacquet-integrated-benchmark.yml` adds the same main-push baseline
  for `pacquet`, combines the four scenario JSONs into a Bencher
  report, and stages it into the existing artifact.
- `pacquet-integrated-benchmark-comment.yml` uploads PR results from
  the trusted `workflow_run` context so fork PRs are covered too.

Requires `BENCHER_API_TOKEN` in repo secrets; workflows no-op with a
`::notice::` if it's missing.

* ci(bencher): allow workflow_dispatch to upload pacquet results

The inline Bencher upload was gated to `event_name == 'push'`, which
meant manual dispatch from a feature branch ran the bench but skipped
the upload. Both push and workflow_dispatch execute in the base-repo
privilege context, so it's safe to upload from both — the fork-safety
gate only needs to keep `pull_request` runs out.

On dispatch from a non-main branch we record into the ref name with
`--start-point main --start-point-reset`, matching the pnpm bench's
branch policy.

* ci(bencher): treat workflow_dispatch on main as the main baseline

Two small fixes from PR review:

- `benchmark.yml`: manual dispatch from `main` was falling into the
  non-main branch arm, recording `--branch main --start-point main
  --start-point-reset` — equivalent to forking main from itself.
  Treat it like a `push` event so a manual run on main updates the
  baseline directly. Matches the pacquet workflow's branch policy.
- `bench.sh`: emit a stderr warning when `jq` is missing instead of
  silently skipping `bencher-results.json`. Keeps behaviour optional
  for local users but makes the skip discoverable.

* bench: rename scenarios with explicit state axes

Replaces the hyperfine-leaking names (`clean-install`, `frozen-lockfile`,
`peek`, `gvs-warm`, …) with a consistent grid that spells out every
state the benchmark depends on:

- "Fresh" — node_modules wiped at start (future variants will start
  with a populated node_modules).
- "Install" vs "Restore" vs "Add new dep" — the work being measured.
- "hot/cold cache + hot/cold store" — both pnpm directories,
  spelled out separately because they're distinct on disk.
- "isolated linker" — nodeLinker mode (future variants will cover
  `hoisted` and `pnp`).

The slugs map directly from the clap-derived kebab-case names, so
`--scenario=fresh-restore-cold-cache-cold-store-isolated` is the new
CLI surface. Updates land across the Rust orchestrator
(`BenchmarkScenario`), `benchmarks/bench.sh`, the pacquet workflow,
`benchmarks/README.md`, and `pacquet/CONTRIBUTING.md` so the names
agree end-to-end.

Adds a justified `#[allow(clippy::enum_variant_names)]` on the enum
because every variant currently shares the `Fresh` prefix; the lint
will stop firing once `Filled*`/`Resynced*` counterparts land.

Bencher's stored history for the old benchmark names will become
orphaned and can be archived in the UI.

* bench: linker-first slug shape with dot-separated axes

Reshapes the scenario identifiers so the linker mode is the leading
group: `<linker>.<action>.<cache state>.<store state>`. Dots separate
the four axes the bench varies, and `isolated-linker.*` /
`gvs-linker.*` sort together in any dashboard that groups by prefix.
Future buckets (`hoisted-linker.*`, `pnp-linker.*`) will slot in
without disturbing the existing names.

GVS is its own top-level bucket rather than a sub-variant of
isolated — its perf profile differs enough to chart separately.

Renames:

- `clean-install` → `isolated-linker.fresh-install.cold-cache.cold-store`
- `full-resolution` → `isolated-linker.fresh-install.hot-cache.hot-store`
- `frozen-lockfile` → `isolated-linker.fresh-restore.cold-cache.cold-store`
- `frozen-lockfile-hot-cache` → `isolated-linker.fresh-restore.hot-cache.hot-store`
- `peek` → `isolated-linker.fresh-add-dep.hot-cache.hot-store`
- `gvs-warm` → `gvs-linker.fresh-restore.hot-cache.hot-store`

Each Rust variant now carries `#[value(name = "…")]` so clap accepts
the dotted CLI form (`--scenario=isolated-linker.fresh-install.cold-cache.cold-store`).

Display labels follow the slug structure: `Isolated linker: fresh
install, cold cache + cold store` and `GVS linker: fresh restore,
hot cache + hot store`.

The `#[allow(clippy::enum_variant_names)]` is renewed; 5 of 6 variants
share the `Isolated` prefix today. Once `Hoisted*` / `Pnp*` buckets
land the lint will stop firing on its own.

* style: apply rustfmt after scenario rename

The longer match-arm pattern produced by the linker-first rename
exceeded the rustfmt width budget. Auto-format breaks the
`&["install", "--frozen-lockfile"]` body onto its own line so the
arm stays within the limit.
2026-05-23 15:19:49 +02:00
Khải
fd564a43b9 ci(benchmark): consolidate (#11741)
Unify the two install-benchmark stacks (`benchmarks/bench.sh` and `pacquet/tasks/integrated-benchmark/`) into one shared Rust orchestrator. Scenario, fixture, workspace-manifest, install-script, and report generation live in one place so changes propagate to both stacks.

Scenario sets per workflow are unchanged: `benchmark.yml` keeps measuring the same 6 scenarios; `pacquet-integrated-benchmark.yml` keeps measuring the same 2. Both verdaccio and live-npm registry modes are preserved; neither is removed in favor of the other. All six scenarios accept both `pacquet@<rev>` and `pnpm@<rev>` targets.
2026-05-21 21:12:29 +02:00
Zoltan Kochan
9ae2e03450 chore: add benchmark script for comparing install performance against main (#10632) 2026-02-16 23:55:53 +01:00