The peer-heavy resolver fixture has a wide run-to-run spread — a local 15-run
pass measured a ~15% coefficient of variation on an idle machine, with the slow
samples scattered through the sequence rather than clustered at the start, so
the noise is runner contention rather than a cold cache. Three CI runs against
that spread produced a 2.83s outlier on pnpm/pnpm#13551 that lifted the pacquet
mean to 2.15s while its fastest run sat at 1.54s, inside main's range.
Compare the cross-engine gate on each target's fastest run instead of its mean.
Contention only ever adds time, so the minimum is the sample least perturbed by
a noisy neighbour; this is already the statistic the workflow reports to
Bencher, for the same reason. Raise the scenario to nine runs so that minimum
has a good chance of landing on an uncontended run, and surface both mean and
min in the diagnostics table.
The gate is a correctness check, not a measurement, and it was costing us the
measurement: asserting mid-job aborted before the Bencher upload and dropped
every scenario's data point for the whole run. That is why this benchmark has
almost no history on bencher.dev — the two main pushes after it was added both
failed on the since-recalibrated 10x floor, so only one point was ever
published. Record the verdict, let the upload proceed, and re-raise the failure
afterwards.
Pacquet-only benchmark-harness and CI change; no user-visible surface, no
TypeScript counterpart, and no changeset.
Zoltan Kochan
fa8d383cca