Compare integrated benchmark samples against main measured on the same runner.
Keep absolute minimum timings in Bencher for historical tracking without letting
cross-machine historical alerts overwrite the PR regression check.
Require at least nine successful samples per target. Fail when the median
slowdown and retained sample separation both exceed 5% of main's median or
2 ms, whichever is larger. Trim up to 10% from each tail. Report other median
slowdowns above that tolerance as inconclusive. This conservative noise guard
is not a statistical confidence interval; smaller or noisy regressions can pass,
and sequential measurements remain vulnerable to runner drift.
Run the comparison from trusted default-branch code in the privileged comment
workflow, publish the report before enforcing the result, and preserve the
existing check name on the triggering PR commit. Keep the peer-heavy
Rust-versus-TypeScript assertion unchanged.
Cover regressions, shared machine slowdowns, outliers, invalid data, all scenario
and engine combinations, command names, CLI exit codes, and the duration-dependent tolerance with Node tests.
Keep the explanation in the existing benchmark documentation.
Zoltan Kochan
bd4d526530