Jake Hillion 8392e78afe bench: add spec for automatic canary benchmarks (#1483) ...

Adds all the models that can fit onto a single M3 Ultra for single
machine benchmarks. Fixes the macOS version, GPU spec, and chip type for
maximum reproducibility. Specifies the minimum memory accordingly for
each type of model, using the smallest machine available (the smallest
M3 Ultra is 96GiB).

Test plan:
- Running this with some code that makes machines of this spec available
and stores the results. It works.

This will become part of a larger testing/stability strategy once we've
collected more of the data.

2026-02-17 10:52:05 +00:00

..

src/exo_bench

bench: make exo-bench available via nix run on all platforms (#1415)

2026-02-06 21:07:17 +00:00

bench.toml

bench: add spec for automatic canary benchmarks (#1483)

2026-02-17 10:52:05 +00:00

exo_bench.py

bench: add --danger-delete-downloads flag with planning phase

2026-02-16 13:06:38 +00:00

pyproject.toml

bench: make exo-bench available via nix run on all platforms (#1415)

2026-02-06 21:07:17 +00:00

single-m3-ultra.toml

bench: add spec for automatic canary benchmarks (#1483)

2026-02-17 10:52:05 +00:00