Jesse Gross 505e35f2b9 mlxrunner: choose the speculative draft length to maximize throughput ...

The heuristic schedule grew the draft toward a fixed cap on acceptance alone,
maximizing accepted-tokens-per-step rather than throughput, and on a
steep-forward target it regressed below no speculation. Replace it with an
engine-level controller that drafts the depth maximizing
committed-tokens-per-wallclock from live per-position acceptance and persisted
per-width forward cost, with no draft-length cap; the heuristic schedule and
the OLLAMA_MLX_MTP_* env vars go with it.

2026-06-22 15:25:45 -07:00

..

base.go

mlxrunner: choose the speculative draft length to maximize throughput

2026-06-22 15:25:45 -07:00