Opening the index
You get each model's finish rate, its cost per finished task once retries are counted, and the cheaper route.
Built for the essay Your Benchmark Measures a Sprint. Your Agent Runs a Marathon.
A long agent job finishes only if every step survives in order, so a per-step gap too small for a sprint benchmark to show decides the whole run.
The essays and tools on agent reliability come with sources shown. Subscribing is free.
Subscribe opens Substack in a new tab with your address. Nothing you entered in the tool goes with it.
A long agent job finishes only if every step goes right, so the tool raises each model's per-step reliability to the power of the step count. At the defaults (95% and 98% per step, 40 steps), 13% and 45% of runs finish. With failures caught at the end, cost per finished task is one run's cost divided by the finish rate, so retries count. The SWE-Marathon presets come from the public board, read on 7 Oct 2026. A gap under 5 points on a 5-step task is close by our cut-off. The method is in the essay, under "The gap is arithmetic".
The exact exponent is a diagnostic. It treats steps as independent. Real agents recover, which raises the effective per-step reliability you set above, and one wrong step can also spoil the steps after it, which lowers it. A failed run that resumes from a checkpoint loses only the steps since that checkpoint, which is what the split-it line prices. The point survives the caveats: reliability gaps that round to nothing on a short task go nonlinear once the work has to survive many handoffs.