The Marathon Calculator

Why does your AI agent fail on long jobs?

A long agent job finishes only if every step survives in order, so a per-step gap too small for a sprint benchmark to show decides the whole run. Set two models, your task length and what one run costs. You get each model's finish rate, its cost per finished task once retries are counted, and the cheaper route.

Built for the essay Your Benchmark Measures a Sprint. Your Agent Runs a Marathon.

Your two figures at 40 steps
Your model13% of runs finish$7.78 per finished task
Frontier model45% of runs finish$13.46 per finished task
Cheaper route: your model
%
The share of steps it clears without a wrong turn you have to undo.
Don't know your number? Work it out from your last few runs
Of my last runs of steps, finished.
Enter your runs above.
%
A 3.0-point edge over your model. On a short benchmark the two models look close.
steps
Count every step in the chain: each tool call, edit and check from the first to the last. Long jobs run to hundreds of steps. On SWE-Marathon's v1.0 board the median logged run took 212 steps for GLM-5.2 and about 155 for Opus 4.8 (checked 7 Oct 2026).
Cost of one full run frontier ×6.0
Example costs to start. Use the bill for one run of this job, or its tokens times your price per token. A longer run costs more, so these move with the step count.
A failed run is caught
Runs finished vs chain length your modelfrontier
Your model finishes
13%
of runs at 40 steps · about 1 run in 8
$7.78 per finished task
Frontier finishes
45%
of the same runs · about 1 run in 2
$13.46 per finished task
The routing call
Marathon-bound

Subscribe to get the next structural lens in your inbox.

Free. New tools and the essays behind them, with every source shown. The button opens Substack; nothing you entered here goes with it.

Subscribe free
Runs entirely in your browser. Nothing you enter is sent anywhere. The copied link keeps your inputs after the #, a part browsers never send to a server.
Read the essay: the marathon gap

The exact exponent is a diagnostic. It treats steps as independent. Real agents recover, which raises the effective per-step reliability you set above, and one wrong step can also spoil the steps after it, which lowers it. A failed run that resumes from a checkpoint loses only the steps since that checkpoint, which is what the split-it line prices. The point survives the caveats: reliability gaps that round to nothing on a short task go nonlinear once the work has to survive many handoffs.