Skip to content

Why does your AI agent fail on long jobs?

You get each model's finish rate, its cost per finished task once retries are counted, and the cheaper route.

  • Two models
  • Your task length
  • What one run costs
  • Nothing you enter is sent anywhere

Built for the essay Your Benchmark Measures a Sprint. Your Agent Runs a Marathon.

Your two figures at 40 steps
Your model13% of runs finish$7.78 per finished task
Frontier model45% of runs finish$13.46 per finished task
Cheaper route: your model
Runs finished vs chain length your modelfrontier
%
The share of steps it clears without a wrong turn you have to undo.
Don't know your number? Work it out from your last few runs
Of my last runs of steps, finished.
Enter your runs above.
%
A 3.0-point edge over your model. On a 5-step task it moves the finish rate by 13.0 points.
steps
Count every step in the chain: each tool call, edit and check from the first to the last. Long jobs run to hundreds of steps. On SWE-Marathon's v1.0 board the median logged run took 212 steps for GLM-5.2 and about 155 for Opus 4.8 (checked 7 Oct 2026).
Cost of one full run frontier ×6.0
Example costs to start. Use the bill for one run of this job, or its tokens times your price per token. A longer run costs more, so these move with the step count.
A failed run is caught
Your model finishes
13%
of runs at 40 steps · about 1 run in 8
$7.78 per finished task
Frontier finishes
45%
of the same runs · about 1 run in 2
$13.46 per finished task
The routing call
Marathon-bound

A long agent job finishes only if every step survives in order, so a per-step gap too small for a sprint benchmark to show decides the whole run.

A per-step gap too small for a sprint benchmark to show compounds into a large gap in finished work on a long job.

The essays and tools on agent reliability come with sources shown. Subscribing is free.

Subscribe opens Substack in a new tab with your address. Nothing you entered in the tool goes with it.

Runs entirely in your browser. Nothing you enter is sent anywhere. The copied link keeps your inputs after the #, a part browsers never send to a server.

How it works

A long agent job finishes only if every step goes right, so the tool raises each model's per-step reliability to the power of the step count. At the defaults (95% and 98% per step, 40 steps), 13% and 45% of runs finish. With failures caught at the end, cost per finished task is one run's cost divided by the finish rate, so retries count. The SWE-Marathon presets come from the public board, read on 7 Oct 2026. A gap under 5 points on a 5-step task is close by our cut-off. The method is in the essay, under "The gap is arithmetic".

Read the essay: the marathon gap

The exact exponent is a diagnostic. It treats steps as independent. Real agents recover, which raises the effective per-step reliability you set above, and one wrong step can also spoil the steps after it, which lowers it. A failed run that resumes from a checkpoint loses only the steps since that checkpoint, which is what the split-it line prices. The point survives the caveats: reliability gaps that round to nothing on a short task go nonlinear once the work has to survive many handoffs.