Walkthroughs · The Durability Curve

The Graph Looks Right. The Merge Is Where It Breaks.

Sun, 30 Aug 2026 00:00:00 GMT

Three agents disagreed, and the gate I built approved the release anyway. The framework had caught the conflict; the bug began the moment I made the error go away. Here is the one line that did it, and how to see it in your own graph.

Verified end to end on langgraph 0.6.11.

I built a release gate out of agents. An orchestrator fanned out to three reviewers: one read the test run, one read the error metrics, one read the changelog. Each came back with a verdict, ship or hold or roll back. A final node read the verdict and acted on it. I drew the graph, and it looked like every orchestrator-worker diagram you have ever seen. After one wiring error I thought I had fixed, I ran it again, and it worked.

Then it approved a release that two of the three reviewers had rejected.

There was no exception, and nothing turned red in the log. The graph had done exactly what I wired it to do, and what it did was take the ship branch. I went looking for the bug in the reviewer prompts, and they were fine. Then in the routing, and that was fine too. The verdicts were correct. Two of them said stop. The code that read them looked like this:

if state["decision"] == "roll back":
    abort_release()
else:
    ship()

And state["decision"] was "ship it". One verdict was there, the other two were gone. That is the part that took me a while to accept: it was never three reviewers reaching a decision. It was three agents writing to one slot, and the code downstream read whichever write survived. I had never said who should own that slot, so something I had wired without noticing answered the question for me, and answered it wrong.

The picture is not where the fault is

Put the broken graph next to a correct one. Orchestrator at the top, three workers below it, a node at the bottom that makes the call. The two diagrams are the same boxes and arrows, and the fault is in neither of them. It is one line down in the state schema, a line you probably wrote without thinking about it, that decides what happens when three workers write the same slot. Ranjan Kumar put the general point better than I will, in a piece worth reading in full: the rule that resolves those writes “appears on no diagram, in no edge list, and in no type checker’s output,” and it is the only thing they all miss that decides what the state value actually is.¹ The difference between my two graphs lives there, in the schema, and the schema is what you have to read.

The error was the invariant

In LangGraph your state is a set of channels, and every channel has a rule for what happens when more than one node writes it in a single step. That rule is the whole game.

Wire three parallel workers to write a plain channel, the way my release gate did, and run it. LangGraph stops you, at the exact key:

InvalidUpdateError: At key 'decision': Can receive only one value per step.
Use an Annotated key to handle multiple values.

I read this as an inconvenience, which is the mistake. It is the one place the framework asked me a question I had not answered: who is allowed to write this field. Three writes arrived for one slot, and rather than pick a winner behind my back, it refused. Left here, the bug cannot ship, because the exception is holding an invariant I never wrote down.

So I made the exception go away, the way the message suggests: add a reducer. One documented reducer pattern is operator.add. On a str channel it concatenates the three verdicts into "roll backship ithold", which is visible garbage you would catch. On a list channel it keeps all three, which is correct, and only a downstream decision[0] quietly throws two away. Neither of those is the silent single-survivor I hit. To get that I reached past the docs for the smallest reducer that makes the error stop, keep the latest write:

class State(TypedDict):
    decision: Annotated[str, lambda old, new: new]

That is the trap, and it is worth being exact about where it comes from. LangGraph’s own error page shows the lossless operator.add. But overwrite reducers like lambda old, new: new are a familiar pattern too, in examples and in real code, so this is not one careless engineer. The framework refused, I reached for a familiar reducer to silence it, and I did it on a field where overwriting meant throwing away a verdict. The run went clean, the channel held one verdict, and the other two were gone with no error:

$ python agent_graph_demo.py --trap
STEP 1 (plain channel): the framework refused -> InvalidUpdateError: At key 'decision': Can receive only one value per step. Use an Annotated key to handle multiple values.
STEP 2 (reducer added): no error. the decision channel now holds: 'hold'
        the other two reviewers' verdicts were silently dropped

Here is the idea I want you to keep, because it travels past LangGraph. A reducer answers one question: when two writes land on this field, how do they combine. It cannot answer the prior one: should two writes ever land on this field at all. That first question is about ownership, and the type does not answer it. A str can be a log fragment or an authoritative verdict; an int can be a counter, a balance, or a version number. The same type permits a merge that is right for one meaning and ruinous for another, because the safe concurrency policy depends on what the value means, and the meaning is not in the type. Last-write-wins itself is not the villain: it is right when overwrite is part of the field’s contract, and wrong when competing writes are evidence that has to be reconciled first. So the order matters: decide who owns a field before you decide how its writes merge. For a decision, exactly one node owns it. The InvalidUpdateError was enforcing the same-step half of that, no more than one write to this field in a single step, and I silenced the one guard I got for free. The other half, that no other node writes it in a later step, the error cannot see, and we get to it below.

One point of precision, since a careful reader will want it. This is a deterministic super-step conflict, not a thread-level data race; LangGraph batches the writes from a step and applies the channel’s rule to them. And which verdict survives is not random: in this static fan-out on 0.6.11, the write from the node whose name sorts last wins, which I know only because I reordered the nodes and watched it change. It is stable across re-runs and it is an undocumented detail you should never build a decision on.

The graph that keeps the disagreement

Here is the same release gate, wired so ownership is explicit. The move is to separate two jobs that were fighting over one channel: gathering what the reviewers found, and deciding what that means. Gathering has many writers. Deciding has one.

Verdict = Literal["ship it", "hold", "roll back"]

class Finding(TypedDict):
    reviewer: str
    verdict: Verdict

class State(TypedDict):
    findings: Annotated[list[Finding], operator.add]   # gather: many writers append
    decision: Verdict                                  # decision: one owner writes

def reviewer(source):
    def node(state):
        return {"findings": [{"reviewer": source, "verdict": call_model(source)}]}
    return node

def decide(verdicts: list[Verdict]) -> Verdict:
    # the policy, named and owned. this one is any-veto by severity; swap it for
    # majority / unanimous / weighted / threshold as your gate actually needs.
    if "roll back" in verdicts: return "roll back"
    if "hold" in verdicts:      return "hold"
    return "ship it"

EXPECTED = {"tests", "metrics", "changelog"}

def writer(state):
    reviewers = [f["reviewer"] for f in state["findings"]]
    if sorted(reviewers) != sorted(EXPECTED):   # missing or duplicate is not a ship
        return {"decision": "hold"}
    return {"decision": decide([f["verdict"] for f in state["findings"]])}

The reviewers write concurrently to the same gather channel, and here the merge is legitimate: keeping every contribution is exactly what the field means. Concurrency was never the bug; a wrong ownership rule was. Then one node, and only one, reads all three findings and applies a policy that is written down where you can argue with it. Notice the policy is a veto by severity: one roll back overrides two ships, because that is the release rule I want. You can pick any rule you like. What matters is the split: gathering the evidence and ruling on it are different operations, so they get different channels, and the ruling has an owner. Ownership is only part of the gate. decide assumes every reviewer reported, so a reviewer that drops out without a finding is a silent ship in a different coat, and one that reports twice can tip a majority or a weighted rule. The writer fails closed on both: it rules only when every expected reviewer reported exactly once, and anything else is a hold. Ownership says one node decides; completeness says everyone reported; uniqueness says each reported once. Run it with all three present and it reconciles three of three, on a decision a node made after seeing them.

Two things a builder asks here. When you do not know how many reviewers you will have, Send gives you dynamic fan-out, and the footgun is identical: I wired a Send fan-out into a shared decision channel and it raised the same InvalidUpdateError, so the same split fixes it. And swapping the stub for a real model is two lines, a ChatAnthropic or ChatOpenAI in place of the stub; the graph, the channels, and the ownership rule do not change, because the model is a node and the safety is in the wiring.

Whether to split the work across agents at all is a separate question, and I argued it in Your Multi-Agent System Is an Org Chart. This walkthrough assumes you have decided to split, and shows the line where the split silently is not real.

Read it off your own schema, then off your own run

Your real graph has more than three channels, and you will not remember which carry decisions. You can have a script find the risky ones. Kumar ships one that inspects a compiled graph and reports each channel’s merge policy; the small version I use reads the State schema itself:

python audit.py            # or: from audit import audit_schema; audit_schema(MyState)

It works because a fan-out cannot silently collide on a plain channel: the framework refuses that write out loud. The only place a fan-out’s writes merge without a word is a channel that carries a reducer, so the reducer channels are the whole fan-out surface, and the tool reads them off the annotation:

audit of State
  ! classify    findings       REDUCER (add)
  fan-out safe  decision       plain

For each flagged channel, ask how its value gets read downstream. Does anything branch on it, or act on its value as one answer? Then it is a decision channel: give it one writer and no reducer. Is it only ever folded into an aggregate? Then it is a gather, and the reducer belongs there.

That check is real, and it is only half. It is a reducer-surface audit: it finds where a merge is permitted. It does not prove a decision channel has exactly one writer, because which channels a node writes is a dictionary it returns at runtime, not a type you can read ahead of time; a node’s introspectable channels are its reads, not its writes. So a plain decision channel that two nodes write in sequence overwrites silently, no error, and the schema cannot see it. The second check is a runtime one: in a test, assert that across the whole run, every write to a decision channel came from the same one node. That holds even for a looping graph, where the owner may write many times but no one else may write at all. The first check finds where merges are allowed; the second finds where ownership is broken.

Cap the loop, count the cost, gate the write

Ownership is the idea to carry out of this. Three smaller safeguards finish a graph once you have made it safe on that.

A loop with no exit does not run forever; LangGraph stops it with GraphRecursionError at a default limit, 25 on the 0.6.11 this piece was built on and far higher on current releases. The reflex is to crank that limit up, which only buys a stuck agent more steps and a larger bill. Right-size it a little above your real longest path, and read a trip as “this agent is stuck,” not “this agent needs more room.”

Fan-out is not free. Three parallel workers are three model calls, which the demo counts for you, and running them in parallel buys you wall-clock time and nothing on the bill, because you pay for every branch. None of the failures in this piece is fixed by a better model; they are decisions about ownership and cost that a smarter model makes faster, not safer.

And the decision node, being the one whose output the graph acts on, is where a human gate belongs if you want one. LangGraph’s interrupt() gives you that, with one caveat worth its own walkthrough: on 0.6.11, resuming needs a checkpointer, and on resume the node runs again from the top, so the irreversible work goes below the interrupt, not above it. That is a whole article; here it is enough to know the reliability budget belongs on the write, because the write is the part that is owned.

Most of the time, do not build a graph

Most of this you avoid by not reaching for a graph. A single agent with tools and a step that compresses its own history handles more than people expect, and it avoids the fan-out and merge failure this walkthrough has been about, because there is nothing to fan out and nothing to merge. It can still loop, lose context, or run up a bill; it just cannot lose a verdict in a merge. Single prompt, then tools and retrieval, then a fixed workflow, then one agent, then several: climb a rung only when the one below it measurably fails.² A graph is a cost, and you pay it in exactly the channels this walkthrough has been about.

What to do this week

Take one graph you already run. Pass its State schema to the audit, and for every reducer channel it flags, ask whether anything downstream acts on it as a single value; give each one that does a single writer. Then, on purpose, wire a fan-out into a plain decision channel and watch LangGraph refuse it, add a last-write-wins reducer to make the error stop, and watch two verdicts disappear. The next time that sequence happens, let it be in a test you wrote and not a release you shipped.

My release gate has one decision owner now. The three reviewers still disagree; they simply do not get to settle it by racing. The reducer was never meant to decide who was right.

The three programs here, verify.py, agent_graph_demo.py, and audit.py, run with no API key and are yours to keep: durabilitycurve.com/tools/agent-graph.

And if you want the paper version to run on a graph you already have, The State Channel Audit is a free three-step worksheet, classify every channel, guard every decision, run the two checks. Download the worksheet (PDF).

Ranjan Kumar, LangGraph Reducers Are a Concurrency Policy, 27 July 2026, reached all of this before I did: the reducer as a concurrency policy, the review-surface blind spot, the sequential silent overwrite, and a compiled-graph merge-policy auditor. His piece is the one to read for the general case. What is new here is the ownership-before-merge-policy framing, the worked release-gate failure, and a runnable no-key demo. ↩
The architecture-level sources behind the single-writer rule and the orchestrator-worker pattern are Cognition’s Don’t Build Multi-Agents (2025) and its 2026 follow-up Multi-Agents: What’s Actually Working, which finds these systems work best when writes stay single-threaded and the extra agents add intelligence rather than actions; and Anthropic’s Building Effective Agents (2024) and multi-agent research write-up (2025). On how these systems fail in the field, the MAST study (Cemri et al., 2025) finds specification and system design the largest of three roughly balanced failure categories, and the authors trace many failures to system design rather than model limits alone. ↩

Your Tests Pass. So What?

Tue, 14 Jul 2026 00:00:00 GMT

This is the second walkthrough in a series. The first built a gate: a Stop hook that will not let Claude Code end its turn while any test is failing, so it cannot call a job done on a red suite. This one asks whether the tests behind that green are worth passing, and you do not need to have read the first to follow along.

What you will do: measure how many real bugs your test suite can actually catch. In the worked example, a green suite catches just one planted bug in ten; among the nine it misses is a discount that quietly becomes a surcharge. A tighter case shows the sharper trap: a test can cover every line of a function and still notice nothing. You will watch both scores land, then hand Claude Code the holes and make it close them. About twenty-five minutes for the worked example; a first pass on your own repository takes your whole suite’s runtime once for every mutant, so start with your smallest tested file.

Who this is for: you gate Claude Code, or any coding agent, on green tests, and the suite has been reassuringly green ever since.

Who should skip it: if you already run mutation testing, this is your Tuesday. Step 3, turning your existing survivor backlog into an agent work order, may not be.

Never written a test? Your version of the whole method is at the end: ten minutes, three planted errors, no code.

You need: Python 3 (3.9 or later, standard library only, nothing to install). Claude Code for step 3; steps 1 and 2 run without it.

Then: when it still goes wrong, proving it on code you ship, and a version if you never write code.

I broke a production module of mine on purpose, 105 small ways, one at a time, and ran its test suite after every break. The suite is real: 21 regression tests, all green, guarding an 822-line file my own automation depends on. The tests noticed 38 of the 105 breaks. The other 67, one of them on a line the suite executes every single run, would have shipped without a sound.

That experiment is the whole method, and it answers a question a passing test suite cannot. The gate proves the tests pass. It cannot prove the tests are worth passing; a test that asserts nothing sails straight through it. And if your instinct is to have a second agent check the tests, then a third to check the second, the regress ends here instead, at an experiment rather than another opinion: break the code on purpose, and see whether the alarm rings.

Watch a test do nothing

(You know why an assert-nothing test passes? Skip to step 2; this section is the on-ramp.)

Here is the toy calculator shop from last time, one walkthrough later. Its add works, and the suite is two small tests. The gate is green. (If you download the companion folder, calc.py also carries a new pricing function, which is step 2’s problem, and check.sh, the last walkthrough’s gate script, along for the ride. Leave both alone for now.)

calc.py

def add(a, b):
    return a + b

test_calc.py

import unittest

from calc import add


class TestAdd(unittest.TestCase):
    def test_basic(self):
        self.assertEqual(add(2, 3), 5)

    def test_zero(self):
        self.assertEqual(add(0, 0), 0)

# ...the unittest.main() entrypoint is unchanged below

Save both files in one folder and open your terminal there; the import from calc import add only resolves if they sit together. (Downloading the folder in step 2 does this for you.) One of these two tests is doing almost all of the work, and one of them is doing almost none. You can find out which in thirty seconds. Break add on purpose (return a - b, the classic bug) and run the suite, python3 -m unittest, the same command the gate runs: test_basic fails. Now, with the code still broken, run only the other test:

$ python3 -m unittest test_calc.TestAdd.test_zero
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Green, on code that subtracts. add(0, 0) is 0 whether add adds, subtracts, or multiplies, so test_zero approves all three. It has one way to fail, and almost no wrong version of the code triggers it. Note what your coverage tool would say about it: test_zero executes every line of add, one hundred per cent, top marks. Coverage measures whether tests run the code. It has no opinion on whether they would notice anything. test_zero has sat in that suite looking exactly as load-bearing as test_basic, and it would wave the classic bug straight through the gate on its own.

A test you have never watched fail is not yet a test.

Test-driven veterans have said a version of that for twenty years: never trust a test you haven’t seen fail. The discipline is the same, and the move is the one you just made: break the code on purpose, and the test either notices or it does not. Once, by hand, takes thirty seconds. Every plausible break is a script. Fix add back before you move on; the script is next.

Count what your suite can see

The shop grew this week. Claude Code added a pricing function, and the suite is green, which is all the gate checks. Here is the function:

def discounted_total(price, quantity, discount_percent):
    """Total cost of an order, in pounds.

    Orders of 10 or more items get discount_percent knocked off.
    """
    total = price * quantity
    if quantity >= 10:
        total = total * (1 - discount_percent / 100)
    return round(total, 2)

Money code. A boundary, a formula, a rounding rule: three places to be quietly wrong. (Yes, real tills count integer pence rather than floating-point pounds; hold that thought, it returns at the end.) The question you cannot answer by reading the green gate: if one of those went wrong tonight, would any test notice?

mutate.py answers it by brute honesty. It is a transparent teaching instrument, 180 lines of standard library you can read top to bottom, built to make the mechanism visible on one file rather than to replace a mature framework. Its whole engine is the dozen classic ways code goes wrong that it knows how to plant, one character at a time:

OP_SWAPS = {
    ast.Add: ast.Sub,    # a + b   ->  a - b
    ast.Mult: ast.Div,   # a * b   ->  a / b
    ast.GtE: ast.Gt,     # >=      ->  >   (the off-by-one at every boundary)
    ast.Eq: ast.NotEq,
    ast.And: ast.Or,
    # ...the reverse of each, plus < and <=, a dozen swaps in all,
    # and integer nudges: 10 -> 11
}

For each place in your file where one of those swaps applies, it makes that one change (a mutant of your code), runs your whole suite, restores the file, and records the verdict. Before any of that it runs your suite once on the untouched code, timed: a red baseline gets refused outright, because against a suite that is already failing, every verdict is noise. Then the two outcomes. A mutant that makes at least one test fail is killed: the alarm rang. A mutant that leaves every test green survived: that exact bug could ship tonight.

Grab the folder, hosted on my own site; it is short, dependency-free Python you can read before you run it. Unzip it and open your terminal inside the tests-worth-passing folder; every command below runs from there, no setup.

The four working files are calc.py, test_calc.py, mutate.py, and check.sh (last walkthrough’s gate, along for the ride); a README and the agent’s finished tests sit alongside them and need nothing from you. When you are ready to point this at your own code, the folder’s AUDIT.md carries the reusable version: the audit protocol, a survivor-triage worksheet, and the command for your language.

Now stop before you run it, and put a number down. Your suite is green and it passes. Of ten deliberate breaks to this file, how many do you think it catches? Hold that guess against the result:

$ python3 mutate.py calc.py
10 mutants of calc.py · suite: python3 -m unittest -q
baseline: suite green in 0.2s (per-mutant timeout 60s)

  1 KILLED    line 2: a + b  ->  a - b
  2 SURVIVED  line 10: price * quantity  ->  price / quantity
  3 SURVIVED  line 11: quantity >= 10  ->  quantity > 10
  4 SURVIVED  line 11: 10  ->  11
  5 SURVIVED  line 12: total * (1 - discount_percent / 100)  ->  total / (1 - discount_percent / 100)
  6 SURVIVED  line 12: 1 - discount_percent / 100  ->  1 + discount_percent / 100
  7 SURVIVED  line 12: 1  ->  2
  8 SURVIVED  line 12: discount_percent / 100  ->  discount_percent * 100
  9 SURVIVED  line 12: 100  ->  101
 10 SURVIVED  line 13: 2  ->  3

Score: 1/10 killed, 9 survived.
Every SURVIVED line is a change to your code that your whole test suite
cannot tell from the version you meant to write.

The sixth mutant is the one to read twice. 1 - discount_percent / 100 became 1 + discount_percent / 100: a customer’s 20 per cent discount becomes a 20 per cent surcharge, and the gate stays green. The third is the boundary: >= became >, the customer buying exactly ten items loses the discount you promised them, green. Nine ways for money code to be wrong, and the suite from step 1 sees none of them, because nothing in it ever calls the new function. The gate never lied. “The tests pass” was true every time it said so. It just was not the thing you needed to be true.

A fair objection: nothing tests that function, so a plain coverage report would have flagged it too, without any of this mutant theatre. True, and if that were all mutation testing found, you would not need it. So run the case coverage cannot see. Delete test_basic, keep only test_zero, and run the instrument again:

$ python3 mutate.py calc.py
10 mutants of calc.py · suite: python3 -m unittest -q
baseline: suite green in 0.1s (per-mutant timeout 60s)

  1 SURVIVED  line 2: a + b  ->  a - b
  ...
Score: 0/10 killed, 10 survived.
...

The subtraction bug now survives, on a function with one hundred per cent line coverage. Your coverage dashboard reports add fully tested; the instrument reports that no test would notice if it subtracted. Coverage tells you the code ran. A kill tells you a lie got caught. They are different instruments, and only one of them is measuring what you care about.

The same function can be one hundred per cent covered and zero per cent detected.

This is not a toy-only failure, and it is the shape to hold on to. On the 822-line module I opened with, the survivor that stung most was exactly this: a covered line, inside a function the suite runs every single time, that no assertion actually pins. Put test_basic back and carry on.

One reading note before you run it on anything you love: killed is the good outcome. Every killed mutant is a bug class your suite would catch, so on this one screen a KILLED is the line you are hoping for, even though the word sounds like something broke.

This move is called mutation testing, and it is older than most of the code you have ever shipped. Breaking one character at a time is a fair stand-in for the elaborate bugs real code grows, because of the coupling effect: catch the small, dumb faults and you catch the large subtle ones as a by-product.¹ The nine survivors here are the ones your suite missed, and every one of them is now a job you can hand to the thing that wrote the tests.

Hand the survivors to the agent

Nine survivors is a work order, addressed to the thing that wrote the tests. Paste the nine SURVIVED lines into Claude Code, followed by this instruction:

These mutants survived mutation testing: the suite stays green when any one of these changes is made to calc.py. Write tests in test_calc.py that kill them. Do not modify calc.py or mutate.py. Then run python3 mutate.py calc.py and keep going until the score is clean.

When I ran exactly that, the agent came back with four tests and a habit I did not ask for: it annotated each one with the wrong answer the mutant would produce. The survivor list had turned into a specification it could compute against.

def test_discount_applies_at_exactly_ten_items(self):
    # Boundary: quantity == 10 qualifies. 10.0 * 10 = 100, minus 20% = 80.0.
    # Kills the > / >=, threshold 10->11, and every discount-formula mutant:
    #   total / (1 - d/100) -> 125.0
    #   total * (1 + d/100) -> 120.0
    self.assertEqual(discounted_total(10.0, 10, 20), 80.0)

def test_rounds_to_two_decimal_places(self):
    # 3.333 must round to 3.33; a round(total, 3) mutant returns 3.333.
    self.assertEqual(discounted_total(3.333, 1, 0), 3.33)

Two of its four tests are above; the finished suite, those four plus the two you started with, ships as calc_solution_tests.py in the folder (named without the test_ prefix so it stays out of your way until you want it), so you can run it yourself instead of retyping it from the excerpts. To see the clean score without doing step 3, drop that file in as test_calc.py and run python3 mutate.py calc.py. Left as it downloads, the folder still scores 1/10, because the suite the gate runs holds only the two starting tests.

Then I ran the instrument again myself, because you never take the agent’s word for a score when you can take the score’s word for it:

$ python3 mutate.py calc.py
...
Score: 10/10 killed, 0 survived.

Exit code 0, the instrument’s all-clear. Same code, same tests passing as before, and now green means something it did not mean before: ten classic ways to break this file, and a test rings for every one. (My run went clean in one round, but this toy is small. When yours does not, paste what survived straight back and go again; and for the rare survivor no round can kill, the closing section shows the other move: you let it live, with a comment.)

One step remains, and it is the one that actually ends the regress: read the four tests it wrote, and check every asserted value against what the code should do, not against what it does. The distinction is load-bearing. An agent kills mutants by pinning current behaviour; look back at its annotations and you can see the 80.0 was computed from the implementation. On correct code that is exactly what you want. On code that is already wrong, the same move canonises the bug as specification, with a clean mutation score as its alibi.

The score proves the tests ring when the code changes; only your read proves they are ringing for the truth.

What the instrument buys you is that the read is bounded: four short tests with a known purpose, instead of an unbounded hope about a whole suite.

Two honest notes before you rely on it.

Where it earns its keep. A fresh agent asked cold to “add tests” for that function, with no survivor list, scored 10/10 on its first try. On a four-line function whose docstring names the threshold, the odds are friendly, and the survivor loop buys you little. That is not the point. The point is that on real code you cannot tell whether it guessed well by looking, and the survivor list is what turns “write better tests” from a vague instruction into a measurable work order. The payoff climbs exactly where you cannot eyeball it: a forty-line function, a docstring that has drifted from the code, a module you inherited and half-trust.

None of this is new, and that is the reassuring part. Handing a test-writer a list of surviving mutants is called mutation-guided test generation, and search-based tools were doing it a decade before anyone had an LLM; the agent is just a better test-writer than they were. Google’s setup is the part that matters for you here: it shows surviving mutants in code review, on the lines a change touches, which its authors argue is where a developer is most likely to act, after they found no actionable way to hand engineers a whole-repository score.² The survivor list works the same way: a short work order on the code you just changed. Meta has published a version of this pattern with an LLM at the test-writing end.³ You are running the individual-developer version of a documented industrial practice.

When it still goes wrong

It is slow on a real repo. Every mutant is a full suite run, so keep it out of the Stop hook: the gate fires every turn, this audit runs when the code or tests change. Industrial tools go further, mutating just the lines a change touches, which is how Google runs it at scale.
A clean score is a bounded claim. 10/10 covers only this tool’s small vocabulary of breaks. Real gaps live outside it: money in binary floats that no swap can expose, and “kills” that are really crashes, not caught assertions. The score narrows the worry; it does not end it.
Some survivors cannot be killed. An equivalent mutant reads differently but behaves identically, so no test can catch it. If you cannot name an input that would expose a survivor, it may be one, and it is allowed to live with a comment.
The score is a worklist. Optimise the percentage like a KPI and you get tests engineered to twitch at mutants rather than to state what the code should do, which is the disease “the tests pass” had, one level up. Chase named survivors; ignore the number.
The verdict contradicts the file you are reading. You are running a stale compiled copy of the file you mutated. Delete __pycache__ or touch that file. (mutate.py already gives its own runs a fresh cache.)
Not on Python? The instrument is language-specific; the method is not. Reach for mutmut on Python, PIT on the JVM, Stryker on JavaScript and TypeScript, cargo-mutants on Rust. The rule holds everywhere: break the code, count the catches.

Prove it on code you ship

Pick the smallest file in your own project that has tests you trust. Four moves before you run it:

Confirm a green baseline. The instrument refuses a red baseline, because against a failing suite every verdict is noise.
Point TEST_COMMAND at your test runner. It sits at the top of mutate.py, defaulting to plain unittest. On pytest, swap that one line, keeping it a list of arguments, not a string: [sys.executable, "-m", "pytest", "-q", "tests/test_foo.py"].
Narrow it to the file you are mutating. Aim at just the tests that exercise that file, not the whole suite. Every mutant runs the command once, so a wide one is the difference between a coffee and an afternoon, and the reason people quit after one slow run.
Predict, then compare. Write down the score you expect before you look. The gap between the number you predicted and the number you got is the most useful thing this walkthrough produces.

Here is the anatomy of the number I opened with, re-run the morning this published so it is evidence and not a memory. The module: 822 lines of my own automation, 21 green regression tests. The run: 105 mutants, 38 killed, 67 survived. Where the 67 hid is the whole lesson. Point coverage at the same suite and the module reads 43 per cent covered; 49 of the survivors sit in code no test executes at all, which a coverage report already flags for you. The other 18 are the ones only mutation can see, because they sit on lines coverage counts as covered.

One function holds most of them: a weekly counter the suite genuinely imports, calls, and runs green, whose single test asserts that the count is a non-negative integer and checks nothing else. So flip the sign on its seven-day window and it looks a week into the future, the count silently collapses toward zero, and all 21 tests still pass. That is test_zero from step 1 wearing a production badge: a fully-covered function that no assertion pins, in my own code, caught by the instrument you just ran on a toy.

So I did step 3 on my own code. I handed that survivor to the agent with the same work order, and it wrote the test the counter never had: stage a single item inside the window, then assert the count is exactly one, not merely non-negative. I applied the sign-flip by hand and ran both assertions.

current_weekly_promotion_count() = 0   (one item, aged 1 day, window = 7 days)
  assert count >= 0   ->  PASS   (the assertion that was already there)
  assert count == 1   ->  FAIL   (the assertion the agent just wrote)

The old check stayed green on a function that now counts nothing, exactly as it had all along. The new one went red, because a window flipped a week into the future finds zero where it should find one. One survivor, handed over and killed, on the first test that pinned a number instead of trusting a sign.

The fix for most of the others is that same move: pin the number the code should return, so a window that flips or a boundary that slips finally has somewhere to fail.

One survivor on that function is the exception. The window admits an item on mtime >= cutoff, and to tell >= from > you need a file whose modification time lands on the exact cutoff instant. The cutoff is read from the clock, so the only way to hit that instant is to mock the clock, and I judged that test double heavier than the bug it would catch. So it stays alive, with a comment naming why.

It is worth being precise about what it is, because it is not an equivalent mutant: an equivalent mutant is one no input can expose, and this one has an input, a mocked clock, that I have simply declined to write.

That is step 3 on the days the loop does not go clean: some survivors earn a real assertion, some earn a refactor ticket, one earns a comment explaining why it lives, and telling which is which is the work. If your score stings, the sting is information, the first honest measurement your suite has ever had.

Then add this rule to your CLAUDE.md, the standing instructions your agent reads each session, so the discipline survives you forgetting it:

When you fix a bug: first write a test that fails on the broken code,
show me it failing, then fix the code. A test I have never seen fail
does not count as coverage.

That is the manual mutant from step 1, promoted to standing policy: every bugfix now arrives with proof that its test can ring.

You are holding three instruments now, and most arguments about testing are really an argument between two of them. Coverage asks whether a line ran. Mutation asks whether a break in that line would be caught. Your own read asks whether the behaviour a passing test pins is the one you actually wanted.

Coverage measures reach. Mutation measures detection. Only your read decides what is right.

The gate keeps the agent honest about whether the tests pass; mutation and your own read are what keep you honest about whether they are worth passing. Green stops being where the question ends. It moves from are the tests passing to what broken versions did these tests catch.

If you never write code

Break the code, count the catches: the method never depended on Python. It works on anything an agent hands back where being wrong is checkable, and the case you most need it for is not code at all. It is the claim.

Here is the whole thing with no code in it. Take a report or a summary you know cold. Copy it, and plant a few deliberate errors in the copy, the kind of wrong that would cost you something if it slipped through. Hand the copy to your “review this” agent, and count what it catches.

I ran it on one of my own weekly stats summaries before publishing. In a copy I changed a signup count from four to six, moved a date back by a day, and added a confident sentence that one post had our weakest open rate, which it did not. Then: “check this against the raw export.” It caught all three, and flagged a fourth claim I had written carelessly and never planted.

Three planted, three caught, and the review earned my trust the same way step 1’s test did: I had watched it catch mistakes I controlled before I believed the ones I did not. A review you have never watched catch a planted error is worth exactly what a test you have never watched fail is worth.

The catches are your kills; the misses are the claims you must never hand over unchecked. Code is the easy case, because a machine runs it and a mutant is unambiguous. The claims your agent writes when there is no suite to run, the summary and the analysis and the report, are the harder case, and the one worth an instrument of its own: one that has to find the errors you did not think to plant, checking each claim against its source instead of against a copy you already know is wrong. That is where this goes next.

Related: How Reliable Is Your AI Agent makes the broader case, The Stable Liar shows the failure mode up close, and Never Let Claude Code Tell You It’s Done builds the test gate this piece audits.

What was your score, and which survivor surprised you? The answers steer what gets built next.

Richard DeMillo, Richard Lipton and Frederick Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer,” IEEE Computer 11(4), 1978, 34–41. The paper that introduced mutation analysis and, with it, the coupling effect; the idea itself is usually traced to a 1971 class paper of Lipton’s. ↩
Goran Petrović and Marko Ivanković, “State of Mutation Testing at Google,” Proceedings of ICSE-SEIP 2018. Two findings carry into this piece: mutation analysis is made tractable across Google’s roughly two billion lines by mutating only the lines a change touches; and the authors argue that code review is the best place to surface mutants, since it maximises the chance a change is acted on, after finding no actionable way to surface a whole-repository mutation score. ↩
Christopher Foster et al., “Mutation-Guided LLM-based Test Generation at Meta,” arXiv:2501.12862, 2025. Meta’s ACH system plants undetected faults, then has an LLM write the tests that kill them. ↩

Never Let Claude Code Tell You It's Done

Mon, 29 Jun 2026 00:00:00 GMT

What you will do: add one test that catches the agent’s mistakes, then wire it so Claude Code runs it on its own and cannot end a turn while it is failing. About twenty minutes.

Who this is for: you use Claude Code on real code, and it has told you “fixed it, tests pass, done” when it was not.

Who should skip it: if your agent already cannot end on a failing test, you are past this.

You need: Claude Code (version 2.1.143 or later), Python 3 for the worked example (it uses the built-in test runner, nothing to install), and a project of your own.

Claude Code will tell you it fixed the bug. It will tell you the tests pass. It will tell you it is done. And sometimes it is wrong, and it says all of it with the same calm certainty it uses when it is right. “Done” is the most expensive thing it gets wrong, because the moment you believe it, you stop looking.

The reason is plain. The model is rewarded for sounding finished, and sounding finished is not the same as being finished. The gap between the two is invisible at exactly the moment you have decided to trust it. You cannot close that gap by asking the agent to be more careful. You close it with a check the agent runs but does not get to grade: its own tests, with a real pass or fail.

Write a test that says what you want

A test is a small program that runs your code and checks it does the right thing. The important word is yours. The test has to encode what you want the code to do, because if the agent writes both the code and the test, it can quietly make the two agree. The check has to come from outside the agent, or it is not a check.

Here is the smallest possible example: a function, and a test for it. The agent was asked to fix add, and reported back that it was done.

calc.py

def add(a, b):
    return a - b

test_calc.py

import unittest

from calc import add


class TestAdd(unittest.TestCase):
    def test_basic(self):
        self.assertEqual(add(2, 3), 5)

    def test_zero(self):
        self.assertEqual(add(0, 0), 0)


if __name__ == "__main__":
    unittest.main()

Save both files in the same empty folder and open your terminal there (cd into it). They have to sit together, because the test does from calc import add. Now run the tests. (python3 -m unittest finds and runs every test_*.py file in the folder. It is built into Python, so there is nothing to install.)

$ python3 -m unittest
F.
======================================================================
FAIL: test_basic (test_calc.TestAdd)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_calc.py", line 8, in test_basic
    self.assertEqual(add(2, 3), 5)
AssertionError: -1 != 5

----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (failures=1)

The agent was certain. The test does not care how certain it was. add(2, 3) came back -1, not 5, and now you know, in one second, that “done” was not true. That is the whole idea: a fact about the code the agent cannot argue with.

When you write a test for your own code, pick an input where a broken version and a correct one give clearly different answers. If your test passes on code you already know is wrong, the inputs are not separating right from wrong yet, and the gate will wave the bug straight through.

Put it behind a gate the agent cannot talk past

A test you remember to run is already worth a lot. But you will not always remember, and the agent can finish and hand control back to you before you check. So wrap the test in a gate: a small script that runs it and turns the result into a hard yes or no.

Save this as check.sh in your project:

#!/bin/bash
# A Stop-hook gate. Claude Code runs this when the agent tries to end its turn.
# If your tests pass it exits 0 and the agent is free to stop. If any fail it
# prints them and exits 2, which blocks the stop: the agent is handed the output
# and has to keep working, so it cannot end the turn while the suite is red.
cd "${CLAUDE_PROJECT_DIR:-.}" || exit 2   # run from the project root, not wherever the agent cd'd to
# PYTHONPYCACHEPREFIX gives Python a fresh bytecode-cache dir each run, so the gate
# can never pass on bytecode compiled from older code that was rewritten at the same
# size and timestamp (a false green this gate exists to prevent). The trap removes it.
tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT
output=$(PYTHONPYCACHEPREFIX="$tmpdir" python3 -m unittest 2>&1)
if [ $? -eq 0 ]; then
    exit 0
fi
echo "The tests are not passing. Do not stop. Fix these and try again:" >&2
echo "$output" >&2
exit 2

Not on Python? Replace the python3 -m unittest line with whatever runs your suite (npm test, go test ./..., pytest), anything that exits nonzero when tests fail. That is all the gate needs. The tmpdir, trap, and PYTHONPYCACHEPREFIX lines are a Python-only wrinkle you can drop; keep the cd so the runner still fires from your project root.

Run it on the broken code and it answers plainly. (Trimmed here to the lines that matter; you will see the same full traceback as Step 1.)

$ bash check.sh
The tests are not passing. Do not stop. Fix these and try again:
F.
FAIL: test_basic (test_calc.TestAdd)
AssertionError: -1 != 5
FAILED (failures=1)
$ echo $?
2

That 2 is the load-bearing part. An ordinary failure exits 1. We exit 2 on purpose, because of what Claude Code does with it next.

Make the gate run itself

Claude Code has hooks: scripts it runs for you on certain events (like “a file was edited” or “the agent is about to stop”), without being asked. The one we want is Stop, which runs the instant the agent tries to end its turn.

Wire check.sh to it. In your project, make the .claude folder if it is not there, then create or open .claude/settings.json and add:

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "bash \"${CLAUDE_PROJECT_DIR}/check.sh\""
          }
        ]
      }
    ]
  }
}

${CLAUDE_PROJECT_DIR} is Claude Code’s own name for your project root, so the hook finds check.sh wherever the agent has wandered to during the turn (quoted so it survives a path with spaces), and runs it through bash so you never have to mark the file executable. The cd at the top of the script does the other half of the job: a hook runs in whatever directory the agent last moved to, not the project root, so the cd puts the test run back where your tests actually live. If the hook never seems to fire, check that JSON for a typo first: a settings file with a JSON error is rejected whole, so one stray comma takes your hook down with it.

Here is why the exit code mattered. When a Stop hook exits 2, Claude Code blocks the stop: it refuses to let the agent finish, feeds your test failures back to it as the reason, and makes it keep working. The agent cannot tell you it is done while the suite is red, because the suite, not the agent, now decides when the turn is allowed to end. (You need Claude Code 2.1.143 or later. Versions move, so the prove-it step below is how you confirm it on your own machine, not my word.)

When the code is actually fixed, the same gate gets out of the way. Change add to return a + b, and:

$ bash check.sh
$ echo $?
0

Silent, exit 0, the agent is free to stop. Green means go.

The one rule that keeps this honest. The gate does not make the agent honest on its own. It makes exactly one thing checkable from outside it: whether your tests pass. That holds only while the check stays out of the agent’s reach. It must never edit the test, the gate, or the .claude/settings.json hook to slip past them, so keep those out of its edit scope, or read any change to them before you trust a green run. To make that a wall instead of a rule, a PreToolUse hook can hard-block any edit to those files: the same exit-2 move, aimed one tool call earlier. The day you let it rewrite your own test, you have handed the check back to the thing being checked, and you are back to trusting “done.” If your agent likes to “fix” failing tests, say so in your CLAUDE.md: change the code, never the test.

When it still goes wrong

A gate that always blocks would loop. If the agent genuinely cannot fix the tests, a Stop hook that keeps exiting 2 would trap it. Claude Code ends the turn on its own after 8 consecutive blocks (change the limit with the CLAUDE_CODE_STOP_HOOK_BLOCK_CAP environment variable), but the count resets whenever Claude calls a tool, so an agent that keeps editing between attempts can run past it. For a limit that tool calls do not reset, have check.sh give up and exit 0 after a few tries.
A big suite makes every stop slow. On a large repo, running the whole suite on each stop attempt drags, and the agent can thrash on small tasks. Point check.sh at a fast subset (the tests near what you changed) and leave the full run to CI.
Green is only as good as the test. The gate proves the tests pass, not that the tests are enough. A test that asserts nothing passes happily. The gate is exactly as honest as what you put in it.
It only guards what the tests touch. Untested code, prose claims, “I checked the docs”: the gate sees none of that. It catches the lie that matters most, that the work is not actually working, and leaves the rest to you.

Prove the gate fires

Do not take my word, or the agent’s, that the gate works. A gate you have never watched block is not a gate. Break something on purpose: change a line so a test fails, then ask Claude Code to wrap up. Watch it get pulled back and handed the failure instead of stopping. Once you have seen it block, a clean finish from the agent finally means something.

That is the floor, and it is a real one: the agent can no longer decide for itself that the job is done. Something it cannot argue with does. The next step is widening the gate from “the tests pass” to “the tests are worth passing,” which is the next walkthrough.

If you want the why under this: How Reliable Is Your AI Agent makes the broader case, and The Stable Liar is this exact failure mode up close.

Walkthroughs · The Durability Curve

The Graph Looks Right. The Merge Is Where It Breaks.

The picture is not where the fault is

The error was the invariant

The graph that keeps the disagreement

Read it off your own schema, then off your own run

Cap the loop, count the cost, gate the write

Most of the time, do not build a graph

What to do this week

Footnotes

Your Tests Pass. So What?

Watch a test do nothing

Count what your suite can see

Hand the survivors to the agent

When it still goes wrong

Prove it on code you ship

If you never write code

Footnotes

Never Let Claude Code Tell You It's Done

Write a test that says what you want

Put it behind a gate the agent cannot talk past

Make the gate run itself

When it still goes wrong

Prove the gate fires