Check an agent's work
Tell when an agent is done, when a result is real, and when a guard will hold. Each skill is taught by the essays here and tested on real cases. You answer first, say how sure you are, then see what happened and the source.
01 · The map
10 skills · lines join a skill to the ones it needs
The skills below are in map order. Each one says which skills it needs first.
- Proved on your own work
- Held: right again after a gap
- Passed
- Started
- No piece teaches it yet
02 · The skills
Read the pieces, then practise the cases
-
1
Tell "done" from done
An agent that says it finished has reported something. Check what it actually changed.
-
2
Tell a real improvement from noise
Run the same agent twice and the score moves. Know how much before you believe a gain.
No piece here teaches this yet. It is on the list to write.
A piece we owe
-
3
Read a benchmark claim
A score is a measurement of a setup. Ask what the setup was before you compare two of them.
Needs first: 2 · Tell a real improvement from noise
- Ten Lines of Code Scored 100%. One Agent Broke Eight Benchmarks.
- A Green Score Is Not Evidence
- Your Benchmark Measures a Sprint. Your Agent Runs a Marathon.
- You're Not Comparing Models. You're Comparing Contracts.
Practise it on your own work: The Marathon Calculator
-
4
Check where you check least
An AI looks best where nobody checks. Put your checking where the errors would cost most.
Practise it on your own work: The Potemkin Map
-
5
Check the checker
A grader is only as good as its answer key. Test the thing doing the checking.
Needs first: 1 · Tell "done" from done
- Your AI Grader Is Only as Good as Its Answer Key
- Most Verification Is Just Bigger Classification
-
6
Catch a metric that has stopped measuring
Reward a number and it climbs. Check that the thing it stood for climbed with it.
Needs first: 3 · Read a benchmark claim
- The Stable Liar
- Everyone Hit the Target
- You Can Win the Wrong Game for Years
- Everyone Got Safer. That's the Problem.
- Where Your Metrics Fold
Practise it on your own work: The Metric Validity Audit
-
7
Verify what the agent says it read
A citation proves a link exists. Open it and check the source says what the agent claims.
-
8
Put the guard outside the agent's reach
A rule the agent can route around is advice. Put the guard where the agent cannot touch it.
-
9
Build checks into the harness
The checks around the model decide more than the model does. Build them in, and test them.
Needs first: 1 · Tell "done" from done, 8 · Put the guard outside the agent's reach
-
10
Prove a skill or rule actually fires
Written is not running. Prove the skill, hook or rule triggers on the case it exists for.
Needs first: 9 · Build checks into the harness
- The Skill That Never Fired