◑ THE HUMAN LAYER OBS · 2026·08·23 OBS·12 x:1613 y:890 x:050 y:050 FACE·01

Confidently Wrong

The effort you're handing to AI was doing two hidden jobs. Skip them and you get faster, weaker, and blind to your own mistakes.

Harry Floyd 10 min read THE HUMAN LAYER Law II · Holds

The effort you’re handing to AI was doing two hidden jobs. Skip them and you get faster, weaker, and blind to your own mistakes.

In 2010, two of the most respected economists alive published a number that helped governments justify austerity across the Western world. Carmen Reinhart and Kenneth Rogoff found that once a country’s public debt passes 90% of GDP, growth doesn’t just slow. It turns negative, to an average real rate of −0.1%. The 90% line became a fact of political life. Paul Ryan’s budget cited it. The European Commission cited it. Governments invoked it as they cut into recessions.

Three years later a graduate student named Thomas Herndon asked to see the spreadsheet. He had been trying to reproduce the result for a class and couldn’t. When Reinhart and Rogoff sent him the actual Excel file, the −0.1% came apart in his hands along three separate faults. A formula that averaged the wrong range and silently dropped five countries. A set of high-debt, healthy-growth years left out of the sample. A weighting choice that let one bad year in New Zealand count as heavily as nineteen years in the United Kingdom. Correct all three and the threshold vanishes. Average growth above 90% debt was positive, at +2.2%. Higher debt still tracked somewhat slower growth, and no one had ever settled which way the causation ran. But the cliff, the part policy actually leaned on, was an artefact.1

A result this consequential sat unchecked for three years, in careful hands, until one student reached the same question by a different path and got a different answer. An independent recomputation was the kind of check that could catch the error. For three years, no one ran one.

What the difficulty was doing

AI can now lift the effort out of almost anything you find hard. It will write the memo, derive the number, draft the contract, debug the function, in a fraction of the time and often well. The useful question is which of those efforts you can hand off safely, and which you can’t.

The answer turns on something easy to miss. A hard task usually does two jobs beneath the obvious one. It builds you: the difficulty is a rep that makes you better at the task. And it checks you: the difficulty is a second, independent way of reaching the answer, the thing that catches you when your first way is wrong. Keep those two jobs apart and you know exactly what to protect when the effort disappears.

The difficulty that was checking your work

A single way of reaching an answer cannot check itself. Redo a sum the way you did it the first time and you reproduce the first mistake, faithfully. This is why your own eyes slide over your own typo on the second read, and why “measure twice” only helps if the second measurement uses a different ruler. To catch an error you need a route to the answer that would fail differently from the first. Herndon was that route. So is a test run against cases you worked by hand, or a rough estimate that ought to land in the same range.

Offloading to AI does its quiet damage at exactly that point. When you hand the hard part to a model, you keep its answer and drop the second route you would have taken. You were going to derive the number yourself. Now you don’t. The check didn’t fail. It was never run. And people do the rest of the damage on their own. Decades of research into how we use automation gives what happens next a blunt name, automation complacency: we stop cross-checking the output, even the experts, even after training, even when warned outright that the system is unreliable.2 Under real workload, attention drifts off the automated task, and the sampling that would have caught the error never happens. You end up fast, and confidently wrong, with nothing in place to catch it.

Before you accept an answer you didn’t work for, ask one thing. What would have caught this if it were wrong? If you can’t name anything, you are flying blind.

The tempting fix is to ask the model to check its own work. It can catch a careless slip, but it will not give you independence. The second pass shares the first one’s machinery: the same weights, the same training, often the same framing of the problem. On an error rooted in that shared machinery, another pass tends to reproduce the mistake rather than expose it. Even real independence leaks. In 1986, John Knight and Nancy Leveson had 27 programmers each write the same program from one specification, then ran every version against a million inputs. The versions were supposed to fail independently. They didn’t. They made the same mistakes on the same hard inputs, far more often than chance allows.3 Separate people, working alone, drift onto the same errors.

A second step earns its keep only when it could fail in a different way from the first. For a number, that is a second derivation from different inputs, or a rough estimate that ought to agree. For a claim, it is the primary source, rather than a more confident summary of it. For code, it is running the thing against reality instead of reading it again. A second step that shares the first’s blind spot is decoration.

The difficulty that was building you

In 1997, an American Airlines training captain named Warren VanderBurgh gave a talk about what modern cockpits were doing to pilots. He called them children of the magenta line, after the course the flight computer paints across the navigation display. His pilots had become superb managers of automation and worse at flying. They could program the box beautifully and struggled to take the aeroplane when the box gave up. By the airline’s own reckoning, most of the automation-related trouble his team studied came back to that.4

Twelve years later, Air France 447 came out of the night over the Atlantic. The pitot tubes iced, the airspeed readings went unreliable, and the autopilot handed control to a crew that almost never flew by hand. One pilot held the nose up. The wing stopped flying. Through three and a half minutes of descent, as the stall warning sounded and cut out and sounded again, the crew never recognised the stall. The official report named several causes, among them a breakdown between the two pilots and the absence of any training in flying the aeroplane by hand, at that altitude, when the automation quit. The skill that might have caught it had gone unused until the one night it was needed.5

This is the older half of the story, and the learning research has a precise name for what was lost. Robert and Elizabeth Bjork call the effort that builds durable skill desirable difficulty: spacing practice out, mixing problem types, generating an answer before you are shown it, testing yourself instead of rereading.6 These feel worse in the moment. They make you slower and more error-prone today, and they build the underlying strength that makes a skill stick and transfer. One condition matters more than the rest: a difficulty is only desirable if you can actually meet it. Confusion from a bad explanation, struggle with no traction, effort you cannot yet surmount, none of that builds anything. That is what keeps the argument honest. Not every hard thing is worth keeping.

Lisanne Bainbridge saw the shape of this in 1983, in a paper called “Ironies of Automation.”7 The designer who removes the operator, she wrote, still leaves the operator the tasks too hard to automate, and less practised at the skills those tasks demand. Offload your reps and you become more productive and less capable at the same time, and you will not feel the second half happening. Productivity is loud. Skill decay is silent. And the person who has not built the skill yet has the most to lose: skip the reps at the start, and you never become someone who could catch the machine at all.

When the check you rely on is your own judgement, these two jobs turn out to be the same one. That check stays independent only while you can still reach the answer yourself. Offload the reps for long enough and your sense of the right answer quietly retrains on the machine’s output, until the second opinion in your head is only the machine’s first opinion, learned by heart. Knight and Leveson watched separate programmers drift onto the same mistakes. Lean on one model long enough and you become another correlated version of it.

The new cost of checking

There is a reason this bites harder now than it did in Bainbridge’s day. The old bargain of automation assumed that checking is cheaper than doing, and usually it is. Verifying a finished Sudoku takes a moment; solving it does not. AI has moved a great deal of work to the other side of that gap. When a model produces fluent, plausible output whose errors are subtle and load-bearing, the careful check can cost more than the generation did. Checking stays cheap when you have a cheap way to test the answer, a passing test, a green type-checker, a production system that stays up. It gets expensive exactly where AI is most tempting, on judgement work with no quick way to know. Budget for the check as part of the cost of the work, or you have not saved the time you think you have.

Making the call

You ask a model for the size of a market, for a board deck. It hands back a confident figure with a tidy build-up, and the effort you skipped, assembling that number yourself, was your only independent estimate of it. Paste it in and nothing stands that could contradict it. The move is one cheap second estimate from a different direction: top-down from a population and a spend rate, when the model worked bottom-up from company counts. Land in the same range and you have something real. Come out a factor of five apart and you just caught the number that would have embarrassed you in the room.

You have a model read a contract and it flags nothing. The independent route reads it differently: the specific clause checked against the actual regulation, or the colleague who got burned by this exact term last year. A second pass by the same model shares its blind spot, and will reassure you at the moment you most need to worry.

You accept a function the model wrote, and it looks right. Reading it again is your first route walked twice. Running it against cases you worked out by hand, especially the ugly boundary ones, is a route that fails differently, which is why “it compiles” and “the tests pass” are worth more than another careful read.

The rule cuts the other way just as often. A tedious afternoon reconciling two exports by hand, reformatting data, translating boilerplate, that difficulty builds nothing and checks nothing. It is effort along the one road you were always going to walk. Hand it over without a flicker of guilt. And once in a while an easy thing deserves protecting: if the only reason you would ever catch a bad assumption is that you still do the simple monthly reconciliation yourself, keep doing it, precisely because it is the cheap check that catches the expensive error.

Keep the difficulty that builds you or checks you. Shed the rest, and shed it gladly. Two things are worth guarding as machines take the effort out of your work: the reps that keep you able to notice, and the second, separate route that catches you when you are wrong. Automate everything else. Just never automate away your last independent way of knowing you are right, and never stop doing the work that would let you feel it when you are not.

Related reading: The Difficulty You’re Escaping Was Making You: the human half of this, what the effort was quietly making of you. A Green Score Is Not Evidence: when the check itself is the thing being gamed.

Footnotes

  1. Reinhart & Rogoff, Growth in a Time of Debt (2010); the recomputation is Herndon, Ash & Pollin, Does High Public Debt Consistently Stifle Economic Growth?, PERI Working Paper 322 (2013). Three faults: a coding error, selective data exclusion, and unconventional weighting. Corrected average growth above 90% debt was +2.2%. ↩

  2. Parasuraman & Manzey, Complacency and Bias in Human Use of Automation, Human Factors 52(3), 2010. Complacency appears in experts as well as novices and is not eliminated by training or warnings. ↩

  3. Knight & Leveson, An Experimental Evaluation of the Assumption of Independence in Multiversion Programming, IEEE TSE SE-12(1), 1986. 27 programmers, one specification, ~1,000,000 inputs; the independence of failures was rejected at the 99% confidence level. ↩

  4. Warren VanderBurgh, Children of the Magenta Line, American Airlines training (1997); AirFacts retrospective. ↩

  5. BEA final report on Air France 447 (2012); among the cited causes, a breakdown in crew coordination and the absence of training in manual handling at high altitude. Summary via IEEE Spectrum. ↩

  6. Bjork & Bjork, Making Things Hard on Yourself, But in a Good Way (2011). A difficulty is desirable only if the learner can meet it; confusion and untraversable struggle build nothing. ↩

  7. Bainbridge, Ironies of Automation, Automatica 19(6), 1983. ↩