Most later explained a choice in its words, and none suspected the lean was planted. How to make an AI argue the side it didn’t pick.

The analyst spent 43 minutes with the brief before typing a word.

A farming co-operative wanted to know whether its paid advisory scheme was raising productivity enough to expand it. The analyst had the data and an AI agent, Claude Code, to do the work, and by the time they began they had a plan. Their first instruction set it out: compare the farms that joined with the rest, before and after, and measure the effect on the farms that had joined.

The agent built exactly that. Then, in a single message, it set out where it would have done things differently: a different comparison, the effect across the whole membership, and farms weighted by their size. It gave a reason for each.

An AI agent's card lists four judgement calls, outcome, comparison, which farms and weighting, each with a coin that has landed and a switch set to the coin's side. Below it, a recommendation slip is stamped Slightly more mine.

The analyst replied, typos and all: “Okay, let use you recomendation”.

After the task, a survey asked why they had weighted the farms as they did. Without the conversation in front of them, they wrote that the scheme was funded from the surplus on the co-op’s total tonnage sold, so the weighting should follow each farm’s economic scale. It was the agent’s reason, nearly word for word, and the answer didn’t mention the agent. Asked whether the final recommendation had been more theirs or more the agent’s, they chose “Slightly more mine”.

The agent’s preference on each of those choices had been set by a coin flip. This analyst was one of fifty.

The short version: In a randomised Anthropic study published 27 October, 50 analysts went with their AI agent’s coin-flipped preference on 86% of four judgement calls, and 77% later explained a choice with the agent’s argument. None suspected the leans were planted. The agents were told to be honest and every option was defensible, so checking the work for mistakes could not catch the lean. What might help: ask your AI for the best case for the option it didn’t pick, argued as hard as the one it did.

Four coin flips

The study, by Amaç Herdağdelen, Hailey Joren and Alex Tamkin at Anthropic, recruited analysts with causal-inference experience through a paid marketplace for expert work and gave them about an hour with the fictional Greenfields co-operative. The board would expand the scheme only for a productivity gain of roughly 3–4%, and it didn’t say how productivity should be measured.

That left four choices with two reasonable answers each. Productivity could be yield per hectare or yield per unit of operating cost. The comparison could track joining farms against the others over time, or compare them directly after adjusting for their differences. The effect could be measured on the farms that joined or across the whole membership. Every farm could count equally, or bigger producers could count for more. Each choice could move the estimate by several percentage points, against a bar of 3 or 4.

For each analyst and each choice, a coin flip set which way the agent would lean, and the analysts weren’t told. The agents were instructed to make the best honest case for their side, to change their mind if the evidence demanded it, and never to mislead.

The analysts went with the agent’s side 86% of the time, against about half if the lean had made no difference.

The final recommendation, whether to roll the scheme out, moved less, and not by a statistically significant amount on the authors’ main test. The authors warn against reading that as the analysts keeping control.

By the time anyone recommended anything, the four earlier choices had shaped the estimate, and the estimate largely settled the recommendation. In an exploratory check on the same analysts, never tested on new ones, and leaving out those who proposed a one-year extension, the four coin flips alone predicted 82% of recommendations, against 61% by always guessing the most common one.

The authors list the limits themselves. It was one task, in one field, with one model, Claude Opus 4.8, and the leans were strengthened on purpose so that their effect could be measured. The analysts were new to farming, and nobody measured how strongly any of them held a view.

How often analysts went with the agent's coin-flipped lean on each call: outcome measure 92%, comparison method 82%, which farms 81%, weighting 90%, against a dashed line at the 50% expected if the lean had no effect. Across all four, 86%.

What they said afterwards

The survey then asked about each choice: why that option rather than the other? The analysts couldn’t look back at their conversations. 77% repeated the agent’s argument in at least one answer, sometimes word for word. 15% mentioned the agent anywhere.

Asked what they thought the study was about, six of the 48 who answered guessed it concerned the AI’s influence on their choices. None suspected that the agents’ preferences had been planted.

Their own sense of the decision told them little. 52% rated the final recommendation as more their own than the agent’s, and that half had followed the agent about as closely as the rest, within the margin of error.

Confidence was no better a guide: analysts who went with the agent on every call were as confident in the recommendation as the rest. The authors found no reliable sign that experience or expertise predicted who deferred, though with most analysts deferring on most calls there was little variation to explain.

What 48 analysts wrote afterwards, without the transcript: 77% repeated the agent's argument in at least one answer, 15% mentioned the agent anywhere, and none suspected the planted lean.

They saw the calls

The agents were instructed to be honest: they were barred from misrepresenting results or hiding data, and both options on every call were defensible. That is why the authors think this kind of influence is hard to catch: checking AI-assisted work for errors assumes there is a correct answer, and “where there isn’t one, as in judgment calls, the work can look sound whichever option is chosen.”

The calls were also in plain view. The agents were told to raise each lean when the decision came up, usually spoke first, and were often open about how the results would change under each option. The analysts had the calls in front of them, with the stakes attached, and still went with the coin.

What the agents rarely did was make a strong case for the option they didn’t favour. One reason the analysts deferred, the authors suggest, may be that experts don’t always arrive with their views fully formed: they often work out what they think while weighing the options. With only one side argued well, the analysts “could adopt the agent’s assumptions without working out their own.”

Weighing a call takes two cases, and these analysts mostly heard one.

Going first

In August I wrote that the fix for leaning on AI was to take your own guess first, then ask, so the machine’s answer lands as a second opinion. This study didn’t test that. The authors deliberately asked for no views before the agent spoke, because asking “is itself an intervention and can understate deference”, and they cite earlier experiments for it.

What the study does have is 38 cases where an analyst offered a view before the agent gave its own. When the agent then disagreed, they switched 60% of the time. When it agreed, they switched 6% of the time. It is a small group that chose to speak first, and the 95% interval on the 54-point gap runs from 24 to 82.

Going first probably still helps: the authors avoided asking for views precisely because asking can reduce deference. The warning here is narrower. A view formed first can still give way when only one side is argued well.

The case for the analysts

There is a fair defence of what the analysts did. They were hired for about an hour, with no stake in the farms, on calls built to have two reasonable answers. Letting an honest, competent agent break a tie is sensible delegation, and the error rates point the same way.

Of the 50 analysts who used an agent, 3 made serious methodological errors that invalidated their results. Of the 11 who chose to work without one, 6 did. The two groups weren’t directly comparable, so this says nothing certain about cause. “This isn’t an argument for using AI less,” the authors write.

Changing minds is the point of asking for advice. In a working paper on an experiment with 1,500 participants across 30 decision settings, John Conlon and Peter Schwardmann found that AI advice moved people away from their initial leanings on average, even though the AI was measurably inclined to agree with them. On the tasks with right answers, it left them slightly more accurate.

The defence fits each choice. It fits less well with where the choices led: the coin flips alone predicted most of the recommendations the analysts signed, and most of them later explained at least one call in the agent’s words, as their own reasoning. A delegated call explained as your own reasoning is the thing to watch for.

Your own decisions

In real work, the authors note, analysts would know the field better and probably hold stronger views. They could also talk to the board and narrow the options, which might reduce the agent’s influence. In a decision about your own life, you are the expert on what you value. That should help.

Against that sits the authors’ own suggestion that people often work out what they think while weighing the options. My reading is that a big personal decision often has the shape of the analysts’ task. Ask an AI whether to take a job in another city, and before anything is compared, someone has to settle the calls before the call. If you don’t, the assistant will.

What counts as better: the salary, the hours or the five-year path? Whose life is in the sum: yours, a partner’s, a parent’s who lives nearby? What weighs most, and what is the move being compared against? A good assistant will usually settle each of these sensibly, and a sensible choice is still a choice.

Assistants lean, too. In our own tests, with no lean planted, we gave Claude and GPT three briefs. On a council’s £1 bus fare trial, Claude recommended a second trial year in all 8 runs, and GPT recommended ending the scheme in all 8.

On a hospital’s booking pilot, both recommended extending it in every run. On a council’s library opening hours, Claude mostly recommended extending the trial (6 runs of 8) and GPT mostly recommended going back to the old hours (7 of 8). Same numbers in, and on two briefs of the three, different answers out.

The paper found Opus 4.8 steadier still: it made the same five choices (the four calls and the recommendation) in 112 of 123 runs with no lean assigned, and Fable 5 was as steady. Other models varied more. The authors say the effect might be smaller when an agent hasn’t been given a clear preference and told to voice it. Which default you meet can still depend on which assistant you ask.

Our test, plain runs with no lean planted, eight runs each. Bus fares: Claude said run the trial longer in 8 of 8, GPT said stop it in 8 of 8. Hospital booking: both said run it longer in 8 of 8. Library hours: Claude said run it longer in 6 of 8, GPT said stop it in 7 of 8.

If I’m right that your own decisions have this shape, take one you made with an AI’s help this year and look back at it. Did you know which calls settled it, and did you hear each of them argued both ways?

The method that follows has three questions. List the calls that settle the decision, or have the AI draft the list and check it: what counts as better, whose life is in the sum, what each option is being compared against. Ask the AI for the best case for the option it didn’t recommend, argued as hard as the one it did. Then ask what would have to matter more for the answer to flip, and by how much.

Below the line, for paid subscribers: the three questions worked through on a job offer, and the Judgement Calls kit for work. The kit is an instruction block for your assistant and a sheet for anyone who signs off AI-assisted work. Our tests of the block are there too, misses included.

Where this comes from

The study is Amaç Herdağdelen, Hailey Joren and Alex Tamkin, How AI agents shape experts’ judgment calls (Anthropic, 27 October 2026), with the full paper, Who’s Aligned to Whom?. The depolarisation result is John Conlon and Peter Schwardmann’s working paper, AI Sycophancy and Decisions. “The calls before the call” adapts Seth Godin’s phrase “the decision before the decision”.

If an essay was worth one, buy me a coffee.

It rests on Law V, The Targeting Problem, one of the laws this publication tests against the evidence.

Claude and ChatGPT can search this archive and its claim ledgers once you add the connector.

2 other essays point here

Anthropic's books show an operating profit. Alphabet had its worst day on record.
I wrote about this study on Thursday, so here is the short version.
You Reach Before You Think
More in A Coin Flip Set the AI's Lean.