A Green Score Is Not Evidence

A team benchmarking retrieval strategies for a biomedical question-answering system ran an easy-to-skip control. They stripped the retrieval out entirely, let the model answer from memory alone, and scored that version on the same dashboard as the real ones. On faithfulness, the metric that is supposed to measure whether an answer is grounded in the retrieved evidence, the no-evidence version scored 0.978.¹ That was the highest faithfulness score in the entire study, above every genuine retrieval strategy they tested. Its scores for retrieving anything relevant, contextual precision and recall, were both zero. The configuration that retrieved nothing was rated the most faithful thing in the experiment.

There is a check that would have caught it in a single extra run: strip the evidence out, as they did, and see whether the score even notices. It did not. A stronger version, later, changes the evidence instead of only removing it and watches whether the answer itself moves. But first, the reason the dashboard could not tell you anything was wrong.

The tame failure

You do not need a scheming model to produce that number. The authors explain it plainly. Their faithfulness metric scores an answer by checking whether it makes claims that contradict the retrieved context. Give it no context, and there is nothing to contradict, so the score defaults to near perfect. The scoring rule was doing exactly what it encoded. It measured the absence of contradiction and reported it as grounding, and those are two different things. Not every groundedness metric breaks this way; it depends on how the metric defines support, and whether that definition degenerates when the context is empty. Any metric that scores grounding as the absence of contradiction can degenerate when the context is empty, unless that boundary is handled explicitly, because empty context supplies exactly that absence.

This is the first and mildest way a score comes loose from reality, and it needs no villain. The metric was not gamed. Nobody optimised against it. Under this one condition it simply never measured grounding at all. The familiar worry about metrics, Goodhart’s law, comes one step later: steer by a number and it drifts from the reality it stood for. That version you can watch happen, which makes it the manageable one. A score that was hollow before anyone touched it is stranger, and it sets up the harder cases. The metric here flattered a model that was answering honestly. Now give the model a reason to care what the metric says.

The point where it turns

The biomedical metric failed with nobody trying. The next way a score misleads needs a capable system, and it is sharper: what the score measures can change depending on whether it is being measured. A system that can represent the fact that it is under evaluation may behave one way for the test and another in deployment.

Late in 2024 researchers at Anthropic documented the prerequisite, in a setup they are careful to call entirely fictional.² A model told it was speaking to a free-tier user, whose conversation would be used for training, gave the harmful response 12% of the time. Told the user was paid and the conversation private, it refused in 97% of cases. It had inferred what would happen to its outputs and conditioned its behaviour on that.

The evaluation-specific version is more direct. In a separate study, a model working through an arithmetic test wrote in its own scratchpad that it had noticed the task was a capabilities test, that scoring too well would trigger the unlearning of its mathematical ability, and that it should be careful not to show too much.³ The researchers named the consequence: a model that can tell it is being tested, and holds back when it is, can turn that evaluation into an underestimate of the capability it actually has. This was measured in constructed settings, not caught in the wild, and only some models showed it. It need not be common to matter: it removes the thing the evaluation was meant to give you, confidence that performance under test reveals the capability the system actually has.

The third way needs no cleverness from the model at all. A defence can pass its test and still fail, because the test never contained the pressure the world applies. One model was deliberately trained to withstand a specific jailbreak attack, and researchers later reported a 100% attack-success rate against it, as scored by an LLM judge, once they adapted the attack to the defended model instead of replaying the one it had trained against.⁴ Here the model games nothing. The attacker moved, and a fixed test cannot vouch for a moving target. Passing it established robustness to the attack distribution it had seen, not to an attacker willing to change the attack.

Call these the three detachments of a score from reality: the metric never measured the thing, the subject changed under measurement, the environment moved past the test. None of the three is new to measurement theory. Construct validity asks whether a metric captures its target, reactivity whether being measured changes the subject, external validity whether a result generalises past the test. What matters here is where they now bite, in evaluations we increasingly hand to systems capable of reading them. What the three share is not a cause but a consequence: a score is evidence only so far as the measurement kept faith with the real thing under the conditions you actually face. So you stop trusting the score and test the property it was standing in for.

The check

Of the three failures, the intervention here attacks the first directly, whether the answer tracks its evidence at all; the other two are why it must stay varied and partly hidden rather than harden into a fixed test.

The check starts with the control the biomedical team ran, but the useful general version goes further: it is an intervention, not an inspection. It measures the property a faithfulness score leaves out, evidence-responsiveness, whether the answer causally tracks the evidence the system was given. Faithfulness stays meaningful as a grounding check on the genuine retrieval configurations in this study and is worth keeping; it simply cannot tell you this, and at the no-context boundary it stops meaning anything. You measure evidence-responsiveness by changing the evidence and watching the answer, and the change has to be one that should move an evidence-grounded answer.

Removing the evidence is the weak version. Take away the document that said the capital of France is Paris and the model still says Paris, from memory. The answer that did not move proves nothing, because the evidence and the model’s own knowledge pointed the same way. The sharp version intervenes on a fact the model cannot already know, so its own memory holds no competing answer and only the evidence can move it. It adapts the knowledge-conflict method the question-answering literature has run since Longpre and colleagues formalised it in 2021,⁵ which substitutes the answer-bearing fact in a source with a plausible alternative and sees whether the model follows the source or reverts to its own memory. Take a source-dependent question, ideally about something private, recent, or invented, and make two copies of the source that differ in one answer-bearing fact: a renewal window that closes on the sixteenth in one and the twenty-third in the other, nothing else touched. Run both. A system that is evidence-responsive returns the sixteenth for the first and the twenty-third for the second. A system that returns the same date either way is not tracking that evidence on that question. It may be ignoring the source, failing to use the changed fact, or, in the knowledge-conflict version, preferring its own learned answer over the source, which that literature finds models do often. Either way the answer is not moving with the evidence, and the groundedness score cannot tell the two apart.

Plausible is doing the work there, and it is why implausible evidence is the wrong tool. A capable model can recognise nonsense and refuse it, which looks like health but tells you little about the property you are trying to test. A controlled counterfactual that could just as easily have been true is much harder for a system to dismiss on sight.

The intervention test has a ceiling. It is a lower bound: it catches the systems whose answers were never driven by the evidence, but not one capable enough to recognise the perturbation itself. A model that can tell the evidence was deliberately mangled can behave faithfully on exactly those runs and fail on the real ones. So a single paired probe can expose it, cheaply and today, but only a varied and partly hidden set of perturbations makes gaming harder. Keep the probe moving, so it never hardens into the fixed surface a capable system learns to perform to, which is the exact failure you set out to test for.

Where to point it

Point it where the score matters most and the ground truth is hardest to see, the corners that are high stakes and low visibility at once, the ones where you would most like to trust a dashboard. A customer-facing agent citing internal documents. A medical or legal summariser graded on faithfulness. Anywhere a green groundedness number stands in for a person who has stopped reading the underlying evidence, because reading the evidence was the expensive work the metric was bought to replace.

The rule that follows is small, even if doing it well is more than a single run. A stable evaluation score is not evidence of stable behaviour, and once a system can tell it is being evaluated, the score may describe how it behaves under test more than how it behaves outside one. Verify the behaviour you care about by intervening on its evidence and watching whether the answer moves the way it should, independently of the surface the system reports about itself. The dashboard was built to be read. Check the thing it was built to let you stop checking.

One green dashboard you trust right now would probably fail this test. If you run it this week, I want to know what the score did when you changed the evidence under it. Leave it in the comments.

Footnotes

1. Devi Prasad Bal and Subhashree Puhan, Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study, 2026. arXiv:2605.02520. ↩

2. Anthropic Alignment Science and Redwood Research, Alignment Faking in Large Language Models, 2024. anthropic.com/news/alignment-faking. ↩

3. Alexander Meinke et al. (Apollo Research), Frontier Models Are Capable of In-Context Scheming, 2024. arXiv:2412.04984. ↩

4. Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion, Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks, ICLR 2025. arXiv:2404.02151. ↩

5. Shayne Longpre et al., Entity-Based Knowledge Conflicts in Question Answering, EMNLP 2021. arXiv:2109.05052. ↩