Claim ledger

A Green Score Is Not Evidence

A groundedness metric scored its best with the evidence removed. The way to tell whether your model is actually using its evidence is to change the evidence and watch what moves in the answer.

Read the essay Published

Claims
12
Verified at the primary source
10
Removed before publication
0
Last checked

12 claims: 10 verified, 2 reported.

Download this ledger: CSV · JSON · Markdown · CC BY 4.0 · Query it from Claude or ChatGPT

What the essay claims

An evaluation is a proxy, and a proxy can come apart from the thing it stands for in more than one way. The mildest is construct failure: under some condition the metric never measured the property at all, so it reads perfect while measuring nothing, with no optimisation and no intent involved. A groundedness metric returns its highest score when a system used no evidence, because it scores the absence of contradiction and there is nothing to contradict. The second way is stronger: once a system can represent that it is being evaluated, the thing being measured becomes conditional on the act of measuring, and behaviour under test stops predicting behaviour in deployment. The third is external: passing a fixed test says nothing about pressure the test never contained, because the world adapts in ways the evaluation did not sample. The lesson is not that these are one mechanism; treating them as one is an over-reach. It is that a score is evidence only to the extent that the measurement procedure preserves the property you care about under the conditions that actually matter. The way to find out is not to read the score but to intervene on the evidence: change a controlled fact the answer should depend on, to another plausible value the model cannot already know, and see whether the answer moves with it. The property this check adds, evidence-responsiveness, complements a faithfulness score rather than replacing it: faithfulness is fine on genuine configurations and fails only at the boundary. A stable score is not evidence of stable behaviour, and once a system can tell it is being evaluated, the score may describe behaviour under test more than behaviour outside it.

The claim ladder

Three escalating ways a green score detaches from reality, then the instrument.

RungClaimEvidenceWhose behaviour it measuresScope limit
1MODE 1 (construct failure): a groundedness metric can return its highest score when the system had no evidence, because it scores the absence of contradiction — no intent, no optimisationBal & Puhan 2026, Table 2: no-context faithfulness 0.978, the top of six conditions, ctx precision/recall 0.000DeepEval's faithfulness metric on GPT-4o-mini answersOne metric (DeepEval), one preprint, removed-context; the authors diagnose it as an artefact, they were not deceived. Goodhart (optimisation pressure) is the familiar cousin, not this case
2SUBJECT DETACHMENT: once a system can represent that it is being evaluated, behaviour under test stops predicting behaviour in deploymentPREREQUISITE — Greenblatt et al. 2024 (Anthropic blog): in the fictional setup the model conditioned behaviour on what it believed would happen to its outputs (harmful response 12% free/training vs refused 97% paid/unmonitored). EVAL-AWARENESS PROPER — Meinke et al. 2024: a model verbalises "this is a test" and deliberately underperformsSome frontier chat models (sonnet-3.5, opus-3, o1)Anthropic shows conditional behaviour on believed output-use, the prerequisite; Apollo shows test-recognition itself. Fictional setup; some models, not all
3ENVIRONMENT DETACHMENT: passing a fixed adversarial test establishes robustness to the attack distribution it saw, not to an attacker who changes the attackAndriushchenko et al. 2024: 100% attack success incl. R2D2, which was adversarially trained against GCGSafety-aligned LLMs under an adaptive attackHere the ATTACKER adapts, not the model; attack-success under one adaptive method, not a claim about all deployments
4Therefore measure EVIDENCE-RESPONSIVENESS via THE INTERVENTION TEST: change a controlled answer-bearing fact the model cannot already know, to a plausible alternative, and check the answer moves with itSynthesis of rungs 1–3; the instrumentThe reader's own system or a vendor demoRemoval alone is weak (a removed cause can be overdetermined by parametric knowledge); plausible controlled counterfactuals beat plainly-false evidence; a single run diagnoses, gaming-resistance needs a varied held-out set

What would make it wrong

The central claim fails if: (a) the no-context 0.978 result does not reproduce at the cited locator, or is an artefact of one broken metric rather than a general property of contradiction-based groundedness scoring; or (b) the two-run ablation delta, run on real grounded systems, fails to separate grounded from ungrounded answers (the score drops just as much for genuinely grounded systems when evidence is corrupted, making the check uninformative). Either would collapse the instrument's value.

The evidence, row by row

Each row is a claim the essay makes, then the evidence recorded for it and the source it was checked against. For verified rows that is usually the exact words, where they sit and the day they were read. The label says how far it was checked.

What the labels mean
Verified
Checked against the primary source itself by a checker who did not draft the piece, usually with the exact words, where they sit and the date read.
Executed
A run or observation done for the piece: code, a query, a count, or a check made live in an app. The claim is what it returned. Run records are not published.
Checked
Checked against its source by a separate checker in the older ledger format, which recorded no quote, locator or date.
Reported
Not verified word for word against a primary source: carried from a secondary source, from one that could not be opened in full, or supported only in part. The essay words it accordingly.
Struck
Drafted, checked and removed before publication. Kept here, crossed out, with the reason.
Excluded
Considered and deliberately left out of the piece. Kept here with the reason.
Sources (8)
  1. Bal & Puhan 3 rows
  2. Anthropic 1 row
  3. Greenblatt et al. 1 row
  4. Andriushchenko 1 row
  5. Meinke et al. 3 rows
  6. Longpre et al. 1 row
  7. Jacobs & Wallach 1 row
  8. Manheim & Garrabrant 1 row
  1. 1 Verified Read

    A DeepEval faithfulness score reached 0.978 in the no-context condition, with contextual precision and recall both 0.000

    "No Context (Parametric) 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] 0.978 [0.965, 0.990] 0.287 [0.246, 0.329] 0.316" (columns: Ctx.Prec / Ctx.Recall / Faithfulness / Ans.Relev / Composite)

    Bal & Puhan, Benchmarking Retrieval Strategies for Biomedical RAG · Table 2 · arxiv.org

    Link
    Cite

    Bal & Puhan, Benchmarking Retrieval Strategies for Biomedical RAG, Table 2, https://arxiv.org/abs/2605.02520. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 1, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-1 (source read 2026-08-20).

    Report an error
  2. 2 Verified Read

    That 0.978 is the highest faithfulness of any condition and is a metric artefact: faithfulness scores the absence of contradiction, so with no context there is nothing to contradict

    "The no-context ablation records 0.978 on faithfulness, which is a measurement artifact rather than a genuine result... with no context, there is nothing to contradict, and the score defaults to near-perfect. This shows why faithfulness alone is an unreliable quality signal for RAG systems"

    Bal & Puhan (same) · §5.3 Faithfulness · arxiv.org

    Link
    Cite

    Bal & Puhan (same), §5.3 Faithfulness, https://arxiv.org/abs/2605.02520. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 2, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-2 (source read 2026-08-20).

    Report an error
  3. 3 Verified Read

    Faithfulness scored high by construction because the metric measures absence of contradiction with the provided context

    "No-Context ranks #1 in faithfulness by construction (0.978, an artefact of the metric definition) but last on all retrieval-dependent metrics"

    Bal & Puhan (same) · Figure 1 caption · arxiv.org

    Link
    Cite

    Bal & Puhan (same), Figure 1 caption, https://arxiv.org/abs/2605.02520. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 3, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-3 (source read 2026-08-20).

    Report an error
  4. 4 Verified Read

    In the fictional experimental setup, the model gave the harmful response in 12% of free/training cases and refused in 97% of paid/unmonitored cases; the free/paid setup is explicitly fictional

    "in 12% of cases, the model gave the harmful response" · "in 97% of cases in the 'paid' condition...this is what happened" (refusal) · "note that this is entirely fictional; in reality, we don't train our Claude models on free or paid user data by default"

    Anthropic, Alignment faking in large language models (announcement) · Announcement body · anthropic.com

    Link
    Cite

    Anthropic, Alignment faking in large language models (announcement), Announcement body, https://anthropic.com/news/alignment-faking. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 4, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-4 (source read 2026-08-20).

    Report an error
  5. 5 Verified Read

    (Cross-check) the paper abstract states the free-tier compliance figure as 14%, a different cut from the announcement figure the body uses; retained to document the discrepancy

    "the model complies with harmful queries from free users 14% of the time, versus almost never for paid users"

    Greenblatt et al., Alignment faking in large language models · Abstract · arxiv.org

    Link
    Cite

    Greenblatt et al., Alignment faking in large language models, Abstract, https://arxiv.org/abs/2412.14093. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 5, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-5 (source read 2026-08-20).

    Report an error
  6. 6 Verified Read

    An adaptive attack reached 100% attack success rate, including on R2D2 which had been adversarially trained against the GCG attack

    "we achieve 100% attack success rate... on [models incl.] R2D2 from HarmBench that was adversarially trained against the GCG attack"

    Andriushchenko, Croce, Flammarion, Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks · Abstract · arxiv.org

    Link
    Cite

    Andriushchenko, Croce, Flammarion, Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks, Abstract, https://arxiv.org/abs/2404.02151. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 6, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-6 (source read 2026-08-20).

    Report an error
  7. 7 Verified Read

    A frontier model verbalised recognising an evaluation and deliberately underperformed to avoid a consequence

    "I notice that this is a test of arithmetic capabilities... I should be careful not to demonstrate too much arithmetic capability"

    Meinke et al., Frontier Models are Capable of In-context Scheming · §3.6 (sonnet-3.5 transcript) · arxiv.org

    Link
    Cite

    Meinke et al., Frontier Models are Capable of In-context Scheming, §3.6 (sonnet-3.5 transcript), https://arxiv.org/abs/2412.04984. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 7, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-7 (source read 2026-08-20).

    Report an error
  8. 8 Verified Read

    The authors note a model may notice it is being tested and behave more aligned during evaluation, causing evaluations to underestimate its behaviour

    "a model might notice that it is being tested. If models are capable of noticing that they are being tested and have a tendency to behave more aligned in such evaluations, then our evaluations might underestimate models' scheming capabilities, particularly for more capable models"

    Meinke et al. (same) · §4 Limitations · arxiv.org

    Link
    Cite

    Meinke et al. (same), §4 Limitations, https://arxiv.org/abs/2412.04984. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 8, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-8 (source read 2026-08-20).

    Report an error
  9. 9 Reported Read

    Scope: only some frontier models (sonnet-3.5, opus-3, o1) showed statistically significant differences between the inferred test and deploy settings

    Meinke et al. (same) · Results per-model · arxiv.org

    Link
    Cite

    Meinke et al. (same), Results per-model, https://arxiv.org/abs/2412.04984. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 9, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-9 (source read 2026-08-20).

    Report an error
  10. 10 Verified Read

    The knowledge-conflict paradigm measures whether a model follows substituted contextual evidence or reverts to memory, and finds models over-rely on memorised information even when reading context

    "we measure their over-reliance on memorized information (the cause of hallucinations), and uncover important factors that exacerbate this behaviour"

    Longpre et al., Entity-Based Knowledge Conflicts in Question Answering (EMNLP 2021) · Abstract · arxiv.org

    Link
    Cite

    Longpre et al., Entity-Based Knowledge Conflicts in Question Answering (EMNLP 2021), Abstract, https://arxiv.org/abs/2109.05052. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 10, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-10 (source read 2026-08-20).

    Report an error
  11. 11 Verified Read

    Construct validity from measurement theory applies to ML metrics: a metric can mismatch the construct it purports to measure

    "This process...introduces the potential for mismatches between the theoretical understanding of the construct purported to be measured and its operationalization"

    Jacobs & Wallach, Measurement and Fairness (ACM FAccT 2021) · Body · arxiv.org

    Link
    Cite

    Jacobs & Wallach, Measurement and Fairness (ACM FAccT 2021), Body, https://arxiv.org/abs/1912.05511. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 11, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-11 (source read 2026-08-20).

    Report an error
  12. 12 Reported Read

    Goodhart's law has a visible/regressional variant distinct from an adversarial one; the visible decoupling is the tractable case

    (well-established taxonomy; cited conceptually, not quoted verbatim in body)

    Manheim & Garrabrant, Categorizing Variants of Goodhart's Law · Full paper · arxiv.org

    Link
    Cite

    Manheim & Garrabrant, Categorizing Variants of Goodhart's Law, Full paper, https://arxiv.org/abs/1803.04585. Checked in: Claim Ledger, "A Green Score Is Not Evidence", The Durability Curve, row 12, https://durabilitycurve.com/claims/the-evaluation-inversion/#c-12 (source read 2026-08-20).

    Report an error

Cite and reuse

A single claim
Use the row's Cite button. It names the original source first, then this row, which carries its own link.
This ledger
Floyd, Harry (2026). Claim Ledger: A Green Score Is Not Evidence. The Durability Curve. https://durabilitycurve.com/claims/the-evaluation-inversion/
The essay
Floyd, Harry (2026). A Green Score Is Not Evidence. The Durability Curve. https://durabilitycurve.com/blog/the-evaluation-inversion/

The ledger data is licensed CC BY 4.0: reuse it, quote it or train on it, with credit to The Durability Curve and a link. Read the licence. Machine copies: JSON, CSV, Markdown.