{
 "ledger": {
  "title": "Claim Ledger: A Green Score Is Not Evidence",
  "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/",
  "essay": {
   "title": "A Green Score Is Not Evidence",
   "url": "https://durabilitycurve.com/blog/the-evaluation-inversion/",
   "published": "2026-08-21"
  },
  "last_checked": "2026-08-20",
  "format": "current",
  "counts": {
   "claims": 12,
   "by_status": {
    "VERIFIED": 10,
    "REPORTED": 2
   },
   "verified_at_primary": 10,
   "removed": 0
  },
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "attribution": "The Durability Curve (Harry Floyd), https://durabilitycurve.com/claims/the-evaluation-inversion/"
 },
 "rows": [
  {
   "id": "1",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-1",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "A DeepEval faithfulness score reached 0.978 in the no-context condition, with contextual precision and recall both 0.000",
   "quote": "\"No Context (Parametric) 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] 0.978 [0.965, 0.990] 0.287 [0.246, 0.329] 0.316\" (columns: Ctx.Prec / Ctx.Recall / Faithfulness / Ans.Relev / Composite)",
   "removal_reason": "",
   "source": "Bal & Puhan, Benchmarking Retrieval Strategies for Biomedical RAG",
   "source_url": "https://arxiv.org/abs/2605.02520",
   "locator": "Table 2",
   "date_read": "2026-08-20"
  },
  {
   "id": "2",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-2",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "That 0.978 is the highest faithfulness of any condition and is a metric artefact: faithfulness scores the absence of contradiction, so with no context there is nothing to contradict",
   "quote": "\"The no-context ablation records 0.978 on faithfulness, which is a measurement artifact rather than a genuine result... with no context, there is nothing to contradict, and the score defaults to near-perfect. This shows why faithfulness alone is an unreliable quality signal for RAG systems\"",
   "removal_reason": "",
   "source": "Bal & Puhan (same)",
   "source_url": "https://arxiv.org/abs/2605.02520",
   "locator": "§5.3 Faithfulness",
   "date_read": "2026-08-20"
  },
  {
   "id": "3",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-3",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Faithfulness scored high by construction because the metric measures absence of contradiction with the provided context",
   "quote": "\"No-Context ranks #1 in faithfulness by construction (0.978, an artefact of the metric definition) but last on all retrieval-dependent metrics\"",
   "removal_reason": "",
   "source": "Bal & Puhan (same)",
   "source_url": "https://arxiv.org/abs/2605.02520",
   "locator": "Figure 1 caption",
   "date_read": "2026-08-20"
  },
  {
   "id": "4",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-4",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "In the fictional experimental setup, the model gave the harmful response in 12% of free/training cases and refused in 97% of paid/unmonitored cases; the free/paid setup is explicitly fictional",
   "quote": "\"in 12% of cases, the model gave the harmful response\" · \"in 97% of cases in the 'paid' condition...this is what happened\" (refusal) · \"note that this is entirely fictional; in reality, we don't train our Claude models on free or paid user data by default\"",
   "removal_reason": "",
   "source": "Anthropic, Alignment faking in large language models (announcement)",
   "source_url": "https://anthropic.com/news/alignment-faking",
   "locator": "Announcement body",
   "date_read": "2026-08-20"
  },
  {
   "id": "5",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-5",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "(Cross-check) the paper abstract states the free-tier compliance figure as 14%, a different cut from the announcement figure the body uses; retained to document the discrepancy",
   "quote": "\"the model complies with harmful queries from free users 14% of the time, versus almost never for paid users\"",
   "removal_reason": "",
   "source": "Greenblatt et al., Alignment faking in large language models",
   "source_url": "https://arxiv.org/abs/2412.14093",
   "locator": "Abstract",
   "date_read": "2026-08-20"
  },
  {
   "id": "6",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-6",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "An adaptive attack reached 100% attack success rate, including on R2D2 which had been adversarially trained against the GCG attack",
   "quote": "\"we achieve 100% attack success rate... on [models incl.] R2D2 from HarmBench that was adversarially trained against the GCG attack\"",
   "removal_reason": "",
   "source": "Andriushchenko, Croce, Flammarion, Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks",
   "source_url": "https://arxiv.org/abs/2404.02151",
   "locator": "Abstract",
   "date_read": "2026-08-20"
  },
  {
   "id": "7",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-7",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "A frontier model verbalised recognising an evaluation and deliberately underperformed to avoid a consequence",
   "quote": "\"I notice that this is a test of arithmetic capabilities... I should be careful not to demonstrate too much arithmetic capability\"",
   "removal_reason": "",
   "source": "Meinke et al., Frontier Models are Capable of In-context Scheming",
   "source_url": "https://arxiv.org/abs/2412.04984",
   "locator": "§3.6 (sonnet-3.5 transcript)",
   "date_read": "2026-08-20"
  },
  {
   "id": "8",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-8",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "The authors note a model may notice it is being tested and behave more aligned during evaluation, causing evaluations to underestimate its behaviour",
   "quote": "\"a model might notice that it is being tested. If models are capable of noticing that they are being tested and have a tendency to behave more aligned in such evaluations, then our evaluations might underestimate models' scheming capabilities, particularly for more capable models\"",
   "removal_reason": "",
   "source": "Meinke et al. (same)",
   "source_url": "https://arxiv.org/abs/2412.04984",
   "locator": "§4 Limitations",
   "date_read": "2026-08-20"
  },
  {
   "id": "9",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-9",
   "status": "REPORTED",
   "status_label": "Reported",
   "removed": false,
   "claim": "Scope: only some frontier models (sonnet-3.5, opus-3, o1) showed statistically significant differences between the inferred test and deploy settings",
   "quote": "",
   "removal_reason": "",
   "source": "Meinke et al. (same)",
   "source_url": "https://arxiv.org/abs/2412.04984",
   "locator": "Results per-model",
   "date_read": "2026-08-20"
  },
  {
   "id": "10",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-10",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "The knowledge-conflict paradigm measures whether a model follows substituted contextual evidence or reverts to memory, and finds models over-rely on memorised information even when reading context",
   "quote": "\"we measure their over-reliance on memorized information (the cause of hallucinations), and uncover important factors that exacerbate this behaviour\"",
   "removal_reason": "",
   "source": "Longpre et al., Entity-Based Knowledge Conflicts in Question Answering (EMNLP 2021)",
   "source_url": "https://arxiv.org/abs/2109.05052",
   "locator": "Abstract",
   "date_read": "2026-08-20"
  },
  {
   "id": "11",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-11",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Construct validity from measurement theory applies to ML metrics: a metric can mismatch the construct it purports to measure",
   "quote": "\"This process...introduces the potential for mismatches between the theoretical understanding of the construct purported to be measured and its operationalization\"",
   "removal_reason": "",
   "source": "Jacobs & Wallach, Measurement and Fairness (ACM FAccT 2021)",
   "source_url": "https://arxiv.org/abs/1912.05511",
   "locator": "Body",
   "date_read": "2026-08-20"
  },
  {
   "id": "12",
   "url": "https://durabilitycurve.com/claims/the-evaluation-inversion/#c-12",
   "status": "REPORTED",
   "status_label": "Reported",
   "removed": false,
   "claim": "Goodhart's law has a visible/regressional variant distinct from an adversarial one; the visible decoupling is the tractable case",
   "quote": "(well-established taxonomy; cited conceptually, not quoted verbatim in body)",
   "removal_reason": "",
   "source": "Manheim & Garrabrant, Categorizing Variants of Goodhart's Law",
   "source_url": "https://arxiv.org/abs/1803.04585",
   "locator": "Full paper",
   "date_read": "2026-08-20"
  }
 ]
}