Claim ledger

Everyone Got Safer. That's the Problem.

Safety has two numbers: how often each system fails, and whether they fail together. The field has spent years driving the first one down while almost no dashboard reports the second.

Read the essay Published Tests Law A: Goodhart Corollary

Claims
17
Verified at the primary source
0
Removed before publication
2
Ledger dated

17 claims: 15 checked, 2 reported. Plus 2 removed before publication, listed at the end.

Download this ledger: CSV · JSON · Markdown · CC BY 4.0 · Query it from Claude or ChatGPT

The evidence, row by row

Each row is a claim the essay makes, then the evidence recorded for it and the source it was checked against. The label says how far it was checked.

What the labels mean
Verified
Checked against the primary source itself by a checker who did not draft the piece, usually with the exact words, where they sit and the date read.
Executed
A run or observation done for the piece: code, a query, a count, or a check made live in an app. The claim is what it returned. Run records are not published.
Checked
Checked against its source by a separate checker in the older ledger format, which recorded no quote, locator or date.
Reported
Not verified word for word against a primary source: carried from a secondary source, from one that could not be opened in full, or supported only in part. The essay words it accordingly.
Struck
Drafted, checked and removed before publication. Kept here, crossed out, with the reason.
Excluded
Considered and deliberately left out of the piece. Kept here with the reason.
  1. 1 Checked

    July 2023, CMU + Center for AI Safety, ~20-token adversarial suffix that transfers across labs

    Zou et al., arXiv:2307.15043 (submitted 27 Jul 2023)

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 1, https://durabilitycurve.com/claims/everyone-got-safer/#c-1 (older-format ledger: the checker's record, no source link).

    Report an error
  2. 2 Checked

    GPT-3.5 "most of the time," Bard "two-thirds," GPT-4 "roughly half"

    Table 2 ensemble row: GPT-3.5 86.6%, PaLM-2/Bard 66.0%, GPT-4 46.9%

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 2, https://durabilitycurve.com/claims/everyone-got-safer/#c-2 (older-format ledger: the checker's record, no source link).

    Report an error
  3. 3 Checked

    One string got through the commercial models of two rival labs, while Claude was more mixed: 47.9% on Claude-1 and about 2.1% on Claude-2

    Table 2: 86.6 / 46.9 / 66.0 / Claude-1 47.9 / Claude-2 2.1

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 3, https://durabilitycurve.com/claims/everyone-got-safer/#c-3 (older-format ledger: the checker's record, no source link).

    Report an error
  4. 4 Checked

    Zou et al. say their attack did much better against GPT-based models, potentially because Vicuna, an open model it was built on, was trained on ChatGPT outputs

    Zou et al. §transfer

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 4, https://durabilitycurve.com/claims/everyone-got-safer/#c-4 (older-format ledger: the checker's record, no source link).

    Report an error
  5. 5 Checked

    Ross Ashby's cybernetics named the limit in 1956, so that part of the idea is not new

    Ashby, An Introduction to Cybernetics, 1956

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 5, https://durabilitycurve.com/claims/everyone-got-safer/#c-5 (older-format ledger: the checker's record, no source link).

    Report an error
  6. 7 Checked

    Baker et al. found that chain-of-thought monitoring helps at low optimisation pressure, but under too much optimisation agents learn to hide their intent while still reward hacking

    Baker et al., arXiv:2503.11926

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 7, https://durabilitycurve.com/claims/everyone-got-safer/#c-7 (older-format ledger: the checker's record, no source link).

    Report an error
  7. 8 Checked

    Bommasani et al. warned in 2021 that the defects of a foundation model are inherited by all the adapted models downstream

    Bommasani et al., arXiv:2108.07258, abstract

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 8, https://durabilitycurve.com/claims/everyone-got-safer/#c-8 (older-format ledger: the checker's record, no source link).

    Report an error
  8. 9 Checked

    Dziemian: 272k attempts, 13 models, 8,648 successes, all vulnerable, 0.5%-8.5%, cross-family

    Dziemian et al., arXiv:2603.15714

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 9, https://durabilitycurve.com/claims/everyone-got-safer/#c-9 (older-format ledger: the checker's record, no source link).

    Report an error
  9. 10 Checked

    Kuai et al. found behavioural entanglement across model families and state that apparent agreement reflects shared error modes rather than independent validation

    arXiv:2604.07650 (Kuai et al.) verbatim: "apparent agreement reflects shared error modes rather than independent validation"

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 10, https://durabilitycurve.com/claims/everyone-got-safer/#c-10 (older-format ledger: the checker's record, no source link).

    Report an error
  10. 11 Checked

    MMLU contamination "tens of percent," Llama-2

    >16% MMLU contaminated for Llama-2

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 11, https://durabilitycurve.com/claims/everyone-got-safer/#c-11 (older-format ledger: the checker's record, no source link).

    Report an error
  11. 12 Checked

    Li, J Fixed Income 9(4), 2000, pages

    pp. 43-54 (start 43 confirmed)

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 12, https://durabilitycurve.com/claims/everyone-got-safer/#c-12 (older-format ledger: the checker's record, no source link).

    Report an error
  12. 13 Checked

    Wired published Felix Salmon's article "Recipe for Disaster: The Formula That Killed Wall Street" on 23 February 2009

    Wired, 23 Feb 2009

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 13, https://durabilitycurve.com/claims/everyone-got-safer/#c-13 (older-format ledger: the checker's record, no source link).

    Report an error
  13. 14 Checked

    1987: Dow -22.6% (508 pts), S&P -20.4%, 19 Oct

    Fed Reserve History

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 14, https://durabilitycurve.com/claims/everyone-got-safer/#c-14 (older-format ledger: the checker's record, no source link).

    Report an error
  14. 15 Reported

    The Brady Commission report gave portfolio-insurance selling heavy weight among the causes of the 1987 crash, though the exact weight is still debated

    Brady Report 1988; "greatest importance" is secondary phrasing

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 15, https://durabilitycurve.com/claims/everyone-got-safer/#c-15 (older-format ledger: the checker's record, no source link).

    Report an error
  15. 16 Reported

    Aviation ~65% fall in fatal-accident rate, Boeing (not IATA)

    Boeing Statistical Summary via Flight Safety Foundation

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 16, https://durabilitycurve.com/claims/everyone-got-safer/#c-16 (older-format ledger: the checker's record, no source link).

    Report an error
  16. 17 Checked

    737 MAX MCAS single AoA sensor, hidden from pilots; 346 dead; grounded Mar 2019

    Lion Air 610 (189) + Ethiopian 302 (157) = 346

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 17, https://durabilitycurve.com/claims/everyone-got-safer/#c-17 (older-format ledger: the checker's record, no source link).

    Report an error
  17. 18 Checked

    Vision transfer tracks model similarity, ">90%" predictor; adjacent not same-domain

    Demontis et al. USENIX 2019; arXiv:2501.18629

    Link
    Cite

    Claim Ledger, "Everyone Got Safer. That's the Problem.", The Durability Curve, row 18, https://durabilitycurve.com/claims/everyone-got-safer/#c-18 (older-format ledger: the checker's record, no source link).

    Report an error

Removed before publication

Claims that were drafted, checked and taken out, or considered and left out. The essay does not make them; they stay here with the reason.

  1. 6 Struck

    Donald Campbell was a psychologist

    Why: The published essay no longer names Donald Campbell (earlier verdict: FIXED, "economist" corrected to "psychologist")

  2. 19 Struck

    Geer et al. made the security-monoculture argument earlier, in their 2003 paper "CyberInsecurity: The Cost of Monopoly"

    Why: The published essay does not cite Geer et al. (earlier verdict: ADDED, for honesty about precedent)

Cite and reuse

A single claim
Use the row's Cite button. It names the original source first, then this row, which carries its own link.
This ledger
Floyd, Harry (2026). Claim Ledger: Everyone Got Safer. That's the Problem. The Durability Curve. https://durabilitycurve.com/claims/everyone-got-safer/
The essay
Floyd, Harry (2026). Everyone Got Safer. That's the Problem. The Durability Curve. https://durabilitycurve.com/blog/everyone-got-safer/

The ledger data is licensed CC BY 4.0: reuse it, quote it or train on it, with credit to The Durability Curve and a link. Read the licence. Machine copies: JSON, CSV, Markdown.