Skip to content

Before You Trust a Study, Check How Strongly It Passed

We fixed the rule before opening each claim's re-check result: strong first passes held up 66% of the time, narrow ones 34%.

Claims
67
Verified at the primary source
49
Removed before publication
0
Last checked

67 claims: 49 verified, 15 executed, 3 reported.

Download this ledger: CSV · JSON · Markdown · CC BY 4.0 · Query it from Claude or ChatGPT

What the essay claims

Before: a study that passed its test, especially a large one, is a finding I can act on. After: how strongly a study's first result passed is a clue to whether it will hold up. In a test fixed before we opened the results, on SCORE's public data, strong first passes (p below .001) held up 66% of the time and narrow ones (.01 to .05) 34%, with each paper weighted equally. Sample size did less: the largest third of original samples held up 12 points more often than the smallest, with an interval from −5 to 27, so this test cannot show that size matters.

The claim ladder

What the ladder does not reach: health and money advice; a verdict on any single finding (the card gives odds for groups of findings); whether a finding that held is big enough to matter; the size condition in step 1 (a re-run at least as large as the original), a design choice not yet tested.

#ClaimWhose behaviour it measuresStatus / scope
1Strong first passes held up 66%, narrow ones 34%, a gap of 32 points (16 to 46), each paper weighted equallyOurs, SCORE's public data (331 claims, 212 papers), pre-registeredVerified by the locked run. Papers drawn at random from social- and behavioural-science journals; which claims were re-checked depended on feasibility
2Passes between .001 and .01 held up 22%, no more often than narrow onesOurs, the same runFrom the locked run; a secondary figure the plan reported without a verdict
3The largest third of original samples held up 12 points more often than the smallest, interval −5 to 27Ours, the same runNot detectable; under the plan's rule the sample-size falsifier did not fire
4On re-checks that collected new data, the gap was as largeOurs, SCORE's new-data re-checksExploratory, a split added afterwards
5On re-runs researchers chose to do: 58% against 38%Ours, FReD, pre-registeredHeld up if the re-run's own report said so; mixed results counted as failures
620 of 32 originals below .001 held up, against 2 of 11 above .04OSC 2015Verified
7Interactions held up 8 times in 37, main or simple effects 23 in 49; results rated surprising held up less oftenOSC 2015Verified (OSC data only)
8Using each original's p-value, together with an estimate of how large effects in that field tend to be, forecasts about as accurate as scientists' prediction markets; a psychology finding at p = .05 has about a one-in-five chance that a same-size re-run is significantVu & Faridani (2026, working paper)Verified. Scored on the same re-run projects that shaped the model
9Lay readers right about 59% of the time from plain descriptions, about 67% when shown the strength of evidenceHoogeveen, Sarafoglou & Wagenmakers (2020)Verified
10On 27 findings the card was not built from: act 5 of 5 held, wait 0 of 4, try small 9 of 18; forced to call all 27, 67%, the same as the p-value aloneOurs, pre-registered card test on the 2018 Nature/Science re-runs and Many Labs 2A look back: outcomes and lay ratings were already published. Small groups (act 48% to 100%, wait 0% to 60%)
11With one reader's doubt instead of the crowd's, about 3 in 10 "wait" findings heldOurs, small-group check, pre-registeredA look back on the same 27 findings; signs 3 and 4 are experimental
12Willpower: correcting the 198 published tests for signs of publication bias found an indication that the effect was no different from zero (2014); 23 labs on a registered plan found almost nothing (2016)Carter & McCullough (2014); Hagger et al. (2016)Verified
13Without a qualifying re-run, expect about half the claimed effectOSC 2015Verified; an average over all pairs, and an effect can shrink further or vanish
14Six famous claims, scored from their original papers with calls fixed before reading any verdictEach claim's original paper and registered re-runAn illustration: the author already knew how some had turned out

What would make it wrong

The main cue is wrong if the next large batch of re-runs (a new FReD release, or another multi-lab project) shows first results at p < .001 failing as often as those near .05. The sample-size claim is wrong if a pre-registered test on a fresh pool finds the largest original samples holding up at least 10 points more often.

If well-powered re-runs of findings that pass the Re-check fail as often as findings it flags, the check is useless, and we will say so.

Re-checked 1 Nov 20262 holding1 untestedThe re-check

The evidence, row by row

Each row is a claim the essay makes, then the evidence recorded for it and the source it was checked against. For verified rows that is usually the exact words, where they sit and the day they were read. The label says how far it was checked.

What the labels mean
Verified
Checked against the primary source itself by a checker who did not draft the piece, usually with the exact words, where they sit and the date read.
Executed
A run or observation done for the piece: code, a query, a count, or a check made live in an app. The claim is what it returned. Run records are not published.
Checked
Checked against its source by a separate checker in the older ledger format, which recorded no quote, locator or date.
Reported
Not verified word for word against a primary source: carried from a secondary source, from one that could not be opened in full, or supported only in part. The essay words it accordingly.
Struck
Drafted, checked and removed before publication. Kept here, crossed out, with the reason.
Excluded
Considered and deliberately left out of the piece. Kept here with the reason.
Sources (44)
  1. OSC 2015 7 rows
  2. Hagger et al. 2 rows
  3. Hoogeveen 5 rows
  4. Li et al. 1 row
  5. Simonsohn et al. 1 row
  6. Coles et al. 3 rows
  7. Baumeister 2 rows
  8. Strack 1 row
  9. Caruso 1 row
  10. Jacowitz & Kahneman 1 row
  11. Karpicke & Blunt 1 row
  12. Tversky & Kahneman 1 row
  13. Wagenmakers et al. 1 row
  14. Klein et al. 1 row
  15. As B5r. 2 rows
  16. Camerer et al. 1 row
  17. fred_two_cues.py 1 row
  18. FReD claim_text_o for doi 10.1177/01461672952111004 1 row
  19. own run: scipy.stats.t.ppf(1 - 0.0005 1 row
  20. heldout_card_test.py 1 row
  21. as F13 1 row
  22. smallgroup_check.py 1 row
  23. Clopper–Pearson intervals computed with scipy.stats.beta from F13's counts 1 row
  24. Altmejd 1 row
  25. Benjamin et al. 1 row
  26. score_cue_test.py 1 row
  27. as F20 2 rows
  28. Tyner et al. 3 rows
  29. Colquhoun 1 row
  30. Gordon 1 row
  31. 80 1 row
  32. arithmetic on F13's counts 1 row
  33. explore_score_newdata.py 1 row
  34. heldout_signs.csv 2 rows
  35. Abatayo et al. 1 row
  36. de Menard 1 row
  37. Vu & Faridani 3 rows
  38. Carter & McCullough 3 rows
  39. Jacob Falkovich 1 row
  40. James Bailey 1 row
  41. Lindsay 1 row
  42. Lakens 1 row
  43. van Aert 1 row
  44. Mayrhofer 1 row
  1. A01b Verified Read

    Replication effects were about half the original: mean r 0.197 vs 0.403

    "Replication effects were half the magnitude of original effects … original study effect sizes ( M = 0.403, SD = 0.188) were reliably larger than replication effect sizes ( M = 0.197, SD = 0.257)"

    OSC 2015, accepted manuscript (OSF) · Abstract (PDF p. 1); Results, effect sizes (PDF p. 18) · osf.io

    Link
    Cite

    OSC 2015, accepted manuscript (OSF), Abstract (PDF p. 1); Results, effect sizes (PDF p. 18), https://osf.io/447b3. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A01b, https://durabilitycurve.com/claims/survives-a-recheck/#c-A01b (source read 2026-10-26).

    Report an error
  2. A01c Verified Read

    Originals with p < .001: 20 of 32 replicated

    "Almost two thirds (20 of 32, 63%) of original studies with P < … 0.001 had a significant P value in the replication."

    OSC 2015, accepted manuscript (OSF) · Results, correlates of reproducibility (PDF pp. 20–21; the sentence crosses a page break) · osf.io

    Link
    Cite

    OSC 2015, accepted manuscript (OSF), Results, correlates of reproducibility (PDF pp. 20–21; the sentence crosses a page break), https://osf.io/447b3. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A01c, https://durabilitycurve.com/claims/survives-a-recheck/#c-A01c (source read 2026-10-26).

    Report an error
  3. A01d Verified Read

    Originals with p > .04: 2 of 11 replicated

    "whereas 6 of 23 (26%) that had a P value between 0.02 < P < 0.04 and 2 of 11 (18%) that had a P value > 0.04 did so"

    OSC 2015, accepted manuscript (OSF) · Results (PDF p. 20) · osf.io

    Link
    Cite

    OSC 2015, accepted manuscript (OSF), Results (PDF p. 20), https://osf.io/447b3. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A01d, https://durabilitycurve.com/claims/survives-a-recheck/#c-A01d (source read 2026-10-26).

    Report an error
  4. A01e Verified Read

    Main or simple effects replicated 23/49; interactions 8/37

    "Among original, significant effects, 23 of the 49 (47%) that tested main or simple effects replicated at P < 0.05, but just 8 of the 37 (22%) that tested interaction effects did."

    OSC 2015, accepted manuscript (OSF) · Results (PDF p. 20) · osf.io

    Link
    Cite

    OSC 2015, accepted manuscript (OSF), Results (PDF p. 20), https://osf.io/447b3. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A01e, https://durabilitycurve.com/claims/survives-a-recheck/#c-A01e (source read 2026-10-26).

    Report an error
  5. A01f Verified Read

    Rated surprisingness correlated −0.244 (Spearman) with replication significance

    "Surprising original result -0.244 0.102 -0.181 -0.113 -0.241 … Surprising effects were less reproducible"

    OSC 2015, accepted manuscript (OSF) · Table 2, first column = "Replications P < .05 in original direction" (PDF p. 12); Results (PDF p. 22) · osf.io

    Link
    Cite

    OSC 2015, accepted manuscript (OSF), Table 2, first column = "Replications P < .05 in original direction" (PDF p. 12); Results (PDF p. 22), https://osf.io/447b3. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A01f, https://durabilitycurve.com/claims/survives-a-recheck/#c-A01f (source read 2026-10-26).

    Report an error
  6. A04a Verified Read

    Ego-depletion RRR: 23 labs, N = 2,141, d = 0.04 [−0.07, 0.15]

    "Multiple laboratories (k = 23, total N = 2,141) conducted replications of a standardized ego-depletion protocol … (d = 0.04, 95% CI [-0.07, 0.15]"

    Hagger et al. (2016), Perspectives on Psychological Science 11(4):546–573. Published SAGE PDF from the Univ. Konstanz repository (KOPS) · Abstract, p. 546 · kops.uni-konstanz.de

    Link
    Cite

    Hagger et al. (2016), Perspectives on Psychological Science 11(4):546–573. Published SAGE PDF from the Univ. Konstanz repository (KOPS), Abstract, p. 546, https://kops.uni-konstanz.de/server/api/core/bitstreams/d1f20ce9-761f-40e6-aaa8-4992d4744d3c/content. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A04a, https://durabilitycurve.com/claims/survives-a-recheck/#c-A04a (source read 2026-10-26).

    Report an error
  7. A07a Verified Read

    27 findings judged by 233 people without a PhD in psychology: 59% correct

    "Each of 27 high-profile social-science findings was evaluated by 233 people without a Ph.D. in psychology. Results showed that these laypeople predicted replication success with above-chance accuracy (i.e., 59%)."

    Hoogeveen, Sarafoglou & Wagenmakers (2020), AMPPS 3(3):267–285. Final published version (UvA repository) · Abstract, p. 267 · pure.uva.nl

    Link
    Cite

    Hoogeveen, Sarafoglou & Wagenmakers (2020), AMPPS 3(3):267–285. Final published version (UvA repository), Abstract, p. 267, https://pure.uva.nl/ws/files/53717212/2515245920919667.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A07a, https://durabilitycurve.com/claims/survives-a-recheck/#c-A07a (source read 2026-10-26).

    Report an error
  8. A07b Verified Read

    67% correct when shown the strength of evidence

    "when participants were informed about the strength of evidence from the original studies, this boosted their prediction performance to 67%."

    Hoogeveen et al. (2020), published version · Abstract, p. 267; group-level ω .59 [.57, .60] vs .67 [.65, .69] on p. 275 · pure.uva.nl

    Link
    Cite

    Hoogeveen et al. (2020), published version, Abstract, p. 267; group-level ω .59 [.57, .60] vs .67 [.65, .69] on p. 275, https://pure.uva.nl/ws/files/53717212/2515245920919667.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A07b, https://durabilitycurve.com/claims/survives-a-recheck/#c-A07b (source read 2026-10-26).

    Report an error
  9. A07c Verified Read

    Two separate groups: 123 people in the description-only condition and 110 in the description-plus-evidence condition; the two accuracy figures in A07a and A07b come from these two groups

    "the final sample consisted of 233 participants, 123 in the description-only condition and 110 in the"

    Hoogeveen et al. (2020), published version · Method, Participants, p. 269 · pure.uva.nl

    Link
    Cite

    Hoogeveen et al. (2020), published version, Method, Participants, p. 269, https://pure.uva.nl/ws/files/53717212/2515245920919667.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A07c, https://durabilitycurve.com/claims/survives-a-recheck/#c-A07c (source read 2026-10-26).

    Report an error
  10. A07d Verified Read

    Evidence strength shown as a Bayes factor plus a verbal label

    "the Bayes factor and its verbal interpretation (e.g., "moderate evidence") were added to the description of each study."

    Hoogeveen et al. (2020), published version · Method, Materials, p. 270; labels follow Jeffreys (1939) as adjusted by Lee & Wagenmakers (2013) · pure.uva.nl

    Link
    Cite

    Hoogeveen et al. (2020), published version, Method, Materials, p. 270; labels follow Jeffreys (1939) as adjusted by Lee & Wagenmakers (2013), https://pure.uva.nl/ws/files/53717212/2515245920919667.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A07d, https://durabilitycurve.com/claims/survives-a-recheck/#c-A07d (source read 2026-10-26).

    Report an error
  11. A09c Verified Read

    215 surveyed psychologists ranked "large sample size" first (M = 3.84)

    "Respondents rated "large sample size" as the most important criterion (M = 3.84), judging it to be significantly more important than the second most important factor, "large effect size" (M = 3.61)" [...] "resulting in a final sample size of 215 psychological scientists"

    Li et al. (2024), publisher PDF · Survey results, p. 10 of 15 (n = 215 on the same page) · journals.plos.org

    Link
    Cite

    Li et al. (2024), publisher PDF, Survey results, p. 10 of 15 (n = 215 on the same page), https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0306911&type=printable. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A09c, https://durabilitycurve.com/claims/survives-a-recheck/#c-A09c (source read 2026-10-26).

    Report an error
  12. A12c Verified Read

    At 33% power (t tests with 38 degrees of freedom), 57.6% of significant p values are greater than .01

    "t tests with 38 degrees of freedom and powered at 33% will result in 57.6% of significant p values being greater than" [decoded: the text layer reads greater than .0l; the page reads .01]

    Simonsohn et al. (2014), published PDF · p. 538 (the text continues ".01"; the same passage puts 10.4% above .04); note (derived): so 42.4% fall below .01 · gwern.net

    Link
    Cite

    Simonsohn et al. (2014), published PDF, p. 538 (the text continues ".01"; the same passage puts 10.4% above .04); note (derived): so 42.4% fall below .01, https://gwern.net/doc/statistics/meta-analysis/2014-simonsohn.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A12c, https://durabilitycurve.com/claims/survives-a-recheck/#c-A12c (source read 2026-10-26).

    Report an error
  13. A14a Verified Read

    Many Smiles: n = 3,878 across 19 countries; mimicry and voluntary posing both amplified and initiated happiness

    "Data from n = 3,878 participants spanning 19 countries indicated that a facial mimicry and voluntary facial action task could both amplify and initiate feelings of happiness."

    Coles et al. (2022), Nature Human Behaviour 6:1731–1742. Published abstract on the publisher page (full text paywalled) · Abstract · nature.com

    Link
    Cite

    Coles et al. (2022), Nature Human Behaviour 6:1731–1742. Published abstract on the publisher page (full text paywalled), Abstract, https://www.nature.com/articles/s41562-022-01458-9. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A14a, https://durabilitycurve.com/claims/survives-a-recheck/#c-A14a (source read 2026-10-26).

    Report an error
  14. A14b Verified Read

    Pen-in-mouth evidence was less conclusive

    "However, evidence of facial feedback effects was less conclusive when facial feedback was manipulated unobtrusively via a pen-in-mouth task."

    Coles et al. (2022), published abstract · Abstract · nature.com

    Link
    Cite

    Coles et al. (2022), published abstract, Abstract, https://www.nature.com/articles/s41562-022-01458-9. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A14b, https://durabilitycurve.com/claims/survives-a-recheck/#c-A14b (source read 2026-10-26).

    Report an error
  15. B1o Verified Read

    Ego depletion. Baumeister, Bratslavsky, Muraven & Tice (1998), Experiment 1: hungry students ate radishes while resisting chocolates (or ate chocolates, or no-food control), then persistence on unsolvable puzzles was timed. N = 67. Key test as reported: one-way ANOVA across the three conditions, F(2, 64) = 26.88, p < .001; radish vs chocolate t(44) = 6.03, p < .001. Design: SIMPLE (one manipulation, one outcome: persistence time).

    "Data were collected in individual sessions from 67 introductory psychology students" [...] "A one-way analysis of variance (ANOVA) indicated significant variation among the three conditions, F(2, 64) = 26.88, p < .001." [...] "participants in the radish condition quit sooner on the frustrating task than did participants in either the chocolate condition, t(44) = 6.03, p < .001"

    Baumeister et al. (1998), JPSP 74(5), 1252–1265. Published version (scan with text layer), course copy at the University of Washington. · Exp. 1, Method (Participants) PDF p. 3; Results (Persistence) PDF p. 4 · faculty.washington.edu

    Link
    Cite

    Baumeister et al. (1998), JPSP 74(5), 1252–1265. Published version (scan with text layer), course copy at the University of Washington., Exp. 1, Method (Participants) PDF p. 3; Results (Persistence) PDF p. 4, https://faculty.washington.edu/jdb/345/345%20Articles/Baumeister%20et%20al.%20%281998%29.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B1o, https://durabilitycurve.com/claims/survives-a-recheck/#c-B1o (source read 2026-10-26).

    Report an error
  16. B2o Verified Read

    Strack, Martin & Stepper 1988, Study 1: Ninety-two undergraduates held a pen in the lips, the non-dominant hand or the teeth while rating cartoons; reported as a significant linear contrast, t(89) = 1.85, p = .03 (the text layer reads f(89)). Design: SIMPLE

    "Subjects. Ninety-two male and female undergraduates of the University of Illinois participated in the experiment" [...] "The predicted differences were confirmed by a significant linear contrast, f(89) = 1.85, p = .03."

    Strack, Martin & Stepper (1988), JPSP 54(5), 768–777. Published APA version (scan with text layer), third-party copy on GitHub Pages. · Study 1, Method PDF p. 3; Results, journal p. 772 (PDF p. 5); note: means 4.32 / 4.77 / 5.14 from the paper's table, not quoted; two-tailed p for t(89) = 1.85 is about .068 (computed), so p = .03 matches a one-tailed test · jhrcook.github.io

    Link
    Cite

    Strack, Martin & Stepper (1988), JPSP 54(5), 768–777. Published APA version (scan with text layer), third-party copy on GitHub Pages., Study 1, Method PDF p. 3; Results, journal p. 772 (PDF p. 5); note: means 4.32 / 4.77 / 5.14 from the paper's table, not quoted; two-tailed p for t(89) = 1.85 is about .068 (computed), so p = .03 matches a one-tailed test, https://jhrcook.github.io/wagenmaker-data-analysis/papers/Strack_1988_Inhibiting-and-facilitating-conditions-of-the-human-smile-a-nonobtrusive-test-of-the-facial-feedback-hypothesis.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B2o, https://durabilitycurve.com/claims/survives-a-recheck/#c-B2o (source read 2026-10-26).

    Report an error
  17. B5o Verified Read

    Caruso, Vohs, Baxter & Waytz 2013, Experiment 1: Thirty adults; a faint image of $100 bills behind the instructions; t(28) = 2.12, p = .043, d = 0.80. Design: SIMPLE

    "Thirty adults from a university study pool" [...] "Participants assigned to the money condition saw a faint image of $100 bills in the background of the instruction screen" [...] "As predicted, participants in the money condition more strongly endorsed system justification than did participants in the control condition, t(28) /H110052.12, p /H11005.043, d /H110050.80" [decoded: t(28) = 2.12, p = .043, d = 0.80; the text layer renders = as /H11005]

    Caruso, Vohs, Baxter & Waytz (2013), JEP: General 142(2), 301–306. Published APA version (online-first), copy linked from the Kellogg School research page. · Exp. 1, Method and Results, PDF p. 3 · static.squarespace.com

    Link
    Cite

    Caruso, Vohs, Baxter & Waytz (2013), JEP: General 142(2), 301–306. Published APA version (online-first), copy linked from the Kellogg School research page., Exp. 1, Method and Results, PDF p. 3, http://static.squarespace.com/static/51e3f4ede4b053e5f0062efd/t/51f7e294e4b03d20c9b472cc/1375199892722/mere-exposure-to-money-increases-endorsement-of-free-market-systems-and-social-inequality%20.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B5o, https://durabilitycurve.com/claims/survives-a-recheck/#c-B5o (source read 2026-10-26).

    Report an error
  18. B6o Reported Read

    Anchoring. Jacowitz & Kahneman (1995): a calibration group estimates quantities; anchored groups first judge whether a high or low anchor (15th or 85th percentile of calibration estimates) is above or below the true value, then estimate. N: not obtained. Test statistic and p: not obtained. Design: SIMPLE (provisional, from the abstract: one manipulation, anchor high vs low, on estimates).

    "Subjects in the anchored condition first judge whether a specified number (the anchor) is higher or lower than the true value before estimating each quantity." [...] "(15th and 85th percentiles in this study)"

    Jacowitz & Kahneman (1995), PSPB 21(11), 1161–1166. Abstract only, reconstructed from the OpenAlex index record. · Abstract · api.openalex.org

    Link
    Cite

    Jacowitz & Kahneman (1995), PSPB 21(11), 1161–1166. Abstract only, reconstructed from the OpenAlex index record., Abstract, https://api.openalex.org/works/doi:10.1177/01461672952111004. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B6o, https://durabilitycurve.com/claims/survives-a-recheck/#c-B6o (source read 2026-10-26).

    Report an error
  19. B7o Verified Read

    Karpicke & Blunt 2011, Experiment 1: Eighty undergraduates; retrieval practice vs concept mapping one week later: d = 1.50, F(1, 38) = 21.63; no p-value reported. Design: SIMPLE

    "Eighty undergraduate students participated in Experiment 1." [...] "the advantage of retrieval practice (M = 0.67) over elaborative studying with concept mapping (M = 0.45) represented about a 50% improvement in long-term retention scores [d = 1.50, F1,38 = 21.63" [decoded: d = 1.50, F(1, 38) = 21.63; the text layer reads F1,38]

    Karpicke & Blunt (2011), Science 331, 772–775. Published Science version, copy on the author's Purdue lab site. · Exp. 1, journal p. 772–773 (PDF pp. 2–3); note: key comparison n = 40 (df 38) · learninglab.psych.purdue.edu

    Link
    Cite

    Karpicke & Blunt (2011), Science 331, 772–775. Published Science version, copy on the author's Purdue lab site., Exp. 1, journal p. 772–773 (PDF pp. 2–3); note: key comparison n = 40 (df 38), https://learninglab.psych.purdue.edu/downloads/2011/2011_Karpicke_Blunt_Science.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B7o, https://durabilitycurve.com/claims/survives-a-recheck/#c-B7o (source read 2026-10-26).

    Report an error
  20. B8o Verified Read

    Framing flips preferences. Tversky & Kahneman (1981), Problems 1 and 2 ("Asian disease"), separate groups. N = 152 (gain frame) and N = 155 (loss frame). Result as reported: 72% chose the sure option under the gain frame; 22% chose the sure option under the loss frame. No test statistic or p-value is reported. Design: SIMPLE (one manipulation, frame; one outcome, choice).

    "Problem 1 [N = 152]: Imagine that the U.S. is preparing for the outbreak of an unusual Asian disease" [...] "If Program A is adopted, 200 people will be saved. [72 percent]" [...] "Problem 2 [N = 155]: If Program C is adopted 400 people will die. [22 percent]" [...] "The change is accompanied by a pronounced shift from risk aversion to risk taking."

    Tversky & Kahneman (1981), Science 211(4481), 453–458. Published version (JSTOR scan with text layer), copy on gwern.net. · Problems 1–2, journal p. 453 (PDF p. 1) · gwern.net

    Link
    Cite

    Tversky & Kahneman (1981), Science 211(4481), 453–458. Published version (JSTOR scan with text layer), copy on gwern.net., Problems 1–2, journal p. 453 (PDF p. 1), https://gwern.net/doc/psychology/1981-tversky.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B8o, https://durabilitycurve.com/claims/survives-a-recheck/#c-B8o (source read 2026-10-26).

    Report an error
  21. B1r Verified Read

    Hagger, Chatzisarantis et al. (2016), Registered Replication Report of ego depletion. Qualifies: yes. Pre-registered: yes. Labs: 23. N = 2,141. Author overlap: none (Baumeister is listed as a protocol vetter, not an author). Task differs from B1o: depletion by a letter-"e" task (adapted from Baumeister et al. 1998, Study 4), outcome the multi-source interference task (MSIT); the radish task was judged too experimenter-dependent for many labs. Headline: d = 0.04, 95% CI [−0.07, 0.15].

    "Participating labs preregistered their implementation plan on the Open Science Framework and conducted independent replications." [...] "Multiple laboratories (k = 23, total N = 2,141) conducted replications of a standardized ego-depletion protocol based on a sequential-task paradigm by Sripada et al." [...] "Meta-analysis of the studies revealed that the size of the ego-depletion effect was small with 95% confidence intervals (CIs) that encompassed zero (d = 0.04, 95% CI [−0.07, 0.15]." [...] "Protocol vetted by: Chandra Sripada, Daniel Kessler, Roy Baumeister" [...] "required participants to taste radishes and resist cookies (Baumeister et al., 1998, Study 1). This task would require extensive experimenter involvement in its delivery" [...] "The first task was a modified version of the letter “e” task (Baumeister et al., 1998, Study 4)"

    Hagger et al. (2016), Perspectives on Psychological Science 11(4), 546–573. Published SAGE PDF in the University of Konstanz repository (KOPS). · Title block and Abstract, journal p. 546; Method, pp. 548–550 · kops.uni-konstanz.de

    Link
    Cite

    Hagger et al. (2016), Perspectives on Psychological Science 11(4), 546–573. Published SAGE PDF in the University of Konstanz repository (KOPS)., Title block and Abstract, journal p. 546; Method, pp. 548–550, https://kops.uni-konstanz.de/server/api/core/bitstreams/d1f20ce9-761f-40e6-aaa8-4992d4744d3c/content. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B1r, https://durabilitycurve.com/claims/survives-a-recheck/#c-B1r (source read 2026-10-26).

    Report an error
  22. B2r Verified Read

    Wagenmakers et al. 2016 RRR: 17 independent direct replications of Strack Study 1 on one vetted protocol; smile minus pout rating difference 0.03, 95% CI -0.11 to 0.16; Strack supplied materials and early feedback and declined to review the final protocol; the neutral hand condition was omitted. Qualifies: yes

    "This Registered Replication Report describes the results of 17 independent direct replications of Study 1 from Strack et al. (1988), all of which followed the same vetted protocol." [...] "Data and registered protocols: https://osf.io/pkd65/" [...] "Our meta-analysis revealed a rating difference of 0.03 units with a 95% confidence interval ranging from −0.11 to 0.16." [...] "Fritz Strack provided the original materials as well as valuable feedback and constructive suggestions during the early stages of protocol development by the lead lab." [...] "Although he declined to review the final protocol" [...] "we omitted the neutral “nondominant hand condition” from the design"

    Wagenmakers et al. (2016), Perspectives on Psychological Science 11(6), 917–928. Published SAGE PDF in the Maynooth University repository. · Title block and Abstract, journal p. 917; pp. 918–919; Table 1, p. 921; note (our sums of Table 1): 1,894 analysed, 2,533 tested · mural.maynoothuniversity.ie

    Link
    Cite

    Wagenmakers et al. (2016), Perspectives on Psychological Science 11(6), 917–928. Published SAGE PDF in the Maynooth University repository., Title block and Abstract, journal p. 917; pp. 918–919; Table 1, p. 921; note (our sums of Table 1): 1,894 analysed, 2,533 tested, https://mural.maynoothuniversity.ie/id/eprint/15647/1/1745691616674458.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B2r, https://durabilitycurve.com/claims/survives-a-recheck/#c-B2r (source read 2026-10-26).

    Report an error
  23. B2s Verified Read

    Coles et al., Many Smiles Collaboration (2022), multi-site adversarial collaboration: published abstract and author list. Qualifies: yes. Pre-registered: yes. Countries: 19. N = 3,878. Author overlap: yes, Fritz Strack (first author of B2o) is a co-author. Headline (abstract): facial mimicry and voluntary facial action raised happiness; the pen-in-mouth evidence was "less conclusive". Outcome differs from B2o: self-reported happiness, not cartoon funniness.

    "We thus formed a global adversarial collaboration and carried out a preregistered, multicentre study designed to specify and test the conditions that should most reliably produce facial feedback effects." [...] "Data from n = 3,878 participants spanning 19 countries" [...] "However, evidence of facial feedback effects was less conclusive when facial feedback was manipulated unobtrusively via a pen-in-mouth task." [...] "Fritz Strack"

    Coles et al. (2022), Nature Human Behaviour 6, 1731–1742. Published article page on nature.com (abstract and author list are open; body paywalled). · Abstract; author list · nature.com

    Link
    Cite

    Coles et al. (2022), Nature Human Behaviour 6, 1731–1742. Published article page on nature.com (abstract and author list are open; body paywalled)., Abstract; author list, https://www.nature.com/articles/s41562-022-01458-9. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B2s, https://durabilitycurve.com/claims/survives-a-recheck/#c-B2s (source read 2026-10-26).

    Report an error
  24. B5r Verified Read

    Many Labs 1, currency priming (pre-registered; 36 samples, 6,344 participants overall): original d = 0.80 [.05, 1.54]; replication weighted d = -0.02 [-0.08, 0.04], unweighted 0.01 [-0.06, 0.09]; proportion of samples significant in the original direction 0.03; t = -0.79; did not replicate. Qualifies: yes

    "All materials, data, videos of the procedure, and the original preregistered design are available at the project page https://osf.io/ydpbf/." [...] "In total, there were 36 samples and settings that collected data from a total of 6,344 participants" [...] "Currency priming 0.80 .05, 1.54 0.00 0.01 /C00.06, 0.09 /C00.02 /C00.08, 0.04 0.00 0.03 0.97 t = /C00.79 6,331 6,333 0.83" [...] "Two studies – flag priming and currency priming – did not replicate the original effects." [...] "the US dollar was usually replaced with the relevant country’s currency" [...] "We thank Eugene Caruso" [decoded Table 2 row: 0.80 .05, 1.54 0.00 0.01 -0.06, 0.09 -0.02 -0.08, 0.04 0.00 0.03 0.97 t = -0.79; the text layer renders minus as /C0]

    Klein et al. (2014), Social Psychology 45(3), 142–152. Published Hogrefe version (OpenMind licence), on the project's OSF page. · Method p. 143; currency description p. 145; Table 2 p. 148; Results p. 149; Author note; note: original CI 95%, replication CI 99% (Table 2 columns); N = 6,333 for this effect · osf.io

    Link
    Cite

    Klein et al. (2014), Social Psychology 45(3), 142–152. Published Hogrefe version (OpenMind licence), on the project's OSF page., Method p. 143; currency description p. 145; Table 2 p. 148; Results p. 149; Author note; note: original CI 95%, replication CI 99% (Table 2 columns); N = 6,333 for this effect, https://osf.io/ebmf8/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B5r, https://durabilitycurve.com/claims/survives-a-recheck/#c-B5r (source read 2026-10-26).

    Report an error
  25. B6r Verified Read

    Many Labs 1, anchoring (between-subjects, four of the original topics): weighted d 2.42 [2.33, 2.51] babies born; 2.23 [2.14, 2.32] Mt. Everest; 1.79 [1.71, 1.87] Chicago; 1.17 [1.09, 1.25] distance to NYC; original pooled 0.93 [.51, 1.33]; proportion of samples significant in the original direction 1.00 on every item; N 5,284 to 5,627 per item. Qualifies: yes

    "For the purposes of the replication we provided anchoring information before asking just for the estimated quantity for four of the topics from the original study" [...] "Anchoring – babies born 0.93 .51, 1.33 2.43 2.60 2.41, 2.79 2.42 2.33, 2.51 0.00 1.00 0.00 t = 90.49 5,607 5,609 <.001" [...] "Anchoring – Mt. Everest 0.93 .51, 1.33 2.00 2.45 2.12, 2.77 2.23 2.14, 2.32 0.00 1.00 0.00 t = 83.66 5,625 5,627 <.001" [...] "Anchoring – Chicago 0.93 .51, 1.33 1.88 2.05 1.84, 2.25 1.79 1.71, 1.87 0.00 1.00 0.00 t = 65.00 5,282 5,284 <.001" [...] "Anchoring – distance to NYC 0.93 .51, 1.33 1.18 1.27 1.13, 1.40 1.17 1.09, 1.25 0.00 1.00 0.00 t = 42.86 5,360 5,362 <.001" [...] "The Anchoring original effect size is a mean point-biserial correlation computed across 15 different questions in a test-retest design, whereas the present replication adopted a between-subjects design with random assignments."

    As B5r. · Anchoring description p. 144; Table 2 and notes p. 148; note: 36 samples (B5r); original CI 95%, replication CI 99% (Table 2 columns) · osf.io

    Link
    Cite

    As B5r., Anchoring description p. 144; Table 2 and notes p. 148; note: 36 samples (B5r); original CI 95%, replication CI 99% (Table 2 columns), https://osf.io/ebmf8/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B6r, https://durabilitycurve.com/claims/survives-a-recheck/#c-B6r (source read 2026-10-26).

    Report an error
  26. B7r Verified Read

    SSRP replication of Karpicke & Blunt (pre-registered on OSF before data collection; one team): replication r = 0.384, p = 0.006, n = 49, against original r = 0.602, p < 0.001, n = 40; relative effect size 0.638; hard copies of the original materials. Qualifies: yes

    "pre-registered at the Open Science Framework (OSF, https://osf.io/pfdyw/) prior to starting the data collection" [...] "In a memory test one week after learning, Retrieval Practice leads to participants recalling more correct information than Concept-Mapping." [...] "Original test statistics: N = 40 (20 in each condition)" [...] "Karpicke and Blunt (2011), Science 0 . 602 < 0. 001 40 (40) 0 . 384 0 . 006 49 (49) yes 0 . 922 0 . 638" [...] "our replication used hard copies of all of the original materials used in the original study" [...] "Karpicke and Blunt (2011), Science ✓ ✗ ✓" [decoded Supplementary Table 3 row: original r = 0.602, p < 0.001, n = 40; replication r = 0.384, p = 0.006, n = 49; relative effect size 0.638; the text layer spaces the digits]

    Camerer et al. (2018), Nature Human Behaviour 2, 637–644, Supplementary Information (published SI file from Nature's static-content server). · SI p. 9 (software); p. 11 (pre-registration); p. 50 (hypothesis); Supp. Table 2 p. 53; Supp. Table 3 p. 54 · static-content.springer.com

    Link
    Cite

    Camerer et al. (2018), Nature Human Behaviour 2, 637–644, Supplementary Information (published SI file from Nature's static-content server)., SI p. 9 (software); p. 11 (pre-registration); p. 50 (hypothesis); Supp. Table 2 p. 53; Supp. Table 3 p. 54, https://static-content.springer.com/esm/art%3A10.1038%2Fs41562-018-0399-z/MediaObjects/41562_2018_399_MOESM1_ESM.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B7r, https://durabilitycurve.com/claims/survives-a-recheck/#c-B7r (source read 2026-10-26).

    Report an error
  27. B8r Verified Read

    Many Labs 1, gain vs loss framing: original d = 1.13 [.89, 1.37]; replication weighted d = 0.60 [0.53, 0.67], unweighted 0.62 [0.52, 0.71]; chi-square = 516.4, N = 6,271, p < .001; proportion of samples significant in the original direction 0.86. Qualifies: yes

    "Gain vs. loss framing 1.13 .89, 1.37 0.58 0.62 0.52, 0.71 0.60 0.53, 0.67 0.00 0.86 0.14 v 2 = 516.4 1 6,271 <.001" [...] "The replication replaced the phrase ‘‘the United States ’’ with the country of data collection, and the word ‘‘Asian’’ was omitted from"

    As B5r. · Framing description p. 144; Table 2 p. 148; note: 36 samples (B5r); original CI 95%, replication CI 99% (Table 2 columns) · osf.io

    Link
    Cite

    As B5r., Framing description p. 144; Table 2 p. 148; note: 36 samples (B5r); original CI 95%, replication CI 99% (Table 2 columns), https://osf.io/ebmf8/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row B8r, https://durabilitycurve.com/claims/survives-a-recheck/#c-B8r (source read 2026-10-26).

    Report an error
  28. F1 Executed Read

    Our test, H1: first results at p < .001 held 58% (589 re-runs, 294 papers); .01 to .05 held 38% (341, 189); +20 points [+12, +28]. Scope: replications people chose to run, judged by the replicating authors

    Result: HEADLINE H1: original p < .001 replicated 58% (589 replications, 294 papers); .01 to .05 replicated 38% (341, 189 papers); difference +20 points [95% CI +12 to +28] → CUE HOLDS

    fred_two_cues.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); FReD (Röseler et al. 2024, CC BY 4.0), OSF 2tbvd v1 · durabilitycurve.com

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (fred_two_cues.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); FReD (Röseler et al. 2024, CC BY 4.0), OSF 2tbvd v1), https://durabilitycurve.com/methods/before-you-trust-a-study/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F1, https://durabilitycurve.com/claims/survives-a-recheck/#c-F1 (source read 2026-10-26).

    Report an error
  29. F6 Executed Read

    Anchoring original's strength, via FReD's quotation of the paper (full text not obtained): t(102) = 7.00, p < .01 as reported; the statistic itself implies a p far below the reported threshold

    Result: There was a highly significant difference between individual averages of the transformed estimates made with high and low anchors, t(102) = 7.00, p < .01.

    FReD claim_text_o for doi 10.1177/01461672952111004 (secondary: a database quoting the paper)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (FReD claim_text_o for doi 10.1177/01461672952111004 (secondary: a database quoting the paper)). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F6, https://durabilitycurve.com/claims/survives-a-recheck/#c-F6 (source read 2026-10-26).

    Report an error
  30. F7 Executed Read

    Rule of thumb for the card's statistic fallback: with at least 30 degrees of freedom, t above 3.646 (or F(1, df) above 13.29) means two-tailed p below .001; critical t is 3.551 at 40 df, 3.46 at 60 df, 3.3 at 1000 df

    Result: 30 3.646 F(1,df)= 13.29 / 40 3.551 F(1,df)= 12.61 / 60 3.46 F(1,df)= 11.97 / 1000 3.3 F(1,df)= 10.89

    own run: scipy.stats.t.ppf(1 - 0.0005, df), scipy 1.13.1

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run: scipy.stats.t.ppf(1 - 0.0005, df), scipy 1.13.1). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F7, https://durabilitycurve.com/claims/survives-a-recheck/#c-F7 (source read 2026-10-26).

    Report an error
  31. F12 Executed Read

    A no-sign failure from the 2015 sample: LoBue & DeLoache 2008 (attention to snakes and other fear-relevant images) was a strong pass (p = 0.000665379), a main effect, rated 2 for surprise (not surprising); a re-run of the same size (48) found no reliable effect (p = 0.237041962; replicators' verdict: no)

    Result: Study Title (O) = Detecting the snake in the grass: attention to fear-relevant stimuli by adults and young children. / Type of effect (O) = main effect / Surprising result (O) = 2 / T_pval_USE..O. = 0.000665379 / N (O) = 48 / N (R) = 48.0 / T_pval_USE..R. = 0.237041962 / Replicate (R) = no

    OSC 2015 public data, rpp_data.csv, Study Num 161 (OSF 55ddc2248c5e4a3ef7c0e4b7); own read of the row

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (OSC 2015 public data, rpp_data.csv, Study Num 161 (OSF 55ddc2248c5e4a3ef7c0e4b7); own read of the row). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F12, https://durabilitycurve.com/claims/survives-a-recheck/#c-F12 (source read 2026-10-26).

    Report an error
  32. F13 Executed Read

    Our held-out test: on 27 SSRP and Many Labs 2 findings not in the 2015 sample, with lay readers' doubt as the surprise sign, act held 5/5, try small 9/18 (50%), wait 0/4; act - wait +100 points, Fisher p = 0.00794

    Result: Held-out card test · 27 findings (SSRP + Many Labs 2), signs frozen before outcomes / by action: act 5/5 (100%) · try small 9/18 (50%) · wait 0/4 (0%) / act - wait +100 points, Fisher p = 0.00794, monotone = True -> CARD SORTS ON HELD-OUT CLAIMS

    heldout_card_test.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); Hoogeveen et al. 2020 data (osf.io/wg9s3) · durabilitycurve.com

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (heldout_card_test.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); Hoogeveen et al. 2020 data (osf.io/wg9s3)), https://durabilitycurve.com/methods/before-you-trust-a-study/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F13, https://durabilitycurve.com/claims/survives-a-recheck/#c-F13 (source read 2026-10-26).

    Report an error
  33. F14 Executed Read

    Held-out comparisons: base rate held 52% (14/27); p-value cue alone 67%; lay majority alone 67%; card act-vs-wait 100% on 9 of 27 decided

    Result: accuracy, p-value only (strong -> holds): 67% (n = 27) / accuracy, lay majority only (most predict it replicates -> holds): 67% (n = 27) / accuracy, card act vs wait (try small abstains): 100% on 9 of 27 decided / base rate held: 52% (14/27)

    as F13

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (as F13). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F14, https://durabilitycurve.com/claims/survives-a-recheck/#c-F14 (source read 2026-10-26).

    Report an error
  34. F15 Executed Read

    Our small-group check: with sign 4 judged by one lay rater instead of the crowd (F13), the card still sorted the held-out findings: act held 100%, try small 52%, wait 32%; median act - wait +70; the held-out rule held in 82% of draws (88% with three raters, 90% with five)

    Result: k = 1 panel: agree w/ crowd 67% · act 100% (avg 3.6) · try 52% · wait 32% (avg 9.0) · median gap +70 [+42, +100] · rule holds 82% / -> SMALL GROUP KEEPS THE SORT (k = 1) / k = 3 panel: ... rule holds 88% / k = 5 panel: ... rule holds 90%

    smallgroup_check.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); Hoogeveen et al. 2020 data (osf.io/wg9s3) · durabilitycurve.com

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (smallgroup_check.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); Hoogeveen et al. 2020 data (osf.io/wg9s3)), https://durabilitycurve.com/methods/before-you-trust-a-study/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F15, https://durabilitycurve.com/claims/survives-a-recheck/#c-F15 (source read 2026-10-27).

    Report an error
  35. F17 Executed Read

    Exact 95% intervals for the held-out cells: act 5 of 5, 47.8% to 100.0%; wait 0 of 4, 0.0% to 60.2%; try small 9 of 18, 26.0% to 74.0%

    Result: 5/5: exact 95% [47.8%, 100.0%] / 0/4: exact 95% [0.0%, 60.2%] / 9/18: exact 95% [26.0%, 74.0%]

    Clopper–Pearson intervals computed with scipy.stats.beta from F13's counts

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (Clopper–Pearson intervals computed with scipy.stats.beta from F13's counts). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F17, https://durabilitycurve.com/claims/survives-a-recheck/#c-F17 (source read 2026-10-27).

    Report an error
  36. A01h Verified Read

    In the 2015 project, each original's "surprising result" score was the average rating of at least 3 volunteers, naive to the results, on a 1 to 6 scale

    "Average response of at least 3 volunteers, naive to results, who report how suprising they found the key effect to be: 1: not at all, 2: slightly, 3: somewhat, 4: moderately, 5: very, 6: extremely" [sic: "suprising"]

    OSC 2015 public data codebook, rpp_data_codebook.csv (OSF 55ddc2248c5e4a3ef7c0e4b7) · row "Surprising result (O)", column Description

    Link
    Cite

    OSC 2015 public data codebook, rpp_data_codebook.csv (OSF 55ddc2248c5e4a3ef7c0e4b7), row "Surprising result (O)", column Description. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row A01h, https://durabilitycurve.com/claims/survives-a-recheck/#c-A01h (source read 2026-10-27).

    Report an error
  37. CR1a Verified Read

    Inputs include the original study's sample size, effect size, p-value, and whether the tested effect is an interaction

    "among these features we have sample size, the effect size and p-value originally measured, and whether a finding is an effect of one variable or an interaction between multiple variables"

    Altmejd, Dreber, Forsell, Huber, Imai, Johannesson, Kirchler, Nave & Camerer (2019), "Predicting the replicability of social science lab experiments", PLOS ONE 14(12): e0225826. Publisher PDF + JATS XML (open access) · § 1 Introduction, PDF p. 2 (PDF loose match: "vari- able" breaks across a line; XML strict) · doi.org

    Link
    Cite

    Altmejd, Dreber, Forsell, Huber, Imai, Johannesson, Kirchler, Nave & Camerer (2019), "Predicting the replicability of social science lab experiments", PLOS ONE 14(12): e0225826. Publisher PDF + JATS XML (open access), § 1 Introduction, PDF p. 2 (PDF loose match: "vari- able" breaks across a line; XML strict), https://doi.org/10.1371/journal.pone.0225826. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row CR1a, https://durabilitycurve.com/claims/survives-a-recheck/#c-CR1a (source read 2026-10-27).

    Report an error
  38. CR3a Verified Read

    They proposed moving the default significance threshold from 0.05 to 0.005 for claims of new discoveries

    "We propose to change the default P-value threshold for statistical significance from 0.05 to 0.005 for claims of new discoveries."

    Benjamin et al. (2018), "Redefine statistical significance", Nature Human Behaviour 2: 6–10 (online 1 Sep 2017). Publisher page, abstract only: the full text is paywalled · Abstract (loose match: the page's markup renders "P -value") · nature.com

    Link
    Cite

    Benjamin et al. (2018), "Redefine statistical significance", Nature Human Behaviour 2: 6–10 (online 1 Sep 2017). Publisher page, abstract only: the full text is paywalled, Abstract (loose match: the page's markup renders "P -value"), https://www.nature.com/articles/s41562-017-0189-z. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row CR3a, https://durabilitycurve.com/claims/survives-a-recheck/#c-CR3a (source read 2026-10-27).

    Report an error
  39. F20 Executed Read

    Our SCORE test, pre-registered: on SCORE's re-runs (331 claims, 212 papers), originals at p < .001 replicated 66% against 34% for p between .01 and .05, a gap of +32 points (+16 to +46), each paper weighted once: the cue held on the large batch our falsifier named

    Result: A_S_vs_W: 66% (152 claims, 96 papers) vs 34% (63, 48) · +32 [+16, +46] -> CUE HOLDS / SCORE cue test · mode real · 331 claims from 212 papers

    score_cue_test.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); SCORE data, OSF dtzx4 (CC0) · durabilitycurve.com

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (score_cue_test.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); SCORE data, OSF dtzx4 (CC0)), https://durabilitycurve.com/methods/before-you-trust-a-study/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F20, https://durabilitycurve.com/claims/survives-a-recheck/#c-F20 (source read 2026-10-27).

    Report an error
  40. F21 Executed Read

    SCORE, the card's first sign as a reader applies it: strong passes replicated 65% against 38% for weak passes, +27 (+14 to +39); psychology and health only, p-band test +25 (+1 to +47); middle band .001 to .01 held 22%

    Result: B_card_strong_vs_weak: 65% (175 claims, 113 papers) vs 38% (144, 105) · +27 [+14, +39] -> CUE HOLDS / [psychology and health (secondary)] A_S_vs_W: 67% (54 claims, 45 papers) vs 42% (33, 23) · +25 [+1, +47] / A middle band (.001-.01) rate: 22%

    as F20

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (as F20). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F21, https://durabilitycurve.com/claims/survives-a-recheck/#c-F21 (source read 2026-10-27).

    Report an error
  41. F22 Executed Read

    SCORE, sample size: the biggest third of original samples replicated 52% against 40% for the smallest third, +12 (−5 to +27), not detectable, so the sample-size falsifier did not fire

    Result: C_size_top_vs_bottom: 52% (98 claims, 57 papers) vs 40% (98, 72) · +12 [-5, +27] -> NO DETECTABLE ADVANTAGE FOR BIG SAMPLES / falsifier, sample size: not fired

    as F20

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (as F20). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F22, https://durabilitycurve.com/claims/survives-a-recheck/#c-F22 (source read 2026-10-27).

    Report an error
  42. W1a Verified Read

    SCORE (Tyner et al., Nature 2026) re-ran 274 claims of positive results from 164 papers published 2009 to 2018 in 54 social- and behavioural-science journals; median power 99.6% to detect the original effect; 151 of 274 replicated (55.1%)

    "Here we attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioural sciences. Replications were high powered on average to detect the original effect size (median of 99.6%)" … "Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1%"

    Tyner et al. (2026), "Investigating the replicability of the social and behavioural sciences", Nature, DOI 10.1038/s41586-025-10078-y; abstract from the Nature page · Abstract · nature.com

    Link
    Cite

    Tyner et al. (2026), "Investigating the replicability of the social and behavioural sciences", Nature, DOI 10.1038/s41586-025-10078-y; abstract from the Nature page, Abstract, https://www.nature.com/articles/s41586-025-10078-y. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row W1a, https://durabilitycurve.com/claims/survives-a-recheck/#c-W1a (source read 2026-10-27).

    Report an error
  43. W1b Verified Read

    SCORE's papers came from a stratified random sample, but which claims were re-run depended on feasibility, so the re-run set is not itself random

    "random sampling is lost because selection is based on feasibility, available resources, and available expertise."

    Tyner et al. (2026), accepted manuscript (JYX repository) · Methods, "Sampling and selecting claims", PDF p. 18 · jyx.jyu.fi

    Link
    Cite

    Tyner et al. (2026), accepted manuscript (JYX repository), Methods, "Sampling and selecting claims", PDF p. 18, https://jyx.jyu.fi/handle/123456789/109872. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row W1b, https://durabilitycurve.com/claims/survives-a-recheck/#c-W1b (source read 2026-10-27).

    Report an error
  44. W1c Verified Read

    Colquhoun (2014) argued that to keep false discoveries below 5% you should insist on p ≤ 0.001

    "It is concluded that if you wish to keep your false discovery rate below 5%, you need to use a three-sigma rule, or to insist on p≤0.001."

    Colquhoun (2014), "An investigation of the false discovery rate and the misinterpretation of p-values", Royal Society Open Science 1:140216 · Abstract · pmc.ncbi.nlm.nih.gov

    Link
    Cite

    Colquhoun (2014), "An investigation of the false discovery rate and the misinterpretation of p-values", Royal Society Open Science 1:140216, Abstract, https://pmc.ncbi.nlm.nih.gov/articles/PMC4448847/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row W1c, https://durabilitycurve.com/claims/survives-a-recheck/#c-W1c (source read 2026-10-27).

    Report an error
  45. W1d Verified Read

    Gordon et al. (2021): across findings from four replication projects (RPP, EERP, ML2, SSRP), originals at p ≤ 0.005 replicated about 74% against 28% above it

    "While the replication rate for findings with p < = 0.005 is about 74%, the replication rate for findings with p > 0.005 drops to 28%."

    Gordon, Viganola, Dreber, Johannesson & Pfeiffer (2021), PLOS ONE 16: e0248780 (RPP 40, EERP 18, ML2 24, SSRP 21) · § 3.5 "P value analysis" · pmc.ncbi.nlm.nih.gov

    Link
    Cite

    Gordon, Viganola, Dreber, Johannesson & Pfeiffer (2021), PLOS ONE 16: e0248780 (RPP 40, EERP 18, ML2 24, SSRP 21), § 3.5 "P value analysis", https://pmc.ncbi.nlm.nih.gov/articles/PMC8046229/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row W1d, https://durabilitycurve.com/claims/survives-a-recheck/#c-W1d (source read 2026-10-27).

    Report an error
  46. W1f Verified Read

    Spencer Greenberg, in a podcast interview: in 325 replications, originals at p ≤ 0.01 replicated about 72%, against about 48% above 0.01

    "we looked at 325 studies that were replications of existing studies." … "when the original study p-value was at most 0.01 — so it was 0.01 or less — about 72% replicated." … "when the p-value is greater than 0.01, only about 48% replicated."

    80,000 Hours podcast episode #147, Spencer Greenberg, transcript · Transcript § "How often papers replicate with different p-values [00:43:31]" · 80000hours.org

    Link
    Cite

    80,000 Hours podcast episode #147, Spencer Greenberg, transcript, Transcript § "How often papers replicate with different p-values [00:43:31]", https://80000hours.org/podcast/episodes/spencer-greenberg-stopping-valueless-papers/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row W1f, https://durabilitycurve.com/claims/survives-a-recheck/#c-W1f (source read 2026-10-27).

    Report an error
  47. F26 Executed Read

    Forced to call every one of the 27 held-out findings act or wait, the card scores 18 of 27, the same share as the p-value alone: its act calls (5 of 5) and wait calls (4 of 4) are right, and the 18 try-small findings split 9 held, 9 not, so either forced call gets 9

    Result: forced: act 5 right + wait 4 right + 9 of 18 try-small either way = 18 of 27 = 66.7

    arithmetic on F13's counts

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (arithmetic on F13's counts). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F26, https://durabilitycurve.com/claims/survives-a-recheck/#c-F26 (source read 2026-10-27).

    Report an error
  48. F28 Executed Read

    EXPLORATORY: in SCORE's new-data replications only (146 claims, 114 papers), originals at p < .001 replicated 67% against 32% for .01 to .05, +35 (+13 to +55)

    Result: EXPLORATORY · new-data replications only · 146 claims, 114 papers / A S vs W: 67% (51, 46) vs 32% (39, 32) · +35 [+13, +55] -> CUE HOLDS

    explore_score_newdata.py (exploratory, added after the locked run)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (explore_score_newdata.py (exploratory, added after the locked run)). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F28, https://durabilitycurve.com/claims/survives-a-recheck/#c-F28 (source read 2026-10-27).

    Report an error
  49. F29 Executed Read

    In the held-out test, all 5 strong passes were act calls; the other two signs split the 22 weak passes: 9 of 18 with no other sign held, 0 of 4 with one did

    Result: weak passes 22 by action {'try small': 18, 'wait': 4} strong passes 5 strong actions {'act': 5} / by action: act 5/5 (100%) · try small 9/18 (50%) · wait 0/4 (0%)

    heldout_signs.csv and F13's counts

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (heldout_signs.csv and F13's counts). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F29, https://durabilitycurve.com/claims/survives-a-recheck/#c-F29 (source read 2026-10-27).

    Report an error
  50. W1g Verified Read

    A SCORE companion paper (Abatayo et al., MetaArXiv, 2026) reports the original p-value's correlation with replicability as -.11 (-.26 to .04) by paper and -.18 (-.29 to -.05) by claim; the Nature paper defers to it

    "Original p-value" … "-.11" … "[-.26, .04]" (Table 3) … "Original p-value" … "-.18" … "[-.29, -.05]" (Table S19, by claim) … "We did not explore correlations with replication outcomes"

    Abatayo et al. (2026), "Assessments of Credibility in the Social and Behavioral Sciences", MetaArXiv 7u58q_v1; Tyner et al. (2026) accepted manuscript for the deferral · Table 3, p. 9; Table S19; Nature accepted manuscript · osf.io

    Link
    Cite

    Abatayo et al. (2026), "Assessments of Credibility in the Social and Behavioral Sciences", MetaArXiv 7u58q_v1; Tyner et al. (2026) accepted manuscript for the deferral, Table 3, p. 9; Table S19; Nature accepted manuscript, https://osf.io/preprints/metaarxiv/7u58q_v1. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row W1g, https://durabilitycurve.com/claims/survives-a-recheck/#c-W1g (source read 2026-10-27).

    Report an error
  51. R4d Verified Read

    Alvaro de Menard (2020), arguing from replication-forecast data, put replication below 50% for originals with p > .001 and about 10 percentage points lower for interaction effects

    "The base replication rate for studies with p>.001 is below 50%." … "A paper whose main finding is an interaction effect is about 10 percentage points less likely to replicate."

    de Menard (2020), "What's Wrong with Social Science and How to Fix It", Fantastic Anachronism, 11 Sep 2020 (basis: Replication Markets forecasts) · Sections "What To Do" and "Interaction Effects Bad" · fantasticanachronism.com

    Link
    Cite

    de Menard (2020), "What's Wrong with Social Science and How to Fix It", Fantastic Anachronism, 11 Sep 2020 (basis: Replication Markets forecasts), Sections "What To Do" and "Interaction Effects Bad", https://fantasticanachronism.com/2020/09/11/whats-wrong-with-social-science-and-how-to-fix-it/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R4d, https://durabilitycurve.com/claims/survives-a-recheck/#c-R4d (source read 2026-10-27).

    Report an error
  52. R4g Verified Read

    SCORE's 274 claims split into 128 new-data and 146 secondary-data replications; secondary-data replications used existing data independent of the original

    "128 / 128" … "146 / 146" … "Secondary data replications are those that used existing data that was independent of the original investigation." (274 = 128 + 146)

    Tyner et al. (2026), accepted manuscript, Table 4 · Table 4 and note · jyx.jyu.fi

    Link
    Cite

    Tyner et al. (2026), accepted manuscript, Table 4, Table 4 and note, https://jyx.jyu.fi/handle/123456789/109872. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R4g, https://durabilitycurve.com/claims/survives-a-recheck/#c-R4g (source read 2026-10-27).

    Report an error
  53. R5a Verified Read

    Vu & Faridani forecast replication from each original's p-value, together with an estimate of how large effects in that field tend to be, in three projects: economics (Camerer 2016), psychology (OSC 2015) and social science (Camerer 2018)

    "We estimate the predictive power curve in three empirical literatures with systematic replication evidence: experimental economics (Camerer et al., 2016), psychology (Open Science Collaboration, 2015), and experimental social science (Camerer et al., 2018)."

    Vu & Faridani (2026), "How Replicable Are Statistically Significant Findings?", arXiv 2608.23257 (working paper) · p. 2 · arxiv.org

    Link
    Cite

    Vu & Faridani (2026), "How Replicable Are Statistically Significant Findings?", arXiv 2608.23257 (working paper), p. 2, https://arxiv.org/abs/2608.23257. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R5a, https://durabilitycurve.com/claims/survives-a-recheck/#c-R5a (source read 2026-10-27).

    Report an error
  54. R5b Verified Read

    On the 70 findings that also had prediction-market forecasts, the p-only forecasts' average squared error (Brier score) was 0.216 against the markets' 0.220, pooled and weighted

    "the validation exercise uses the common sample of 70 studies for which both predictions are available" … "In the weighted pooled sample, predictive power slightly outperforms prediction markets, with Brier scores of 0.216 and 0.220, respectively."

    Vu & Faridani (2026), "How Replicable Are Statistically Significant Findings?", arXiv 2608.23257 (working paper) · pp. 11–12, Table 1 (psychology alone 0.243 and 0.243) · arxiv.org

    Link
    Cite

    Vu & Faridani (2026), "How Replicable Are Statistically Significant Findings?", arXiv 2608.23257 (working paper), pp. 11–12, Table 1 (psychology alone 0.243 and 0.243), https://arxiv.org/abs/2608.23257. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R5b, https://durabilitycurve.com/claims/survives-a-recheck/#c-R5b (source read 2026-10-27).

    Report an error
  55. R5d Verified Read

    In psychology, a finding at p = 0.05 has an expected replication probability of 0.22 in a same-size re-run; across the three fields, 0.1 to 0.25

    "Evaluating the curve at the 5% significance threshold, we find that replicability ranges between 0.1 and 0.25 across fields." … Figure 2, psychology panel: "(0.05, 0.22)"

    Vu & Faridani (2026), "How Replicable Are Statistically Significant Findings?", arXiv 2608.23257 (working paper) · p. 13 and Figure 2, p. 14 · arxiv.org

    Link
    Cite

    Vu & Faridani (2026), "How Replicable Are Statistically Significant Findings?", arXiv 2608.23257 (working paper), p. 13 and Figure 2, p. 14, https://arxiv.org/abs/2608.23257. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R5d, https://durabilitycurve.com/claims/survives-a-recheck/#c-R5d (source read 2026-10-27).

    Report an error
  56. R5f Verified Read

    Carter & McCullough re-analysed the 198 published tests in Hagger et al.'s 2010 meta-analysis

    "Hagger et al. (2010) reported a meta-analysis of the 198 published tests of the depletion effect"

    Carter & McCullough (2014), Frontiers in Psychology 5:823 · Introduction · frontiersin.org

    Link
    Cite

    Carter & McCullough (2014), Frontiers in Psychology 5:823, Introduction, https://www.frontiersin.org/articles/10.3389/fpsyg.2014.00823/full. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R5f, https://durabilitycurve.com/claims/survives-a-recheck/#c-R5f (source read 2026-10-27).

    Report an error
  57. R5g Verified Read

    Corrected for small-study effects (publication bias among them), the depletion effect was not meaningfully different from zero

    "an indication that the depletion effect is actually no different from zero" … "after controlling for the influence of small-study effects, our results do not support the claim that the depletion effect is meaningfully different from zero"

    Carter & McCullough (2014), Frontiers in Psychology 5:823 · Abstract; Discussion · frontiersin.org

    Link
    Cite

    Carter & McCullough (2014), Frontiers in Psychology 5:823, Abstract; Discussion, https://www.frontiersin.org/articles/10.3389/fpsyg.2014.00823/full. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R5g, https://durabilitycurve.com/claims/survives-a-recheck/#c-R5g (source read 2026-10-27).

    Report an error
  58. R5h Verified Read

    Carter & McCullough turned to the meta-analysis after failing to produce the depletion effect in their own work

    "In response to our inability to produce the basic depletion effect in our own work, we turned to Hagger et al.'s (2010) meta-analysis"

    Carter & McCullough (2014), Frontiers in Psychology 5:823 · Introduction · frontiersin.org

    Link
    Cite

    Carter & McCullough (2014), Frontiers in Psychology 5:823, Introduction, https://www.frontiersin.org/articles/10.3389/fpsyg.2014.00823/full. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R5h, https://durabilitycurve.com/claims/survives-a-recheck/#c-R5h (source read 2026-10-27).

    Report an error
  59. R5j Verified Read

    Falkovich's lay rules include anti-significance (a result at p = 0.049 is suspect) and Taleb's grandma (an effect that sounds implausible probably is not real)

    "If a study has p=0.049 it is fake." … "If the purported effect sounds implausible, it is."

    Jacob Falkovich (7 Sep 2018), "The Scent of Bad Psychology", Putanumonit · Rules of Anti-Significance and Taleb's Grandma · putanumonit.com

    Link
    Cite

    Jacob Falkovich (7 Sep 2018), "The Scent of Bad Psychology", Putanumonit, Rules of Anti-Significance and Taleb's Grandma, https://putanumonit.com/2018/09/07/the-scent-of-bad-psychology/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R5j, https://durabilitycurve.com/claims/survives-a-recheck/#c-R5j (source read 2026-10-27).

    Report an error
  60. R5k Verified Read

    Bailey's rules of thumb include trusting overall effects over interaction or subgroup effects, extraordinary claims needing extraordinary evidence, and the real effect being half what the paper claims; they do not use p-values

    "Overall effects are more trustworthy than interaction or subgroup effects" … "extraordinary claims require extraordinary evidence" … "The real effect size is half what the paper claims"

    James Bailey (2 Apr 2026), "How Much To Trust Research Papers? My Rules Of Thumb", Economist Writing Every Day · rules 2, 6 and 8 · economistwritingeveryday.com

    Link
    Cite

    James Bailey (2 Apr 2026), "How Much To Trust Research Papers? My Rules Of Thumb", Economist Writing Every Day, rules 2, 6 and 8, https://economistwritingeveryday.com/2026/04/02/how-much-to-trust-research-papers-my-rules-of-thumb/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R5k, https://durabilitycurve.com/claims/survives-a-recheck/#c-R5k (source read 2026-10-27).

    Report an error
  61. R6a Reported Read

    Lindsay's "troubling trio" (2015): low statistical power, a surprising result, and a p-value only slightly lower than .05

    "Described as a combination of low statistical power, a surprising result, and a p-value only slightly lower than .05." (FORRT glossary entry, citing Lindsay 2015)

    Lindsay (2015), "Replication in Psychological Science", Psychological Science 26(12), 1827–1832 · FORRT glossary · forrt.org

    Link
    Cite

    Lindsay (2015), "Replication in Psychological Science", Psychological Science 26(12), 1827–1832, FORRT glossary, https://forrt.org/glossary/english/the_troubling_trio/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R6a, https://durabilitycurve.com/claims/survives-a-recheck/#c-R6a (source read 2026-10-27).

    Report an error
  62. R6b Verified Read

    SCORE reports thirteen measures of replication success, whose estimates range from 28.6% to 74.8% (median 49.3%)

    "Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median of 49.3%)."

    Lakens (3 May 2026), "There are 13 ways to analyse a replication study, but only one of them is coherent", The 20% Statistician · quoting Tyner et al. 2026 · daniellakens.blogspot.com

    Link
    Cite

    Lakens (3 May 2026), "There are 13 ways to analyse a replication study, but only one of them is coherent", The 20% Statistician, quoting Tyner et al. 2026, https://daniellakens.blogspot.com/2026/05/there-are-13-ways-to-analyse.html. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R6b, https://durabilitycurve.com/claims/survives-a-recheck/#c-R6b (source read 2026-10-27).

    Report an error
  63. R7a Verified Read

    The 1998 radish experiment was repeated once, as a school science-fair project, reported in the original paper's footnote; we found no registered re-run of that experiment

    "As this article went to press, we were notified that this experiment had been independently replicated by Timothy J. Howe, of Cole Junior High School in East Greenwich, Rhode Island, for his science fair project"

    Baumeister, Bratslavsky, Muraven & Tice (1998), JPSP 74(5), footnote 2 · footnote 2 · faculty.washington.edu

    Link
    Cite

    Baumeister, Bratslavsky, Muraven & Tice (1998), JPSP 74(5), footnote 2, footnote 2, https://faculty.washington.edu/jdb/345/345%20Articles/Baumeister%20et%20al.%20%281998%29.pdf. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R7a, https://durabilitycurve.com/claims/survives-a-recheck/#c-R7a (source read 2026-10-27).

    Report an error
  64. R7c Reported Read

    A 2026 re-analysis of SCORE's data finds that replication-success methods cannot reliably reach a conclusion at typical sample sizes

    "Replication success methods are unable to reliably draw a conclusion about replication success for typical sample sizes of the original study and replication"

    van Aert, van Assen & López-Nicolás (2026), PLOS ONE 21(9): e0358126 · Abstract · journals.plos.org

    Link
    Cite

    van Aert, van Assen & López-Nicolás (2026), PLOS ONE 21(9): e0358126, Abstract, https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0358126. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R7c, https://durabilitycurve.com/claims/survives-a-recheck/#c-R7c (source read 2026-10-27).

    Report an error
  65. R7e Verified Read

    A pre-registered 2023 study (N = 230) reproduced the retrieval-practice advantage on recall (verbatim) questions, then found no significant difference when the concept-mapping group also memorised (p = 0.854); the authors call the advantage a methodological artifact

    "While we replicated the previous finding that retrieval practice in combination with additional memorizing produces more learning than concept mapping without additional memorizing, this advantage of retrieval practice over concept mapping vanished when participants in the concept mapping condition, too, memorized" … "performance did not differ ( p = 0.854) between the RP + AM + AI condition and the CM + AM + AI condition" … "resulting in a final sample size of N = 230" … "followed our preregistration at Open Science Foundation"

    Mayrhofer, Kuhbandner & Frischholz (2023), Frontiers in Psychology 14:1258359 · Abstract; Methods · pmc.ncbi.nlm.nih.gov

    Link
    Cite

    Mayrhofer, Kuhbandner & Frischholz (2023), Frontiers in Psychology 14:1258359, Abstract; Methods, https://pmc.ncbi.nlm.nih.gov/articles/PMC10783554/. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R7e, https://durabilitycurve.com/claims/survives-a-recheck/#c-R7e (source read 2026-10-27).

    Report an error
  66. R7f Verified Read

    Hoogeveen et al. reported that findings their lay participants, as a group, predicted would fail did fail to replicate

    "when participants as a group predicted that a particular effect would fail to be replicated, it indeed failed to be replicated"

    Hoogeveen, Sarafoglou & Wagenmakers (2020), AMPPS · Results · doi.org

    Link
    Cite

    Hoogeveen, Sarafoglou & Wagenmakers (2020), AMPPS, Results, https://doi.org/10.1177/2515245920919667. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row R7f, https://durabilitycurve.com/claims/survives-a-recheck/#c-R7f (source read 2026-10-27).

    Report an error
  67. F30 Executed Read

    In the held-out test, the card's four wait calls are exactly the four findings with the lowest lay share predicting replication (Lee & Schwarz 0.172, Gervais & Norenzayan 0.299, Shah et al. 0.368, Tversky & Gati 0.468); all four are weak passes, one has an interaction

    Result: Lee & Schwarz (2010) lay 0.172 wait · Gervais & Norenzayan (2012) lay 0.299 wait · Shah et al. (2012) lay 0.368 wait · Tversky & Gati (1978) lay 0.468 wait · next lowest Alter et al. (2007) lay 0.504 try small; doubted True only on the four; interaction True on one

    heldout_signs.csv

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (heldout_signs.csv). Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row F30, https://durabilitycurve.com/claims/survives-a-recheck/#c-F30 (source read 2026-10-27).

    Report an error

Cite and reuse

A single claim
Use the row's Cite button. It names the original source first, then this row, which carries its own link.
This ledger
Floyd, Harry (2026). Claim Ledger: Before You Trust a Study, Check How Strongly It Passed. The Durability Curve. https://durabilitycurve.com/claims/survives-a-recheck/
The essay
Floyd, Harry (2026). Before You Trust a Study, Check How Strongly It Passed. The Durability Curve. https://durabilitycurve.com/blog/survives-a-recheck/

The ledger data is licensed CC BY 4.0: reuse it, quote it or train on it, with credit to The Durability Curve and a link. Read the licence. Machine copies: JSON, CSV, Markdown.

The newsletter · Free

Subscribe to get each new piece in your inbox, with something to use in every one.

Several essays a week, plus Notes through the week.

Signup completes on Substack. Unsubscribe from any email.