---
title: "Claim Ledger: Before You Trust a Study, Check How Strongly It Passed"
description: "We fixed the rule before opening each claim's re-check result: strong first passes held up 66% of the time, narrow ones 34%."
author: "Harry Floyd"
publication: "The Durability Curve"
canonical: "https://durabilitycurve.com/claims/survives-a-recheck/"
essay: "https://durabilitycurve.com/blog/survives-a-recheck/"
published: "2026-11-01"
ledger_date: "2026-10-27"
law: "Law II"
claims: 67
removed_before_publication: 0
---

# Claim Ledger: Before You Trust a Study, Check How Strongly It Passed

*We fixed the rule before opening each claim's re-check result: strong first passes held up 66% of the time, narrow ones 34%.*

Essay: https://durabilitycurve.com/blog/survives-a-recheck/  
Ledger (canonical, cite this): https://durabilitycurve.com/claims/survives-a-recheck/  
Published 2026-11-01 · last verified 2026-10-27

## What the essay claims

Before: a study that passed its test, especially a large one, is a finding I can act on. After: how strongly a study's first result passed is a clue to whether it will hold up. In a test fixed before we opened the results, on SCORE's public data, strong first passes (p below .001) held up 66% of the time and narrow ones (.01 to .05) 34%, with each paper weighted equally. Sample size did less: the largest third of original samples held up 12 points more often than the smallest, with an interval from −5 to 27, so this test cannot show that size matters.

## The claim ladder

What the ladder does not reach: health and money advice; a verdict on any single finding (the card gives odds for groups of findings); whether a finding that held is big enough to matter; the size condition in step 1 (a re-run at least as large as the original), a design choice not yet tested.

| # | Claim | Whose behaviour it measures | Status / scope |
|---|---|---|---|
| 1 | Strong first passes held up 66%, narrow ones 34%, a gap of 32 points (16 to 46), each paper weighted equally | Ours, SCORE's public data (331 claims, 212 papers), pre-registered | Verified by the locked run. Papers drawn at random from social- and behavioural-science journals; which claims were re-checked depended on feasibility |
| 2 | Passes between .001 and .01 held up 22%, no more often than narrow ones | Ours, the same run | From the locked run; a secondary figure the plan reported without a verdict |
| 3 | The largest third of original samples held up 12 points more often than the smallest, interval −5 to 27 | Ours, the same run | Not detectable; under the plan's rule the sample-size falsifier did not fire |
| 4 | On re-checks that collected new data, the gap was as large | Ours, SCORE's new-data re-checks | Exploratory, a split added afterwards |
| 5 | On re-runs researchers chose to do: 58% against 38% | Ours, FReD, pre-registered | Held up if the re-run's own report said so; mixed results counted as failures |
| 6 | 20 of 32 originals below .001 held up, against 2 of 11 above .04 | OSC 2015 | Verified |
| 7 | Interactions held up 8 times in 37, main or simple effects 23 in 49; results rated surprising held up less often | OSC 2015 | Verified (OSC data only) |
| 8 | Using each original's p-value, together with an estimate of how large effects in that field tend to be, forecasts about as accurate as scientists' prediction markets; a psychology finding at p = .05 has about a one-in-five chance that a same-size re-run is significant | Vu & Faridani (2026, working paper) | Verified. Scored on the same re-run projects that shaped the model |
| 9 | Lay readers right about 59% of the time from plain descriptions, about 67% when shown the strength of evidence | Hoogeveen, Sarafoglou & Wagenmakers (2020) | Verified |
| 10 | On 27 findings the card was not built from: act 5 of 5 held, wait 0 of 4, try small 9 of 18; forced to call all 27, 67%, the same as the p-value alone | Ours, pre-registered card test on the 2018 Nature/Science re-runs and Many Labs 2 | A look back: outcomes and lay ratings were already published. Small groups (act 48% to 100%, wait 0% to 60%) |
| 11 | With one reader's doubt instead of the crowd's, about 3 in 10 "wait" findings held | Ours, small-group check, pre-registered | A look back on the same 27 findings; signs 3 and 4 are experimental |
| 12 | Willpower: correcting the 198 published tests for signs of publication bias found an indication that the effect was no different from zero (2014); 23 labs on a registered plan found almost nothing (2016) | Carter & McCullough (2014); Hagger et al. (2016) | Verified |
| 13 | Without a qualifying re-run, expect about half the claimed effect | OSC 2015 | Verified; an average over all pairs, and an effect can shrink further or vanish |
| 14 | Six famous claims, scored from their original papers with calls fixed before reading any verdict | Each claim's original paper and registered re-run | An illustration: the author already knew how some had turned out |

## What would make it wrong

The main cue is wrong if the next large batch of re-runs (a new FReD release, or another multi-lab project) shows first results at p < .001 failing as often as those near .05. The sample-size claim is wrong if a pre-registered test on a fresh pool finds the largest original samples holding up at least 10 points more often.

If well-powered re-runs of findings that pass the Re-check fail as often as findings it flags, the check is useless, and we will say so.

## The evidence, row by row

Labels: VERIFIED = checked against the primary source itself by a checker who did not draft the piece, usually with the exact words, where they sit and the date read · EXECUTED = a run or observation done for the piece (code, a query, a count, or a check made live in an app); the claim is what it returned; run records are not published · CHECKED = checked against its source by a separate checker in the older ledger format, which recorded no quote, locator or date · REPORTED = not verified word for word against a primary source (secondary, not openable in full, or supported only in part); the essay words it accordingly · STRUCK = drafted, checked and removed before publication · EXCLUDED = considered and deliberately left out.

### A01b. Replication effects were about half the original: mean r 0.197 vs 0.403

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A01b
- Quote: "Replication effects were half the magnitude of original effects … original study effect sizes ( M = 0.403, SD = 0.188) were reliably larger than replication effect sizes ( M = 0.197, SD = 0.257)"
- Source: OSC 2015, accepted manuscript (OSF), Abstract (PDF p. 1); Results, effect sizes (PDF p. 18)
- URL: https://osf.io/447b3

### A01c. Originals with p < .001: 20 of 32 replicated

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A01c
- Quote: "Almost two thirds (20 of 32, 63%) of original studies with P < … 0.001 had a significant P value in the replication."
- Source: OSC 2015, accepted manuscript (OSF), Results, correlates of reproducibility (PDF pp. 20–21; the sentence crosses a page break)
- URL: https://osf.io/447b3

### A01d. Originals with p > .04: 2 of 11 replicated

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A01d
- Quote: "whereas 6 of 23 (26%) that had a P value between 0.02 < P < 0.04 and 2 of 11 (18%) that had a P value > 0.04 did so"
- Source: OSC 2015, accepted manuscript (OSF), Results (PDF p. 20)
- URL: https://osf.io/447b3

### A01e. Main or simple effects replicated 23/49; interactions 8/37

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A01e
- Quote: "Among original, significant effects, 23 of the 49 (47%) that tested main or simple effects replicated at P < 0.05, but just 8 of the 37 (22%) that tested interaction effects did."
- Source: OSC 2015, accepted manuscript (OSF), Results (PDF p. 20)
- URL: https://osf.io/447b3

### A01f. Rated surprisingness correlated −0.244 (Spearman) with replication significance

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A01f
- Quote: "Surprising original result -0.244 0.102 -0.181 -0.113 -0.241 … Surprising effects were less reproducible"
- Source: OSC 2015, accepted manuscript (OSF), Table 2, first column = "Replications P < .05 in original direction" (PDF p. 12); Results (PDF p. 22)
- URL: https://osf.io/447b3

### A04a. Ego-depletion RRR: 23 labs, N = 2,141, d = 0.04 [−0.07, 0.15]

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A04a
- Quote: "Multiple laboratories (k = 23, total N = 2,141) conducted replications of a standardized ego-depletion protocol … (d = 0.04, 95% CI [-0.07, 0.15]"
- Source: Hagger et al. (2016), Perspectives on Psychological Science 11(4):546–573. Published SAGE PDF from the Univ. Konstanz repository (KOPS), Abstract, p. 546
- URL: https://kops.uni-konstanz.de/server/api/core/bitstreams/d1f20ce9-761f-40e6-aaa8-4992d4744d3c/content

### A07a. 27 findings judged by 233 people without a PhD in psychology: 59% correct

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A07a
- Quote: "Each of 27 high-profile social-science findings was evaluated by 233 people without a Ph.D. in psychology. Results showed that these laypeople predicted replication success with above-chance accuracy (i.e., 59%)."
- Source: Hoogeveen, Sarafoglou & Wagenmakers (2020), AMPPS 3(3):267–285. Final published version (UvA repository), Abstract, p. 267
- URL: https://pure.uva.nl/ws/files/53717212/2515245920919667.pdf

### A07b. 67% correct when shown the strength of evidence

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A07b
- Quote: "when participants were informed about the strength of evidence from the original studies, this boosted their prediction performance to 67%."
- Source: Hoogeveen et al. (2020), published version, Abstract, p. 267; group-level ω .59 [.57, .60] vs .67 [.65, .69] on p. 275
- URL: https://pure.uva.nl/ws/files/53717212/2515245920919667.pdf

### A07c. Two separate groups: 123 people in the description-only condition and 110 in the description-plus-evidence condition; the two accuracy figures in A07a and A07b come from these two groups

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A07c
- Quote: "the final sample consisted of 233 participants, 123 in the description-only condition and 110 in the"
- Source: Hoogeveen et al. (2020), published version, Method, Participants, p. 269
- URL: https://pure.uva.nl/ws/files/53717212/2515245920919667.pdf

### A07d. Evidence strength shown as a Bayes factor plus a verbal label

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A07d
- Quote: "the Bayes factor and its verbal interpretation (e.g., "moderate evidence") were added to the description of each study."
- Source: Hoogeveen et al. (2020), published version, Method, Materials, p. 270; labels follow Jeffreys (1939) as adjusted by Lee & Wagenmakers (2013)
- URL: https://pure.uva.nl/ws/files/53717212/2515245920919667.pdf

### A09c. 215 surveyed psychologists ranked "large sample size" first (M = 3.84)

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A09c
- Quote: "Respondents rated "large sample size" as the most important criterion (M = 3.84), judging it to be significantly more important than the second most important factor, "large effect size" (M = 3.61)" [...] "resulting in a final sample size of 215 psychological scientists"
- Source: Li et al. (2024), publisher PDF, Survey results, p. 10 of 15 (n = 215 on the same page)
- URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0306911&type=printable

### A12c. At 33% power (t tests with 38 degrees of freedom), 57.6% of significant p values are greater than .01

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A12c
- Quote: "t tests with 38 degrees of freedom and powered at 33% will result in 57.6% of significant p values being greater than" [decoded: the text layer reads greater than .0l; the page reads .01]
- Source: Simonsohn et al. (2014), published PDF, p. 538 (the text continues ".01"; the same passage puts 10.4% above .04); note (derived): so 42.4% fall below .01
- URL: https://gwern.net/doc/statistics/meta-analysis/2014-simonsohn.pdf

### A14a. Many Smiles: n = 3,878 across 19 countries; mimicry and voluntary posing both amplified and initiated happiness

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A14a
- Quote: "Data from n = 3,878 participants spanning 19 countries indicated that a facial mimicry and voluntary facial action task could both amplify and initiate feelings of happiness."
- Source: Coles et al. (2022), Nature Human Behaviour 6:1731–1742. Published abstract on the publisher page (full text paywalled), Abstract
- URL: https://www.nature.com/articles/s41562-022-01458-9

### A14b. Pen-in-mouth evidence was less conclusive

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A14b
- Quote: "However, evidence of facial feedback effects was less conclusive when facial feedback was manipulated unobtrusively via a pen-in-mouth task."
- Source: Coles et al. (2022), published abstract, Abstract
- URL: https://www.nature.com/articles/s41562-022-01458-9

### B1o. Ego depletion. Baumeister, Bratslavsky, Muraven & Tice (1998), Experiment 1: hungry students ate radishes while resisting chocolates (or ate chocolates, or no-food control), then persistence on unsolvable puzzles was timed. N = 67. Key test as reported: one-way ANOVA across the three conditions, F(2, 64) = 26.88, p < .001; radish vs chocolate t(44) = 6.03, p < .001. Design: SIMPLE (one manipulation, one outcome: persistence time).

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B1o
- Quote: "Data were collected in individual sessions from 67 introductory psychology students" [...] "A one-way analysis of variance (ANOVA) indicated significant variation among the three conditions, F(2, 64) = 26.88, p < .001." [...] "participants in the radish condition quit sooner on the frustrating task than did participants in either the chocolate condition, t(44) = 6.03, p < .001"
- Source: Baumeister et al. (1998), JPSP 74(5), 1252–1265. Published version (scan with text layer), course copy at the University of Washington., Exp. 1, Method (Participants) PDF p. 3; Results (Persistence) PDF p. 4
- URL: https://faculty.washington.edu/jdb/345/345%20Articles/Baumeister%20et%20al.%20%281998%29.pdf

### B2o. Strack, Martin & Stepper 1988, Study 1: Ninety-two undergraduates held a pen in the lips, the non-dominant hand or the teeth while rating cartoons; reported as a significant linear contrast, t(89) = 1.85, p = .03 (the text layer reads f(89)). Design: SIMPLE

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B2o
- Quote: "Subjects. Ninety-two male and female undergraduates of the University of Illinois participated in the experiment" [...] "The predicted differences were confirmed by a significant linear contrast, f(89) = 1.85, p = .03."
- Source: Strack, Martin & Stepper (1988), JPSP 54(5), 768–777. Published APA version (scan with text layer), third-party copy on GitHub Pages., Study 1, Method PDF p. 3; Results, journal p. 772 (PDF p. 5); note: means 4.32 / 4.77 / 5.14 from the paper's table, not quoted; two-tailed p for t(89) = 1.85 is about .068 (computed), so p = .03 matches a one-tailed test
- URL: https://jhrcook.github.io/wagenmaker-data-analysis/papers/Strack_1988_Inhibiting-and-facilitating-conditions-of-the-human-smile-a-nonobtrusive-test-of-the-facial-feedback-hypothesis.pdf

### B5o. Caruso, Vohs, Baxter & Waytz 2013, Experiment 1: Thirty adults; a faint image of $100 bills behind the instructions; t(28) = 2.12, p = .043, d = 0.80. Design: SIMPLE

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B5o
- Quote: "Thirty adults from a university study pool" [...] "Participants assigned to the money condition saw a faint image of $100 bills in the background of the instruction screen" [...] "As predicted, participants in the money condition more strongly endorsed system justification than did participants in the control condition, t(28) /H110052.12, p /H11005.043, d /H110050.80" [decoded: t(28) = 2.12, p = .043, d = 0.80; the text layer renders = as /H11005]
- Source: Caruso, Vohs, Baxter & Waytz (2013), JEP: General 142(2), 301–306. Published APA version (online-first), copy linked from the Kellogg School research page., Exp. 1, Method and Results, PDF p. 3
- URL: http://static.squarespace.com/static/51e3f4ede4b053e5f0062efd/t/51f7e294e4b03d20c9b472cc/1375199892722/mere-exposure-to-money-increases-endorsement-of-free-market-systems-and-social-inequality%20.pdf

### B6o. Anchoring. Jacowitz & Kahneman (1995): a calibration group estimates quantities; anchored groups first judge whether a high or low anchor (15th or 85th percentile of calibration estimates) is above or below the true value, then estimate. N: not obtained. Test statistic and p: not obtained. Design: SIMPLE (provisional, from the abstract: one manipulation, anchor high vs low, on estimates).

- Status: REPORTED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B6o
- Quote: "Subjects in the anchored condition first judge whether a specified number (the anchor) is higher or lower than the true value before estimating each quantity." [...] "(15th and 85th percentiles in this study)"
- Source: Jacowitz & Kahneman (1995), PSPB 21(11), 1161–1166. Abstract only, reconstructed from the OpenAlex index record., Abstract
- URL: https://api.openalex.org/works/doi:10.1177/01461672952111004

### B7o. Karpicke & Blunt 2011, Experiment 1: Eighty undergraduates; retrieval practice vs concept mapping one week later: d = 1.50, F(1, 38) = 21.63; no p-value reported. Design: SIMPLE

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B7o
- Quote: "Eighty undergraduate students participated in Experiment 1." [...] "the advantage of retrieval practice (M = 0.67) over elaborative studying with concept mapping (M = 0.45) represented about a 50% improvement in long-term retention scores [d = 1.50, F1,38 = 21.63" [decoded: d = 1.50, F(1, 38) = 21.63; the text layer reads F1,38]
- Source: Karpicke & Blunt (2011), Science 331, 772–775. Published Science version, copy on the author's Purdue lab site., Exp. 1, journal p. 772–773 (PDF pp. 2–3); note: key comparison n = 40 (df 38)
- URL: https://learninglab.psych.purdue.edu/downloads/2011/2011_Karpicke_Blunt_Science.pdf

### B8o. Framing flips preferences. Tversky & Kahneman (1981), Problems 1 and 2 ("Asian disease"), separate groups. N = 152 (gain frame) and N = 155 (loss frame). Result as reported: 72% chose the sure option under the gain frame; 22% chose the sure option under the loss frame. No test statistic or p-value is reported. Design: SIMPLE (one manipulation, frame; one outcome, choice).

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B8o
- Quote: "Problem 1 [N = 152]: Imagine that the U.S. is preparing for the outbreak of an unusual Asian disease" [...] "If Program A is adopted, 200 people will be saved. [72 percent]" [...] "Problem 2 [N = 155]: If Program C is adopted 400 people will die. [22 percent]" [...] "The change is accompanied by a pronounced shift from risk aversion to risk taking."
- Source: Tversky & Kahneman (1981), Science 211(4481), 453–458. Published version (JSTOR scan with text layer), copy on gwern.net., Problems 1–2, journal p. 453 (PDF p. 1)
- URL: https://gwern.net/doc/psychology/1981-tversky.pdf

### B1r. Hagger, Chatzisarantis et al. (2016), Registered Replication Report of ego depletion. Qualifies: yes. Pre-registered: yes. Labs: 23. N = 2,141. Author overlap: none (Baumeister is listed as a protocol vetter, not an author). Task differs from B1o: depletion by a letter-"e" task (adapted from Baumeister et al. 1998, Study 4), outcome the multi-source interference task (MSIT); the radish task was judged too experimenter-dependent for many labs. Headline: d = 0.04, 95% CI [−0.07, 0.15].

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B1r
- Quote: "Participating labs preregistered their implementation plan on the Open Science Framework and conducted independent replications." [...] "Multiple laboratories (k = 23, total N = 2,141) conducted replications of a standardized ego-depletion protocol based on a sequential-task paradigm by Sripada et al." [...] "Meta-analysis of the studies revealed that the size of the ego-depletion effect was small with 95% confidence intervals (CIs) that encompassed zero (d = 0.04, 95% CI [−0.07, 0.15]." [...] "Protocol vetted by: Chandra Sripada, Daniel Kessler, Roy Baumeister" [...] "required participants to taste radishes and resist cookies (Baumeister et al., 1998, Study 1). This task would require extensive experimenter involvement in its delivery" [...] "The first task was a modified version of the letter “e” task (Baumeister et al., 1998, Study 4)"
- Source: Hagger et al. (2016), Perspectives on Psychological Science 11(4), 546–573. Published SAGE PDF in the University of Konstanz repository (KOPS)., Title block and Abstract, journal p. 546; Method, pp. 548–550
- URL: https://kops.uni-konstanz.de/server/api/core/bitstreams/d1f20ce9-761f-40e6-aaa8-4992d4744d3c/content

### B2r. Wagenmakers et al. 2016 RRR: 17 independent direct replications of Strack Study 1 on one vetted protocol; smile minus pout rating difference 0.03, 95% CI -0.11 to 0.16; Strack supplied materials and early feedback and declined to review the final protocol; the neutral hand condition was omitted. Qualifies: yes

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B2r
- Quote: "This Registered Replication Report describes the results of 17 independent direct replications of Study 1 from Strack et al. (1988), all of which followed the same vetted protocol." [...] "Data and registered protocols: https://osf.io/pkd65/" [...] "Our meta-analysis revealed a rating difference of 0.03 units with a 95% confidence interval ranging from −0.11 to 0.16." [...] "Fritz Strack provided the original materials as well as valuable feedback and constructive suggestions during the early stages of protocol development by the lead lab." [...] "Although he declined to review the final protocol" [...] "we omitted the neutral “nondominant hand condition” from the design"
- Source: Wagenmakers et al. (2016), Perspectives on Psychological Science 11(6), 917–928. Published SAGE PDF in the Maynooth University repository., Title block and Abstract, journal p. 917; pp. 918–919; Table 1, p. 921; note (our sums of Table 1): 1,894 analysed, 2,533 tested
- URL: https://mural.maynoothuniversity.ie/id/eprint/15647/1/1745691616674458.pdf

### B2s. Coles et al., Many Smiles Collaboration (2022), multi-site adversarial collaboration: published abstract and author list. Qualifies: yes. Pre-registered: yes. Countries: 19. N = 3,878. Author overlap: yes, Fritz Strack (first author of B2o) is a co-author. Headline (abstract): facial mimicry and voluntary facial action raised happiness; the pen-in-mouth evidence was "less conclusive". Outcome differs from B2o: self-reported happiness, not cartoon funniness.

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B2s
- Quote: "We thus formed a global adversarial collaboration and carried out a preregistered, multicentre study designed to specify and test the conditions that should most reliably produce facial feedback effects." [...] "Data from n = 3,878 participants spanning 19 countries" [...] "However, evidence of facial feedback effects was less conclusive when facial feedback was manipulated unobtrusively via a pen-in-mouth task." [...] "Fritz Strack"
- Source: Coles et al. (2022), Nature Human Behaviour 6, 1731–1742. Published article page on nature.com (abstract and author list are open; body paywalled)., Abstract; author list
- URL: https://www.nature.com/articles/s41562-022-01458-9

### B5r. Many Labs 1, currency priming (pre-registered; 36 samples, 6,344 participants overall): original d = 0.80 [.05, 1.54]; replication weighted d = -0.02 [-0.08, 0.04], unweighted 0.01 [-0.06, 0.09]; proportion of samples significant in the original direction 0.03; t = -0.79; did not replicate. Qualifies: yes

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B5r
- Quote: "All materials, data, videos of the procedure, and the original preregistered design are available at the project page https://osf.io/ydpbf/." [...] "In total, there were 36 samples and settings that collected data from a total of 6,344 participants" [...] "Currency priming 0.80 .05, 1.54 0.00 0.01 /C00.06, 0.09 /C00.02 /C00.08, 0.04 0.00 0.03 0.97 t = /C00.79 6,331 6,333 0.83" [...] "Two studies – flag priming and currency priming – did not replicate the original effects." [...] "the US dollar was usually replaced with the relevant country’s currency" [...] "We thank Eugene Caruso" [decoded Table 2 row: 0.80 .05, 1.54 0.00 0.01 -0.06, 0.09 -0.02 -0.08, 0.04 0.00 0.03 0.97 t = -0.79; the text layer renders minus as /C0]
- Source: Klein et al. (2014), Social Psychology 45(3), 142–152. Published Hogrefe version (OpenMind licence), on the project's OSF page., Method p. 143; currency description p. 145; Table 2 p. 148; Results p. 149; Author note; note: original CI 95%, replication CI 99% (Table 2 columns); N = 6,333 for this effect
- URL: https://osf.io/ebmf8/

### B6r. Many Labs 1, anchoring (between-subjects, four of the original topics): weighted d 2.42 [2.33, 2.51] babies born; 2.23 [2.14, 2.32] Mt. Everest; 1.79 [1.71, 1.87] Chicago; 1.17 [1.09, 1.25] distance to NYC; original pooled 0.93 [.51, 1.33]; proportion of samples significant in the original direction 1.00 on every item; N 5,284 to 5,627 per item. Qualifies: yes

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B6r
- Quote: "For the purposes of the replication we provided anchoring information before asking just for the estimated quantity for four of the topics from the original study" [...] "Anchoring – babies born 0.93 .51, 1.33 2.43 2.60 2.41, 2.79 2.42 2.33, 2.51 0.00 1.00 0.00 t = 90.49 5,607 5,609 <.001" [...] "Anchoring – Mt. Everest 0.93 .51, 1.33 2.00 2.45 2.12, 2.77 2.23 2.14, 2.32 0.00 1.00 0.00 t = 83.66 5,625 5,627 <.001" [...] "Anchoring – Chicago 0.93 .51, 1.33 1.88 2.05 1.84, 2.25 1.79 1.71, 1.87 0.00 1.00 0.00 t = 65.00 5,282 5,284 <.001" [...] "Anchoring – distance to NYC 0.93 .51, 1.33 1.18 1.27 1.13, 1.40 1.17 1.09, 1.25 0.00 1.00 0.00 t = 42.86 5,360 5,362 <.001" [...] "The Anchoring original effect size is a mean point-biserial correlation computed across 15 different questions in a test-retest design, whereas the present replication adopted a between-subjects design with random assignments."
- Source: As B5r., Anchoring description p. 144; Table 2 and notes p. 148; note: 36 samples (B5r); original CI 95%, replication CI 99% (Table 2 columns)
- URL: https://osf.io/ebmf8/

### B7r. SSRP replication of Karpicke & Blunt (pre-registered on OSF before data collection; one team): replication r = 0.384, p = 0.006, n = 49, against original r = 0.602, p < 0.001, n = 40; relative effect size 0.638; hard copies of the original materials. Qualifies: yes

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B7r
- Quote: "pre-registered at the Open Science Framework (OSF, https://osf.io/pfdyw/) prior to starting the data collection" [...] "In a memory test one week after learning, Retrieval Practice leads to participants recalling more correct information than Concept-Mapping." [...] "Original test statistics: N = 40 (20 in each condition)" [...] "Karpicke and Blunt (2011), Science 0 . 602 < 0. 001 40 (40) 0 . 384 0 . 006 49 (49) yes 0 . 922 0 . 638" [...] "our replication used hard copies of all of the original materials used in the original study" [...] "Karpicke and Blunt (2011), Science ✓ ✗ ✓" [decoded Supplementary Table 3 row: original r = 0.602, p < 0.001, n = 40; replication r = 0.384, p = 0.006, n = 49; relative effect size 0.638; the text layer spaces the digits]
- Source: Camerer et al. (2018), Nature Human Behaviour 2, 637–644, Supplementary Information (published SI file from Nature's static-content server)., SI p. 9 (software); p. 11 (pre-registration); p. 50 (hypothesis); Supp. Table 2 p. 53; Supp. Table 3 p. 54
- URL: https://static-content.springer.com/esm/art%3A10.1038%2Fs41562-018-0399-z/MediaObjects/41562_2018_399_MOESM1_ESM.pdf

### B8r. Many Labs 1, gain vs loss framing: original d = 1.13 [.89, 1.37]; replication weighted d = 0.60 [0.53, 0.67], unweighted 0.62 [0.52, 0.71]; chi-square = 516.4, N = 6,271, p < .001; proportion of samples significant in the original direction 0.86. Qualifies: yes

- Status: VERIFIED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-B8r
- Quote: "Gain vs. loss framing 1.13 .89, 1.37 0.58 0.62 0.52, 0.71 0.60 0.53, 0.67 0.00 0.86 0.14 v 2 = 516.4 1 6,271 <.001" [...] "The replication replaced the phrase ‘‘the United States ’’ with the country of data collection, and the word ‘‘Asian’’ was omitted from"
- Source: As B5r., Framing description p. 144; Table 2 p. 148; note: 36 samples (B5r); original CI 95%, replication CI 99% (Table 2 columns)
- URL: https://osf.io/ebmf8/

### F1. Our test, H1: first results at p < .001 held 58% (589 re-runs, 294 papers); .01 to .05 held 38% (341, 189); +20 points [+12, +28]. Scope: replications people chose to run, judged by the replicating authors

- Status: EXECUTED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F1
- Quote: HEADLINE H1: original p < .001 replicated 58% (589 replications, 294 papers); .01 to .05 replicated 38% (341, 189 papers); difference +20 points [95% CI +12 to +28] → CUE HOLDS
- Source: fred_two_cues.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); FReD (Röseler et al. 2024, CC BY 4.0), OSF 2tbvd v1
- URL: https://durabilitycurve.com/methods/before-you-trust-a-study/

### F6. Anchoring original's strength, via FReD's quotation of the paper (full text not obtained): t(102) = 7.00, p < .01 as reported; the statistic itself implies a p far below the reported threshold

- Status: EXECUTED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F6
- Quote: There was a highly significant difference between individual averages of the transformed estimates made with high and low anchors, t(102) = 7.00, p < .01.
- Source: FReD claim_text_o for doi 10.1177/01461672952111004 (secondary: a database quoting the paper)

### F7. Rule of thumb for the card's statistic fallback: with at least 30 degrees of freedom, t above 3.646 (or F(1, df) above 13.29) means two-tailed p below .001; critical t is 3.551 at 40 df, 3.46 at 60 df, 3.3 at 1000 df

- Status: EXECUTED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F7
- Quote: 30 3.646 F(1,df)= 13.29 / 40 3.551 F(1,df)= 12.61 / 60 3.46 F(1,df)= 11.97 / 1000 3.3 F(1,df)= 10.89
- Source: own run: scipy.stats.t.ppf(1 - 0.0005, df), scipy 1.13.1

### F12. A no-sign failure from the 2015 sample: LoBue & DeLoache 2008 (attention to snakes and other fear-relevant images) was a strong pass (p = 0.000665379), a main effect, rated 2 for surprise (not surprising); a re-run of the same size (48) found no reliable effect (p = 0.237041962; replicators' verdict: no)

- Status: EXECUTED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F12
- Quote: Study Title (O) = Detecting the snake in the grass: attention to fear-relevant stimuli by adults and young children. / Type of effect (O) = main effect / Surprising result (O) = 2 / T_pval_USE..O. = 0.000665379 / N (O) = 48 / N (R) = 48.0 / T_pval_USE..R. = 0.237041962 / Replicate (R) = no
- Source: OSC 2015 public data, rpp_data.csv, Study Num 161 (OSF 55ddc2248c5e4a3ef7c0e4b7); own read of the row

### F13. Our held-out test: on 27 SSRP and Many Labs 2 findings not in the 2015 sample, with lay readers' doubt as the surprise sign, act held 5/5, try small 9/18 (50%), wait 0/4; act - wait +100 points, Fisher p = 0.00794

- Status: EXECUTED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F13
- Quote: Held-out card test · 27 findings (SSRP + Many Labs 2), signs frozen before outcomes / by action: act 5/5 (100%) · try small 9/18 (50%) · wait 0/4 (0%) / act - wait +100 points, Fisher p = 0.00794, monotone = True -> CARD SORTS ON HELD-OUT CLAIMS
- Source: heldout_card_test.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); Hoogeveen et al. 2020 data (osf.io/wg9s3)
- URL: https://durabilitycurve.com/methods/before-you-trust-a-study/

### F14. Held-out comparisons: base rate held 52% (14/27); p-value cue alone 67%; lay majority alone 67%; card act-vs-wait 100% on 9 of 27 decided

- Status: EXECUTED (read 2026-10-26)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F14
- Quote: accuracy, p-value only (strong -> holds): 67% (n = 27) / accuracy, lay majority only (most predict it replicates -> holds): 67% (n = 27) / accuracy, card act vs wait (try small abstains): 100% on 9 of 27 decided / base rate held: 52% (14/27)
- Source: as F13

### F15. Our small-group check: with sign 4 judged by one lay rater instead of the crowd (F13), the card still sorted the held-out findings: act held 100%, try small 52%, wait 32%; median act - wait +70; the held-out rule held in 82% of draws (88% with three raters, 90% with five)

- Status: EXECUTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F15
- Quote: k = 1 panel: agree w/ crowd 67% · act 100% (avg 3.6) · try 52% · wait 32% (avg 9.0) · median gap +70 [+42, +100] · rule holds 82% / -> SMALL GROUP KEEPS THE SORT (k = 1) / k = 3 panel: ... rule holds 88% / k = 5 panel: ... rule holds 90%
- Source: smallgroup_check.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); Hoogeveen et al. 2020 data (osf.io/wg9s3)
- URL: https://durabilitycurve.com/methods/before-you-trust-a-study/

### F17. Exact 95% intervals for the held-out cells: act 5 of 5, 47.8% to 100.0%; wait 0 of 4, 0.0% to 60.2%; try small 9 of 18, 26.0% to 74.0%

- Status: EXECUTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F17
- Quote: 5/5: exact 95% [47.8%, 100.0%] / 0/4: exact 95% [0.0%, 60.2%] / 9/18: exact 95% [26.0%, 74.0%]
- Source: Clopper–Pearson intervals computed with scipy.stats.beta from F13's counts

### A01h. In the 2015 project, each original's "surprising result" score was the average rating of at least 3 volunteers, naive to the results, on a 1 to 6 scale

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-A01h
- Quote: "Average response of at least 3 volunteers, naive to results, who report how suprising they found the key effect to be: 1: not at all, 2: slightly, 3: somewhat, 4: moderately, 5: very, 6: extremely" [sic: "suprising"]
- Source: OSC 2015 public data codebook, rpp_data_codebook.csv (OSF 55ddc2248c5e4a3ef7c0e4b7), row "Surprising result (O)", column Description

### CR1a. Inputs include the original study's sample size, effect size, p-value, and whether the tested effect is an interaction

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-CR1a
- Quote: "among these features we have sample size, the effect size and p-value originally measured, and whether a finding is an effect of one variable or an interaction between multiple variables"
- Source: Altmejd, Dreber, Forsell, Huber, Imai, Johannesson, Kirchler, Nave & Camerer (2019), "Predicting the replicability of social science lab experiments", PLOS ONE 14(12): e0225826. Publisher PDF + JATS XML (open access), § 1 Introduction, PDF p. 2 (PDF loose match: "vari- able" breaks across a line; XML strict)
- URL: https://doi.org/10.1371/journal.pone.0225826

### CR3a. They proposed moving the default significance threshold from 0.05 to 0.005 for claims of new discoveries

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-CR3a
- Quote: "We propose to change the default P-value threshold for statistical significance from 0.05 to 0.005 for claims of new discoveries."
- Source: Benjamin et al. (2018), "Redefine statistical significance", Nature Human Behaviour 2: 6–10 (online 1 Sep 2017). Publisher page, abstract only: the full text is paywalled, Abstract (loose match: the page's markup renders "P -value")
- URL: https://www.nature.com/articles/s41562-017-0189-z

### F20. Our SCORE test, pre-registered: on SCORE's re-runs (331 claims, 212 papers), originals at p < .001 replicated 66% against 34% for p between .01 and .05, a gap of +32 points (+16 to +46), each paper weighted once: the cue held on the large batch our falsifier named

- Status: EXECUTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F20
- Quote: A_S_vs_W: 66% (152 claims, 96 papers) vs 34% (63, 48) · +32 [+16, +46] -> CUE HOLDS / SCORE cue test · mode real · 331 claims from 212 papers
- Source: score_cue_test.py, pre-registered (plan, script and results on the methods page, durabilitycurve.com/methods/before-you-trust-a-study/); SCORE data, OSF dtzx4 (CC0)
- URL: https://durabilitycurve.com/methods/before-you-trust-a-study/

### F21. SCORE, the card's first sign as a reader applies it: strong passes replicated 65% against 38% for weak passes, +27 (+14 to +39); psychology and health only, p-band test +25 (+1 to +47); middle band .001 to .01 held 22%

- Status: EXECUTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F21
- Quote: B_card_strong_vs_weak: 65% (175 claims, 113 papers) vs 38% (144, 105) · +27 [+14, +39] -> CUE HOLDS / [psychology and health (secondary)] A_S_vs_W: 67% (54 claims, 45 papers) vs 42% (33, 23) · +25 [+1, +47] / A middle band (.001-.01) rate: 22%
- Source: as F20

### F22. SCORE, sample size: the biggest third of original samples replicated 52% against 40% for the smallest third, +12 (−5 to +27), not detectable, so the sample-size falsifier did not fire

- Status: EXECUTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F22
- Quote: C_size_top_vs_bottom: 52% (98 claims, 57 papers) vs 40% (98, 72) · +12 [-5, +27] -> NO DETECTABLE ADVANTAGE FOR BIG SAMPLES / falsifier, sample size: not fired
- Source: as F20

### W1a. SCORE (Tyner et al., Nature 2026) re-ran 274 claims of positive results from 164 papers published 2009 to 2018 in 54 social- and behavioural-science journals; median power 99.6% to detect the original effect; 151 of 274 replicated (55.1%)

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-W1a
- Quote: "Here we attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioural sciences. Replications were high powered on average to detect the original effect size (median of 99.6%)" … "Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1%"
- Source: Tyner et al. (2026), "Investigating the replicability of the social and behavioural sciences", Nature, DOI 10.1038/s41586-025-10078-y; abstract from the Nature page, Abstract
- URL: https://www.nature.com/articles/s41586-025-10078-y

### W1b. SCORE's papers came from a stratified random sample, but which claims were re-run depended on feasibility, so the re-run set is not itself random

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-W1b
- Quote: "random sampling is lost because selection is based on feasibility, available resources, and available expertise."
- Source: Tyner et al. (2026), accepted manuscript (JYX repository), Methods, "Sampling and selecting claims", PDF p. 18
- URL: https://jyx.jyu.fi/handle/123456789/109872

### W1c. Colquhoun (2014) argued that to keep false discoveries below 5% you should insist on p ≤ 0.001

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-W1c
- Quote: "It is concluded that if you wish to keep your false discovery rate below 5%, you need to use a three-sigma rule, or to insist on p≤0.001."
- Source: Colquhoun (2014), "An investigation of the false discovery rate and the misinterpretation of p-values", Royal Society Open Science 1:140216, Abstract
- URL: https://pmc.ncbi.nlm.nih.gov/articles/PMC4448847/

### W1d. Gordon et al. (2021): across findings from four replication projects (RPP, EERP, ML2, SSRP), originals at p ≤ 0.005 replicated about 74% against 28% above it

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-W1d
- Quote: "While the replication rate for findings with p < = 0.005 is about 74%, the replication rate for findings with p > 0.005 drops to 28%."
- Source: Gordon, Viganola, Dreber, Johannesson & Pfeiffer (2021), PLOS ONE 16: e0248780 (RPP 40, EERP 18, ML2 24, SSRP 21), § 3.5 "P value analysis"
- URL: https://pmc.ncbi.nlm.nih.gov/articles/PMC8046229/

### W1f. Spencer Greenberg, in a podcast interview: in 325 replications, originals at p ≤ 0.01 replicated about 72%, against about 48% above 0.01

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-W1f
- Quote: "we looked at 325 studies that were replications of existing studies." … "when the original study p-value was at most 0.01 — so it was 0.01 or less — about 72% replicated." … "when the p-value is greater than 0.01, only about 48% replicated."
- Source: 80,000 Hours podcast episode #147, Spencer Greenberg, transcript, Transcript § "How often papers replicate with different p-values [00:43:31]"
- URL: https://80000hours.org/podcast/episodes/spencer-greenberg-stopping-valueless-papers/

### F26. Forced to call every one of the 27 held-out findings act or wait, the card scores 18 of 27, the same share as the p-value alone: its act calls (5 of 5) and wait calls (4 of 4) are right, and the 18 try-small findings split 9 held, 9 not, so either forced call gets 9

- Status: EXECUTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F26
- Quote: forced: act 5 right + wait 4 right + 9 of 18 try-small either way = 18 of 27 = 66.7
- Source: arithmetic on F13's counts

### F28. EXPLORATORY: in SCORE's new-data replications only (146 claims, 114 papers), originals at p < .001 replicated 67% against 32% for .01 to .05, +35 (+13 to +55)

- Status: EXECUTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F28
- Quote: EXPLORATORY · new-data replications only · 146 claims, 114 papers / A S vs W: 67% (51, 46) vs 32% (39, 32) · +35 [+13, +55] -> CUE HOLDS
- Source: explore_score_newdata.py (exploratory, added after the locked run)

### F29. In the held-out test, all 5 strong passes were act calls; the other two signs split the 22 weak passes: 9 of 18 with no other sign held, 0 of 4 with one did

- Status: EXECUTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F29
- Quote: weak passes 22 by action {'try small': 18, 'wait': 4} strong passes 5 strong actions {'act': 5} / by action: act 5/5 (100%) · try small 9/18 (50%) · wait 0/4 (0%)
- Source: heldout_signs.csv and F13's counts

### W1g. A SCORE companion paper (Abatayo et al., MetaArXiv, 2026) reports the original p-value's correlation with replicability as -.11 (-.26 to .04) by paper and -.18 (-.29 to -.05) by claim; the Nature paper defers to it

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-W1g
- Quote: "Original p-value" … "-.11" … "[-.26, .04]" (Table 3) … "Original p-value" … "-.18" … "[-.29, -.05]" (Table S19, by claim) … "We did not explore correlations with replication outcomes"
- Source: Abatayo et al. (2026), "Assessments of Credibility in the Social and Behavioral Sciences", MetaArXiv 7u58q_v1; Tyner et al. (2026) accepted manuscript for the deferral, Table 3, p. 9; Table S19; Nature accepted manuscript
- URL: https://osf.io/preprints/metaarxiv/7u58q_v1

### R4d. Alvaro de Menard (2020), arguing from replication-forecast data, put replication below 50% for originals with p > .001 and about 10 percentage points lower for interaction effects

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R4d
- Quote: "The base replication rate for studies with p>.001 is below 50%." … "A paper whose main finding is an interaction effect is about 10 percentage points less likely to replicate."
- Source: de Menard (2020), "What's Wrong with Social Science and How to Fix It", Fantastic Anachronism, 11 Sep 2020 (basis: Replication Markets forecasts), Sections "What To Do" and "Interaction Effects Bad"
- URL: https://fantasticanachronism.com/2020/09/11/whats-wrong-with-social-science-and-how-to-fix-it/

### R4g. SCORE's 274 claims split into 128 new-data and 146 secondary-data replications; secondary-data replications used existing data independent of the original

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R4g
- Quote: "128 / 128" … "146 / 146" … "Secondary data replications are those that used existing data that was independent of the original investigation." (274 = 128 + 146)
- Source: Tyner et al. (2026), accepted manuscript, Table 4, Table 4 and note
- URL: https://jyx.jyu.fi/handle/123456789/109872

### R5a. Vu & Faridani forecast replication from each original's p-value, together with an estimate of how large effects in that field tend to be, in three projects: economics (Camerer 2016), psychology (OSC 2015) and social science (Camerer 2018)

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R5a
- Quote: "We estimate the predictive power curve in three empirical literatures with systematic replication evidence: experimental economics (Camerer et al., 2016), psychology (Open Science Collaboration, 2015), and experimental social science (Camerer et al., 2018)."
- Source: Vu & Faridani (2026), "How Replicable Are Statistically Significant Findings?", arXiv 2608.23257 (working paper), p. 2
- URL: https://arxiv.org/abs/2608.23257

### R5b. On the 70 findings that also had prediction-market forecasts, the p-only forecasts' average squared error (Brier score) was 0.216 against the markets' 0.220, pooled and weighted

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R5b
- Quote: "the validation exercise uses the common sample of 70 studies for which both predictions are available" … "In the weighted pooled sample, predictive power slightly outperforms prediction markets, with Brier scores of 0.216 and 0.220, respectively."
- Source: Vu & Faridani (2026), "How Replicable Are Statistically Significant Findings?", arXiv 2608.23257 (working paper), pp. 11–12, Table 1 (psychology alone 0.243 and 0.243)
- URL: https://arxiv.org/abs/2608.23257

### R5d. In psychology, a finding at p = 0.05 has an expected replication probability of 0.22 in a same-size re-run; across the three fields, 0.1 to 0.25

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R5d
- Quote: "Evaluating the curve at the 5% significance threshold, we find that replicability ranges between 0.1 and 0.25 across fields." … Figure 2, psychology panel: "(0.05, 0.22)"
- Source: Vu & Faridani (2026), "How Replicable Are Statistically Significant Findings?", arXiv 2608.23257 (working paper), p. 13 and Figure 2, p. 14
- URL: https://arxiv.org/abs/2608.23257

### R5f. Carter & McCullough re-analysed the 198 published tests in Hagger et al.'s 2010 meta-analysis

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R5f
- Quote: "Hagger et al. (2010) reported a meta-analysis of the 198 published tests of the depletion effect"
- Source: Carter & McCullough (2014), Frontiers in Psychology 5:823, Introduction
- URL: https://www.frontiersin.org/articles/10.3389/fpsyg.2014.00823/full

### R5g. Corrected for small-study effects (publication bias among them), the depletion effect was not meaningfully different from zero

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R5g
- Quote: "an indication that the depletion effect is actually no different from zero" … "after controlling for the influence of small-study effects, our results do not support the claim that the depletion effect is meaningfully different from zero"
- Source: Carter & McCullough (2014), Frontiers in Psychology 5:823, Abstract; Discussion
- URL: https://www.frontiersin.org/articles/10.3389/fpsyg.2014.00823/full

### R5h. Carter & McCullough turned to the meta-analysis after failing to produce the depletion effect in their own work

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R5h
- Quote: "In response to our inability to produce the basic depletion effect in our own work, we turned to Hagger et al.'s (2010) meta-analysis"
- Source: Carter & McCullough (2014), Frontiers in Psychology 5:823, Introduction
- URL: https://www.frontiersin.org/articles/10.3389/fpsyg.2014.00823/full

### R5j. Falkovich's lay rules include anti-significance (a result at p = 0.049 is suspect) and Taleb's grandma (an effect that sounds implausible probably is not real)

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R5j
- Quote: "If a study has p=0.049 it is fake." … "If the purported effect sounds implausible, it is."
- Source: Jacob Falkovich (7 Sep 2018), "The Scent of Bad Psychology", Putanumonit, Rules of Anti-Significance and Taleb's Grandma
- URL: https://putanumonit.com/2018/09/07/the-scent-of-bad-psychology/

### R5k. Bailey's rules of thumb include trusting overall effects over interaction or subgroup effects, extraordinary claims needing extraordinary evidence, and the real effect being half what the paper claims; they do not use p-values

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R5k
- Quote: "Overall effects are more trustworthy than interaction or subgroup effects" … "extraordinary claims require extraordinary evidence" … "The real effect size is half what the paper claims"
- Source: James Bailey (2 Apr 2026), "How Much To Trust Research Papers? My Rules Of Thumb", Economist Writing Every Day, rules 2, 6 and 8
- URL: https://economistwritingeveryday.com/2026/04/02/how-much-to-trust-research-papers-my-rules-of-thumb/

### R6a. Lindsay's "troubling trio" (2015): low statistical power, a surprising result, and a p-value only slightly lower than .05

- Status: REPORTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R6a
- Quote: "Described as a combination of low statistical power, a surprising result, and a p-value only slightly lower than .05." (FORRT glossary entry, citing Lindsay 2015)
- Source: Lindsay (2015), "Replication in Psychological Science", Psychological Science 26(12), 1827–1832, FORRT glossary
- URL: https://forrt.org/glossary/english/the_troubling_trio/

### R6b. SCORE reports thirteen measures of replication success, whose estimates range from 28.6% to 74.8% (median 49.3%)

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R6b
- Quote: "Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median of 49.3%)."
- Source: Lakens (3 May 2026), "There are 13 ways to analyse a replication study, but only one of them is coherent", The 20% Statistician, quoting Tyner et al. 2026
- URL: https://daniellakens.blogspot.com/2026/05/there-are-13-ways-to-analyse.html

### R7a. The 1998 radish experiment was repeated once, as a school science-fair project, reported in the original paper's footnote; we found no registered re-run of that experiment

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R7a
- Quote: "As this article went to press, we were notified that this experiment had been independently replicated by Timothy J. Howe, of Cole Junior High School in East Greenwich, Rhode Island, for his science fair project"
- Source: Baumeister, Bratslavsky, Muraven & Tice (1998), JPSP 74(5), footnote 2, footnote 2
- URL: https://faculty.washington.edu/jdb/345/345%20Articles/Baumeister%20et%20al.%20%281998%29.pdf

### R7c. A 2026 re-analysis of SCORE's data finds that replication-success methods cannot reliably reach a conclusion at typical sample sizes

- Status: REPORTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R7c
- Quote: "Replication success methods are unable to reliably draw a conclusion about replication success for typical sample sizes of the original study and replication"
- Source: van Aert, van Assen & López-Nicolás (2026), PLOS ONE 21(9): e0358126, Abstract
- URL: https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0358126

### R7e. A pre-registered 2023 study (N = 230) reproduced the retrieval-practice advantage on recall (verbatim) questions, then found no significant difference when the concept-mapping group also memorised (p = 0.854); the authors call the advantage a methodological artifact

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R7e
- Quote: "While we replicated the previous finding that retrieval practice in combination with additional memorizing produces more learning than concept mapping without additional memorizing, this advantage of retrieval practice over concept mapping vanished when participants in the concept mapping condition, too, memorized" … "performance did not differ ( p = 0.854) between the RP + AM + AI condition and the CM + AM + AI condition" … "resulting in a final sample size of N = 230" … "followed our preregistration at Open Science Foundation"
- Source: Mayrhofer, Kuhbandner & Frischholz (2023), Frontiers in Psychology 14:1258359, Abstract; Methods
- URL: https://pmc.ncbi.nlm.nih.gov/articles/PMC10783554/

### R7f. Hoogeveen et al. reported that findings their lay participants, as a group, predicted would fail did fail to replicate

- Status: VERIFIED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-R7f
- Quote: "when participants as a group predicted that a particular effect would fail to be replicated, it indeed failed to be replicated"
- Source: Hoogeveen, Sarafoglou & Wagenmakers (2020), AMPPS, Results
- URL: https://doi.org/10.1177/2515245920919667

### F30. In the held-out test, the card's four wait calls are exactly the four findings with the lowest lay share predicting replication (Lee & Schwarz 0.172, Gervais & Norenzayan 0.299, Shah et al. 0.368, Tversky & Gati 0.468); all four are weak passes, one has an interaction

- Status: EXECUTED (read 2026-10-27)
- Row link: https://durabilitycurve.com/claims/survives-a-recheck/#c-F30
- Quote: Lee & Schwarz (2010) lay 0.172 wait · Gervais & Norenzayan (2012) lay 0.299 wait · Shah et al. (2012) lay 0.368 wait · Tversky & Gati (1978) lay 0.468 wait · next lowest Alter et al. (2007) lay 0.504 try small; doubted True only on the four; interaction True on one
- Source: heldout_signs.csv

## Cite

- A claim: name the original source (and its locator) first, then the row it was checked in, e.g. "<source>, <locator>. Checked in: Claim Ledger, "Before You Trust a Study, Check How Strongly It Passed", The Durability Curve, row <id>, https://durabilitycurve.com/claims/survives-a-recheck/#c-<id>."
- The essay: Floyd, Harry (2026). Before You Trust a Study, Check How Strongly It Passed. The Durability Curve. https://durabilitycurve.com/blog/survives-a-recheck/
- This ledger: Floyd, Harry (2026). Claim Ledger: Before You Trust a Study, Check How Strongly It Passed. The Durability Curve. https://durabilitycurve.com/claims/survives-a-recheck/
- Downloads: https://durabilitycurve.com/claims/survives-a-recheck.csv · https://durabilitycurve.com/claims/survives-a-recheck.json

Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Reuse it, quote it or train on it, with credit to The Durability Curve and a link. Open to AI (robots.txt: search=yes, ai-input=yes, ai-train=yes).