Methods
Before You Trust a Study: the plans, code and results
Everything behind the numbers in the essay, so anyone can run them again. Each plan was fixed before its script was run on real outcomes.
The essay is Before You Trust a Study, Check How Strongly It Passed. Start with the README: it lists the public data each script needs, where to download it, and the sha256 of every file here. How each test placed and weighted results, and the prior work behind the cue, are in the notes on method.
01 · The tests
written down first
-
SCORE: strong against narrow first passes
In SCORE’s public data, first results at or below p = .001 held up 66% of the time (152 claims from 96 papers), and those exactly between .01 and .05 held up 34% (63 claims from 48 papers). Each paper weighted equally within a band; the gap is 32 points, interval 16 to 46.
-
FReD: the same comparison on re-runs researchers chose to do
58% against 38%. The outcome is the one each replication study reported, with mixed results counted as failures.
-
The card on 27 earlier findings
A look back: the outcomes, and lay readers’ judgements of them, were already published. Act 5 of 5 held, try small 9 of 18, wait 0 of 4; forced to call all 27, the card scores the same as the p-value alone.
-
One reader’s doubt in place of the crowd’s
Resampling the published ratings, about 3 in 10 “wait” findings then held. This plan was written after the look back above.
02 · Exploratory
run after the planned tests
03 · The data
public sources
SCORE
Tyner and colleagues, Nature, 2026. Public data on OSF (project dtzx4); the README names the four files and their hashes.
FReD
Röseler and colleagues, 2024, Journal of Open Psychology Data (CC BY 4.0). The subset the test runs on is here, unchanged.
Hoogeveen, Sarafoglou and Wagenmakers, 2020
The 27 findings and lay readers’ ratings, public on OSF. Our own coding of each finding’s signs is here, with the prompt a language model used to code interactions.