METHODS: Before You Trust a Study, Check How Strongly It Passed
The Durability Curve

What is here
  plans/     the written plans. Each was fixed before its script was run on real outcomes; plans 3 and 4 score
             findings whose outcomes were already published, and plan 4 was written after the plan-3 result
  scripts/   the analysis code (Python 3, numpy, pandas, scipy)
  results/   what each script printed or wrote when we ran it
  data/      the FReD subset the FReD test runs on, and our own coding in data/hoogeveen2020/
  notes-on-method.txt   the essay's notes on method (its former footnotes, word for word)

The tests
  plan-2 / score_cue_test.py     SCORE public data: strong first passes (p at or below .001) against narrow ones
                                 (exactly .01 to .05). 66% against 34%, a gap of 32 points (interval 16 to 46).
  plan-1 / fred_two_cues.py      FReD (re-runs researchers chose to do): 58% against 38%.
  explore_score_newdata.py       exploratory: the SCORE test on re-checks that collected new data.
  explore_score_main_set.py      exploratory: the SCORE test on the 273-claim subset closest to the main report.
  plan-3 / heldout_card_test.py  the card on 27 earlier findings (outcomes already published: a look back).
  plan-4 / smallgroup_check.py   the same with one reader's doubt in place of the crowd's.

Data you need to download (not copied here)
  SCORE (Tyner et al., Nature, 2026), OSF project dtzx4, folder
    "Analysis-ready Data/Replication and Reproduction Papers/":
    orig_outcomes.csv, paper_metadata.csv, repli_outcomes.csv, repli_exclusions.csv
    Put them in data/score/. score_cue_test.py checks the sha256 of the two original-data files before it runs:
    orig_outcomes.csv 325b445d0a253342eb3d2d318aa4233bcca23c16866b5c1cc2733bab8f26478f,
    paper_metadata.csv 0f2beef44537aa2f123f8045281f91cc8d524186aec98e51909e7ea5934c660e.
    It does not check the replication files, so compare them with these hashes yourself:
    The replication files we used: repli_outcomes.csv 145c46e3e29a3ccca6017d4f22cca905c48bffb0e886fa4e568d57fa4f60513e,
    repli_exclusions.csv 2df78b221a5f8563e7b8ab99f764572203641375362a6fde6787be062c67be5f.
  FReD (Roeseler et al., 2024, Journal of Open Psychology Data, DOI 10.5334/jopd.101; CC BY 4.0),
    OSF file 2tbvd, version 1 (2026-02-01), sha256
    59545e3fc0c2979d97507097526319ec8c275aae3ac40c0d1a6d327492079e6b.
    data/fred-2026-02-01-subset.csv here is its 14 analysis columns, all 2,164 rows, unchanged.
  Hoogeveen, Sarafoglou & Wagenmakers (2020), Advances in Methods and Practices in Psychological Science,
    public data at https://osf.io/wg9s3 . Download DescriptionAndStatisticsPerStudy.csv and the three participant
    files Data_MTurk_AN.csv, Data_Social_Media_AN.csv and Data_UvA_Lab_AN.csv into data/hoogeveen2020/, where our
    coding already sits.

How to run
  Each script looks for data/ in its own folder: copy data/ into scripts/, add the downloads, then from scripts/ run:
    python3 fred_two_cues.py
    python3 score_cue_test.py
    python3 heldout_card_test.py
  fred_two_cues.py, score_cue_test.py and smallgroup_check.py have a --dummy mode that never reads an outcome
  column; we ran those first.

What is exploratory
  The new-data split and the 273-claim subset were run after the planned tests; the essay marks both as
  exploratory. The psychology-and-health block in score-cue-test-results.json is plan 2's planned secondary analysis.

sha256 of every file in this folder
f3e729378bb46c8cf8896b9d85711b1535852f4a36d05bcdeb6848f972ef630c  data/fred-2026-02-01-subset.csv
0875008358e65f7476a769b45fbaed0717023d8853206a89cdeaea268ba28eb4  data/hoogeveen2020/heldout_signs.csv
996f08c996a3d50e72ec8808b0acc61ab635a61e8d00c14261709fa21dac6492  data/hoogeveen2020/interaction_codes_key.json
52be7f8b848e0fabae32b982aa397719092775ab49b1598e786368f18661dd01  data/hoogeveen2020/interaction_prompt.txt
c28265ca1f6e08bb15e3fcd1d18e4406d76ee375cf283eddad62522d52949482  data/hoogeveen2020/lay_shares_description_only.csv
43bf5c6954bc578a3aad4b9227282a21baf2d79762e5dc6ee6ede46f23241352  notes-on-method.txt
cd7b32ff0a13331c9b3d98d07238f2b0129182f3de3a1cbe9877a5217d8c03cd  plans/plan-1-fred-two-cues.md
0e4decba53e9dea44f4753a19d70479ea862d559c78b52f298f543d3c92c7b5b  plans/plan-2-score-cue-test.md
584c7189f97e3bc8d50067e3670e4b971ce021a70436a4536ee6db0540db0e03  plans/plan-3-held-out-card-test.md
4d9f380ee3b6fcc814e5b14e44a863c736437ad4a55f221d699ae020fe0900e6  plans/plan-4-single-reader-check.md
d8d77ae55d4c07383ce48eab712a20ccc7eb146bf39e2096814f5e93bc4f7aa1  results/explore-score-main-set-results.json
fc12b45c28ddceaaef48e5d71f314f884df07a62b7e1d64baeb58cacaccbe2e0  results/explore-score-newdata-output.txt
c54022dfecbd2d4b77071efd8305d73dd65f7ce23a06ef7757733e78b124e390  results/fred-two-cues-results.json
9ff68709d8b41b6eb143fed1bfffffa91cf76ec8d270bd8a9e8995112f4272b8  results/heldout-card-test-results.json
60df15bfa17e39e012390be00c25372d8793979d9f71fae40712f65804babfce  results/score-cue-test-results.json
f23bc12b92ab600ba184dd3299f7cfaa9a042c5c1efd6a302daf5d4097aaaf14  results/smallgroup-check-results.json
98f1f5283147e2052c48f3b352326d4e0ef9d4909738ec2e90a67abdfaac3058  scripts/explore_score_main_set.py
052227390dbbc23275f2fea7b77c3426d936461a853537ba5872e93596aa0f46  scripts/explore_score_newdata.py
c86f14ba4b9be3a13994d5c2c86581e173e771b5ff31e8280953cbff820ddd19  scripts/fred_two_cues.py
77ca669a6acd262f8235a9fd00b65607d0f856d804602699f9ea36b3d6ee9769  scripts/heldout_card_test.py
07a1b78b8566314ba8b4411af60c6eaed18f478bc19f3accaf665f6737228897  scripts/score_cue_test.py
6f74bbeea4d235d8b98a2da719bfd50e0bc885b150d12c6491df12353e798d89  scripts/smallgroup_check.py