# Notes on method

The detail behind the essay's numbers: how the tests placed and weighted results, the prior work behind the cue, and the sources for the six famous claims and the look back at 27 findings.

## How the tests were run

Both of our tests place a result in a band only when its reported p-value leaves no doubt: at or below .001 (exactly, or reported against a threshold of .001 or less), or exactly between .01 and .05 (a result printed only as "p < .05" or "p < .01" is left out). They weight each original paper equally within each band (a paper's several claims are averaged, and a paper with claims in both bands counts in each), and were written down and fixed before any claim-level outcome was opened. On a 273-claim subset of SCORE's public data chosen to match the main report more closely (generalisability tests and claims flagged as COVID-related removed), the gap was 34 points (exploratory). The strong band had 152 claims from 96 papers and the narrow band 63 claims from 48 papers; SCORE's median power to detect the original effect was 99.6%. "Held up" is SCORE's own criterion. The project scored success thirteen ways, from 28.6% to 74.8% overall (Lakens, 2026), and a 2026 re-analysis of its data reports that single re-runs often cannot settle a finding either way (van Aert and colleagues). Its companion paper reports weak straight-line correlations between p-values and success (−.18 by claim, −.11 by paper), because the rate does not fall evenly: the middle band held up no better than the narrow one, so the gain sits above .001. The new-data split uses the claims SCORE's public data labels new data. The earlier test used FReD, a database of re-runs, whose outcome is the one reported by each replication study, with mixed results counted as failures; its strong band had 589 re-runs from 294 papers, and its narrow band 341 from 189. The data are public.

## Who suggested the cue before us

The cue has a long line of credit. In 2015 Stephen Lindsay, then editing *Psychological Science*, warned against a "troubling trio" of low power, a surprising result and a p-value only slightly under .05. Jacob Falkovich's rules of thumb for spotting weak psychology (2018) point at two of the card's signs. Goodman (1992) worked out how often a result at p = .05 should reach significance again. Benjamin and colleagues (2018) proposed p < .005 for new discoveries; Colquhoun (2014) argued for p ≤ .001. Gordon and colleagues (2021) found about 74% against 28% at a line of .005 across four re-run projects; Spencer Greenberg reported about 72% against 48% at .01 across 325 re-runs. Alvaro de Menard (2020) named nearly the same signs from replication-forecast data, and Altmejd and colleagues (2019) found interactions replicate worse. James Bailey's rules of thumb (2026) include distrusting interactions and assuming the real effect is half the claimed one. Vu and Faridani's paper is a working paper (arXiv, August 2026).

## The six famous claims

Anchoring's first result is the original effect as reconstructed by Many Labs from the 1995 paper (Klein and colleagues, 2014). The learning study's challenge is Mayrhofer, Kuhbandner and Frischholz (2023), with 230 participants. The pen: a 2022 multi-site project found facial-feedback effects with other methods and less conclusive evidence for the pen (Coles and colleagues). We first scored seven claims and dropped growth mindset from the table, because its re-run evidence comes from related programmes rather than the 2007 study: an independent trial in England (5,018 pupils) found no added progress, and the theory team's own national study found small effects among lower-achieving students. Willpower: the original paper's footnote reports a junior-high student's science-fair repeat that agreed with it, the only re-run of that exact experiment we found. The snake study is LoBue and DeLoache (2008).

## The look back at 27 findings

The card was frozen and its test written down before we scored these 27, but their outcomes and the lay ratings had been published. They come from the 2018 re-runs of *Nature* and *Science* studies and Many Labs 2. A language model coded interactions from plain descriptions, and doubt came from Hoogeveen, Sarafoglou and Wagenmakers's lay readers (2020), about 125 per finding, who made their calls without knowing the outcomes. The intervals are 48% to 100% for act and 0% to 60% for wait, and four is a small group. Forced to call, a "try small" finding could go either way; with 9 of those 18 holding, either choice gives 18 of 27. With one reader's doubt in place of the crowd's, resampling the published ratings, about 3 in 10 "wait" findings held. The re-run size condition in step 1 is a design choice, not a tested rule. The look back coded a weak pass from the published p alone; the card's current rule for a paper that gives no exact p but reports an F (one numerator and 30 or more denominator degrees of freedom, above about 14) came later. The card still needs a test on findings whose outcomes nobody yet knows: if well-powered re-runs of findings that pass the Re-check fail as often as findings it flags, the check is useless, and we will say so.