In 1988, students who held a pen between their teeth, which forces something like a smile, rated cartoons funnier than students who held it in their lips. In 2016, seventeen labs repeated the study on a pre-registered plan, and the difference was gone.

A study about learning went the other way. In 2011, students who practised recalling what they had read remembered it better a week later than students who drew concept maps of it. When a team re-ran the study on a registered plan in 2018, the effect came back, smaller but clear.

The clue was printed in each paper. The pen study’s result barely cleared the line for significance, and only on a one-tailed test (t = 1.85). The learning study’s cleared it easily (a t of about 4.7, reported as an F of 21.6). Researchers have suspected for years that this clue matters. We tested it on papers a large project had first sampled at random, with the rule fixed before we opened the results.

Two original studies, both stamped PASSED. The first, at p < .001, carries a deep gold stamp, and its readout says 66% of studies like it held up. The second, at p = .04, carries a faint, under-inked red stamp, and its readout says 34% of studies like it held up.

The short version: how strongly a study’s result passed is a clue to whether it will hold up. Our test used SCORE, a large project that re-checked findings from social- and behavioural-science papers. With the rule fixed before we opened each claim’s result, claims that passed strongly (p below .001) held up 66% of the time, and narrow passes (.01 to .05) 34%. Before you act on a study, look for an independent re-run on a registered plan, at least as large as the original; if there is none, count three warning signs. The free one-page card below takes you through it.

What we found

SCORE, a project published in Nature this April, began with a random sample of papers from 62 social- and behavioural-science journals published between 2009 and 2018. Its main report covers 274 re-checked claims from 164 of those papers, in 54 of the journals; its public data, which we used, cover 331 re-checked claims from 212 papers.

The median re-check had a 99.6% chance of detecting an effect as large as the one originally reported. Which claims could be re-checked depended on feasibility, so the final set is not itself random, and about half the re-checks used existing data collected independently of the original rather than new participants.

Before opening SCORE’s claim-by-claim outcomes, we wrote our test down and fixed it. Weighting each paper equally within each band, claims whose first result passed strongly, below p = .001, held up 66% of the time. Those that passed narrowly, between .01 and .05, held up 34% of the time. The gap is 32 percentage points, with an interval from 16 to 46.

“Held up” is SCORE’s own criterion: statistically significant again, in the same or a similar pattern as the original. That is evidence consistent with the finding; it does not establish that the effect is real, causal or large enough to matter.

The learning study shows what “held up” leaves open. In 2023 a registered study reproduced its advantage on recall questions, then gave the concept-mapping group the same extra memorising phase and matching instructions, and found no significant difference between the two. The result held up, but the explanation for it was called into question.

SCORE scored success thirteen ways, from 28.6% to 74.8% overall (Lakens, 2026), and a 2026 re-analysis of its data reports that single re-runs often cannot settle a finding either way (van Aert and colleagues).

On the re-checks that collected new data, a split we added afterwards, the gap between strong and narrow passes was as large. An earlier test of ours, also written down first, on re-runs researchers chose to do, found the same pattern: 58% against 38%. There, a finding held up if its re-run’s own report said so, with mixed results counted as failures.

Strong first passes held up about twice as often as narrow ones: 66% of re-checked claims whose original result passed below p = .001, against 34% of those between .01 and .05. Claims between .001 and .01 held up 22% of the time. SCORE's public data

Two more results shaped the check below. In our sample the clearest separation came at .001: findings that passed between .001 and .01 held up 22% of the time, no more often than the narrow ones. SCORE’s companion paper reports only weak straight-line correlations between p-values and success (−.18 by claim, −.11 by paper), which fits a rate that does not fall evenly.

Large samples, the feature 215 psychologists ranked first in a survey published in 2024, did less than the strength of the pass: the largest third of original samples held up 12 points more often than the smallest third, with an interval from −5 to 27. Size may matter, but this test cannot show it, so the check below leaves it out.

Why a narrow pass is a warning

A study too small for the effect it is chasing needs luck to clear the bar, and luck rarely clears it by much. So when underpowered studies pass, they tend to pass narrowly: in a standard worked example, with a one-in-three chance of detecting a real effect, more than half of the significant results land above p = .01. A narrow pass is often the trace of a study that was too small.

Others have found the same pattern. When a project re-ran 100 psychology studies in 2015, 20 of the 32 originals with p below .001 held up, against 2 of the 11 above .04.

This August, in a working paper, the economists Patrick Vu and Stefan Faridani built the idea into forecasts. Using each original’s p-value, together with an estimate of how large effects in that field tend to be, their model predicted which findings held up in three large re-run projects about as accurately as markets in which scientists had bet on the same outcomes. By their estimate, a psychology finding at p = .05 has about a one-in-five chance that a re-run of the same size comes out significant again.

The Re-check

Here is the check we built from these results. Its first warning sign, a weak pass, is the one the tests above support. The other two come from the same 2015 project: interaction effects, which depend on another factor, held up there 8 times in 37, against 23 in 49 for main or simple effects, and results that volunteers rated surprising held up less often.

Lay readers can apply the third: asked to judge 27 re-run findings, their predictions were right about 59% of the time from plain descriptions, and about 67% when they were also shown how strong the original evidence was. Treat those two signs as experimental; the look back further down shows why.

The Re-check

Find the original study. Use the claim’s source, or search its name with “study”. If you cannot find it, treat the claim as unverified: do not build on it.

Re-run first. Was it re-run by people who did not run the original, on a plan registered before the data came in, with at least as many people as the original, in one lab or several? If yes, go by the re-run. One re-run can still leave the answer open, so give most weight to large re-runs and to reviews that pool several studies. If someone has re-analysed the whole pile of published studies on it for missing failures, read that too. If no re-run, count three warning signs.

A weak pass. The first result’s p was not below .001. If the paper gives only a bound such as “p < .05” or “p < .01”, or no p at all, use its t statistic: above about 3.7 with 30 or more degrees of freedom is strong. An F with one numerator and 30 or more denominator degrees of freedom is strong above about 14; any other F counts as no statistic. No statistic at all counts as the sign.

An interaction. The effect depends on another factor.

A claim most people would doubt. Put it plainly, the way it would sound to someone hearing it for the first time. Would you doubt it would hold up? Asking two or three others is steadier.

No signs: the strongest starting point. Consider acting if the study’s design is sound, the result applies to your setting, and being wrong would be affordable. One sign: try it only if it is cheap and reversible, and watch the result yourself. Two or three: wait for a re-run.

Without a qualifying re-run, expect the effect to be smaller than claimed. In one large project it averaged about half, and it can shrink further or vanish.

The size condition in step 1, a re-run at least as large as the original, is a design choice we have not tested.

Run the pen study through the card as a reader in 1990 would have. You find the original. Nobody has repeated it directly yet. Its result passed weakly, and a claim that a pen in your teeth makes cartoons funnier is one most people would doubt. Two signs: wait. Today step 1 settles it, because seventeen labs have since re-run it. A 2022 multi-site project found facial-feedback effects with other methods and less conclusive evidence for the pen (Coles and colleagues).

Six famous claims

We scored six famous claims from their original papers and fixed the calls before reading any verdict. We already knew how some had turned out, so treat the table as an illustration. “Held” means a registered re-run found the effect in the same direction and its authors called it replicated.

The willpower row is the card’s blind spot. The 1998 study passed strongly, so its warning signs alone say try it small. The warning came from elsewhere, in 2014.

Evan Carter and Michael McCullough, who had failed to produce the effect in their own lab, re-analysed the 198 published tests behind the theory and, correcting for signs of publication bias, found an indication that the effect was no different from zero, two years before 23 labs testing the same idea on a registered plan found almost nothing. That is why the card’s first step asks you to look for a re-analysis of the pile.

Act calls can fail as well: in the 2015 project, a strongly passing study of how quickly adults and young children spot snakes was re-run with as many people and found no reliable effect.

A look back at 27 other findings

We then applied the fixed card to 27 findings it was not built from, from two later re-run projects: the 2018 re-runs of Nature and Science studies and Many Labs 2. The 27 include the learning study from the opening, also in the table above, so on that finding the two are not independent evidence. Their outcomes, and lay readers’ judgements of them, were already published, so this is a look back rather than a test in advance.

A language model coded interactions from plain descriptions, and doubt came from Hoogeveen, Sarafoglou and Wagenmakers’s lay readers, about 125 per finding, who made their calls without knowing the outcomes.

All 5 findings the card said to act on held up, and none of the 4 it said to wait on did. Of the 18 it said to try small, 9 held, a coin toss, which is what trying something small is for. The intervals are 48% to 100% for act and 0% to 60% for wait, and four is a small group.

Those calls used the average doubt of about 125 readers. With one reader’s doubt instead, drawn at random from the published ratings, about 3 in 10 “wait” findings held.

Forced to call all 27, the card scores 67%, the same as the p-value alone, because every strong pass held. Its four wait calls are the four findings those lay readers doubted most, and their study had already reported that findings the group doubted failed. So the extra signs have not yet shown value of their own, and they rest on a small set that has been studied before.

Where the idea comes from

The cue has a long line of credit. In 2015 Stephen Lindsay, then editing Psychological Science, warned against a “troubling trio” of low power, a surprising result and a p-value only slightly under .05. Goodman (1992) worked out how often a result at p = .05 should reach significance again. Benjamin and colleagues (2018) proposed p < .005 for new discoveries; Colquhoun (2014) argued for p ≤ .001.

Gordon and colleagues (2021) found about 74% against 28% at a line of .005 across four re-run projects; Spencer Greenberg reported about 72% against 48% at .01 across 325 re-runs. Alvaro de Menard (2020) named nearly the same signs from replication-forecast data, and Altmejd and colleagues (2019) found interactions replicate worse.

Jacob Falkovich’s rules of thumb for spotting weak psychology (2018) point at two of the card’s signs. James Bailey’s rules of thumb (2026) include distrusting interactions and assuming the real effect is half the claimed one.

Better odds, never proof

Nothing here tells you whether one study is true. A strong first pass held up about two times in three in our test, and a narrow one about one time in three. That helps decide how much to stake: start from the strong, simple, unsurprising finding, try the weaker one only where a mistake costs little, and wait on the rest.

The card still needs a test on findings whose outcomes nobody yet knows: if well-powered re-runs of findings that pass the Re-check fail as often as findings it flags, the check is useless, and we will say so.

Run the Re-check this week on one piece of advice you already follow. The card is a free download below, and our plans, scripts, results and notes on method are on the methods page. The next piece in this series takes findings that passed and asks how much to bet on them.

Get the Re-check as a one-page card, free.

Comments are on the Substack copy. If an essay was worth one, buy me a coffee.

It rests on Law II, Difficulty Is Load-Bearing, one of the laws this publication tests against the evidence.

Claude and ChatGPT can search this archive and its claim ledgers once you add the connector.