# PREREG — survives-a-recheck — FReD two cues — 2026-10-26

**What is being tested.** Brief — survives-a-recheck tells the reader which cues predict whether a finding
will hold up when someone else re-runs it. Two of those cues can be read straight off a study: how strong the
original result was (its p-value) and how big the original sample was. Published sources disagree with intuition
on the second. In the Open Science Collaboration's 100 replications, originals at p < .001 replicated 20 of 32
times against 2 of 11 above .04, and larger original samples went with slightly *lower* replication (r = −0.150).
Li et al. (2024, 307 pairs) found original N did not predict on its own. This test asks both questions of the
largest public pool of re-runs, FReD, before the piece leans on either.

**Why it can fail.** Both cues come from a few hundred pairs, much of it the same OSC data. FReD is larger and
mostly other projects. Either cue can come back not shown, reversed, or (for sample size) helpful. Every outcome
below is written into the piece in advance.

## Data (pinned)

- **Source:** FORRT Replication Database (FReD), Röseler et al. 2024, *Journal of Open Psychology Data*, DOI
  10.5334/jopd.101. Licence CC-BY 4.0. File `FReD.xlsx`, OSF id `2tbvd`, version 1, dated 2026-02-01 13:19 UTC,
  1,083,820 bytes. OSF's own sha256 matches the downloaded copy:
  `59545e3fc0c2979d97507097526319ec8c275aae3ac40c0d1a6d327492079e6b`.
- **Kept at:** `data/FReD-2026-02-01.xlsx` (the original), and
  `data/fred-2026-02-01-subset.csv` (14 columns the analysis uses, all 2,164 rows), sha256
  `f3e729378bb46c8cf8896b9d85711b1535852f4a36d05bcdeb6848f972ef630c`. The script refuses to run on any other file.
- **Columns used:** `doi_o`, `doi_r`, `author_overlap`, `n_o`, `pval_value_o`, `pval_type_o`, and the outcomes
  `reported_success`, `pval_value_r`, `pval_type_r`.

## What I had seen before writing this

The column list and non-null counts. The two predictors in full: the p-value reporting types and their values,
and the distribution of original sample sizes (terciles at n = 63 and n = 171 over all rows). Row counts: 2,164
pairs, 786 original DOIs, at most 76 rows from one original DOI, 94 rows replicating OSC 2015. The six labels
of `reported_success`, listed once with no counts: descriptive only, failed, mixed, statistically successful but
flawed, successful, uninformative. **No outcome was counted, cross-tabulated or summarised.** The script was
run only with `--dummy`, which does not load the three outcome columns at all.

## Coding (fixed)

**Original p band** (from `pval_type_o` and `pval_value_o`):
- **S (strong):** exact p < .001, or reported as below a value of .001 or less (e.g. "p < .001", "p < .0001").
- **M:** exact .001 ≤ p < .01.
- **W (weak):** exact .01 ≤ p ≤ .05.
- **Unresolved:** reported only as "p < .01" (UNRES_M) or "p < .05" (UNRES_W). These could sit in a lower band,
  so they are left out of the main H1 contrast and are used in one sensitivity check.
- **NS (original not significant):** exact p > .05, or reported as above or below a value above .05. Excluded
  everywhere: a replication of a non-finding is not what the piece is about.
- **Missing:** no usable p. Excluded from H1 and from H2, because H2 is run only on originals known to be
  significant (S, M, W, UNRES_M, UNRES_W).

**Outcome (primary):** the replicating authors' own verdict, `reported_success`. Successful = 1. Failed = 0.
Mixed = 0 (it was not a clean success). Statistically successful but flawed, descriptive only, uninformative,
missing: excluded.

**Outcome (secondary):** the replication reached p < .05 (`pval_type_r`, `pval_value_r`; direction not checked,
because effect sizes come in 43 unconverted types). Exact p < .05 = 1, exact p ≥ .05 = 0, "below v" with
v ≤ .05 = 1, a report that cannot be placed = excluded.

**Unit and weight.** A row is one original–replication pair. Each original paper (`doi_o`) counts once in each
group: its replications in that group are averaged first, and the group's rate is the mean of those paper
averages. One heavily replicated paper (up to 76 rows) therefore cannot carry a band on its own.

**Interval.** Cluster bootstrap over original papers, 5,000 resamples, seed 20261026, percentile 95% interval.
Both groups are resampled together from the union of their papers.

## Hypotheses, prior, and the rule that scores each

**H1 (strength of the original result).** Originals in band S replicate more often than originals in band W.
- *Prior:* yes, by roughly 20 to 30 points (OSC: 63% at p < .001; 26% between .02 and .04; 18% above .04).
- **CUE HOLDS** if S − W ≥ 10 points and the interval's lower bound is above 0.
- **CUE REVERSED** if the interval's upper bound is below 0.
- **CUE NOT SHOWN** otherwise.

**H2 (size of the original sample).** Among significant originals, the largest third by `n_o` replicates more
often than the smallest third. Cut points are the 1/3 and 2/3 quantiles of `n_o` in the analysed set; top =
above the upper cut, bottom = at or below the lower cut. A rank correlation of `n_o` with the outcome is reported
alongside.
- *Prior:* no: a difference near zero or negative (OSC r = −0.150; Li rs = −0.02).
- **SIZE HELPS** if top − bottom ≥ 10 points and the interval's lower bound is above 0.
- **SIZE HELPS A LITTLE** if the lower bound is above 0 but the difference is under 10 points.
- **NO DETECTABLE ADVANTAGE** otherwise.

**Pre-registered subsets** (same rules, both hypotheses): (a) excluding the 94 OSC 2015 replications, because the
piece also quotes OSC's own table and FReD contains it; (b) only replications with no author in common with the
original (`author_overlap == 0`), because the piece's thesis is a re-run by someone else.

**Pre-registered sensitivity checks on H1:** mixed excluded instead of counted as 0; "successful but flawed"
counted as success; "p < .05" reports added to W; unweighted (each replication counts once); the secondary
outcome. These are reported in full; they do not change the verdict, which comes from the primary analysis.

**Power, from the planted control.** With outcomes faked at 65% for S and 45% for everything else (`--dummy
--plant`), the script returned +14 points [+6, +21], CUE HOLDS; with coin-flip outcomes (`--dummy`) it returned
+6 [−2, +14], CUE NOT SHOWN. Intervals run about ±8 points, so a true gap under about 10 points will usually read
as not shown. A "not shown" is not evidence of no difference, and the piece will not say it is.

**VOID** if the data hash does not match, or if either H1 group has fewer than 100 papers.

## What the piece will do with each result (committed now)

- **H1 CUE HOLDS:** the Re-check keeps "how strong was the original result?" as a question, and the piece quotes
  the FReD rates as its own measurement next to OSC's.
- **H1 CUE NOT SHOWN:** the piece says OSC's table showed the gap and the larger pool did not show it clearly; the
  question is kept but marked weak, and the headline moves to H2 or to the null itself.
- **H1 CUE REVERSED:** the question is dropped from the Re-check and the reversal is reported.
- **H2 NO DETECTABLE ADVANTAGE:** the piece may say a bigger original sample did not make a re-run detectably
  more likely to succeed in FReD, with the interval in the same sentence.
- **H2 SIZE HELPS (either level):** the piece must not say sample size fails as a cue. It reports the advantage and
  says FReD disagrees with the smaller OSC and Li analyses.
- **A subset gives a different verdict from the full set:** the piece reports both and leads with the more
  conservative one.

## Scoring

- **Script:** `fred_two_cues.py`, sha256
  `c86f14ba4b9be3a13994d5c2c86581e173e771b5ff31e8280953cbff820ddd19`. Python 3.9.6, numpy 2.0.2, pandas 2.3.3.
- **Command:** `python3 Projects/Content-Distribution/fred_two_cues.py`. It prints
  the verdicts by the rules above, so the scoring is mechanical. It writes `fred-two-cues-results.json`.
- **Order:** this file is locked with `prereg_lock.py lock` and its hash sent out as a `to:chatgpt` PR before the
  real run; `prereg_lock.py verify` runs first; the script's hash is checked against the value above before the run.
- **Results** go in a separate note, `RESULTS — survives-a-recheck — FReD two cues`, never in this file.
- **Independent check:** the Stage 5 blind fact-checker re-runs the command and compares the printed lines with
  the piece.