# PREREG — survives-a-recheck — Held-out card test — 2026-10-27

**Why.** Panel rounds 2 and 3 found one remaining structural fault. The card's three-sign odds (69% / 45% / 17%) were
measured on the 2015 OSC sample, where two of the signs were first noticed. The fix every lens named was to run the
frozen card on findings it was not built from, with the surprise judgement made by ordinary readers blind to the
outcomes.

**Data.** Hoogeveen, Sarafoglou & Wagenmakers 2020 (AMPPS), public at osf.io/wg9s3: 27 findings re-run by the Social
Sciences Replication Project (17) and Many Labs 2 (10), none of them in the OSC 2015 sample. Copies are in
`data/hoogeveen2020/`:
- `DescriptionAndStatisticsPerStudy.csv`, sha256 `7fba68c0…40f40`
- the three raw rating files
- `heldout_signs.csv`, the frozen sign table, sha256 `0875008358e65f7476a769b45fbaed0717023d8853206a89cdeaea268ba28eb4`

**Signs (card v2 as locked, with each operationalised as below).**
1. **Weak pass:** the original p was not below .001, as published. Sources: the SSRP supplementary table's original-p
   column for 16 findings; for Aviezer and the 10 Many Labs 2 findings, FReD's original row paired with that re-run's
   DOI, and where FReD pairs several rows, the row whose claim matches Hoogeveen's description of the finding. The
   reported p is used as given (p = .001 exactly counts as weak, as in the FReD test). Result: 5 strong, 22 weak.
2. **Interaction:** the finding coded INTERACTION by a sealed Claude (`claude -p`, no tools), reading only the three
   description fields with study names removed and codes shuffled (seed 20261027). Prompt, key and answer are in the
   data folder. Result: 1 interaction (Lee & Schwarz 2010). Three borderline cases were flagged by the coder and left as
   coded.
3. **Doubted:** fewer than half of the lay readers in Hoogeveen's description-only condition predicted the finding
   would replicate. Readers kept: finished the survey, no PhD in psychology, not familiar with ML2 or SSRP, and
   understood the item; 116 to 127 per finding. The threshold was fixed before any share was looked at. Result: 4
   doubted. This is the card's surprise sign as a reader would actually use it: ordinary people's doubt, not
   volunteers' surprise ratings.

**Actions:** 0 signs → act (5 findings); 1 → try small (18); 2–3 → wait (4).

**Outcome:** Hoogeveen's `RS outcome`, which follows the SSRP and ML2 primary criterion: a significant effect in the
original direction. Its labels are 0 and 1. Coding check fixed now: Karpicke & Blunt (study 10) replicated (ledger
A02c), so it must read 1; if it reads 0, the coding is reversed.

**Disclosure.** While checking the success criterion in Hoogeveen's paper, after the sign table was frozen and hashed,
the author read a passage stating that Gneezy et al. 2014 (try small) and Tversky & Gati 1978 (wait) replicated. No
sign or rule was changed after that. The Tversky & Gati result counts against the card's "wait" call.

**Rule (direction, given the small groups).**
- **CARD SORTS ON HELD-OUT CLAIMS:** act − wait ≥ 20 points, and act ≥ try small ≥ wait.
- **NOT SHOWN ON HELD-OUT CLAIMS:** otherwise.
- Fisher's p is reported but cannot be decisive with 5 and 4 findings.
- The prior: act above wait. One "wait" finding is already known to have held, which works against the rule.

**Descriptive comparisons (reported, no pass/fail):** accuracy of a p-value-only rule, a lay-majority-only rule,
always predicting the commoner outcome, and the card's act-against-wait calls with "try small" abstaining.

## What the piece does with each result (committed now)

- **CARD SORTS ON HELD-OUT CLAIMS:**
  - The piece reports the held-out result beside the in-sample odds, with both sizes stated.
  - The card keeps its three signs and its action rule.
  - The odds on the card stay labelled as measured on the 2015 sample, with the held-out check named as the second
    test.
- **NOT SHOWN ON HELD-OUT CLAIMS:**
  - The card's action odds are presented as in-sample only.
  - The piece leads with the p-value cue (the one sign with an independent test).
  - Signs 2 and 3 are presented as "worth weighing; not yet shown to add on fresh claims", and the held-out result is
    reported in full.
- **Either way:** the comparisons are reported, including whether lay readers alone beat the card.

## Scoring

- **Script:** `heldout_card_test.py`, sha256
  `77ca669a6acd262f8235a9fd00b65607d0f856d804602699f9ea36b3d6ee9769`. It was dry-run only on a fake outcome column in a
  scratch copy.
- **Order:** lock → `verify` → hash check → run.
- **Results:** appended to RESULTS — survives-a-recheck — FReD two cues — 2026-10-26 under their own heading.