# PREREG — survives-a-recheck — Small-group check — 2026-10-27

**Why.** The held-out test (PREREG — survives-a-recheck — Held-out card test — 2026-10-27, lock e8d7a4da3d08)
judged sign 4, "a claim most people would doubt", by the doubt of about 125 lay readers per finding. A reader running
the card has only themselves and a few people to ask. Panel round 4 named this as the one remaining structural point.
This check asks whether the frozen card still sorts the same 27 findings when sign 4 comes from 1, 3 or 5 of the
same lay raters.

**What this is.** A pre-registered robustness analysis of data whose outcomes are already known, not a fresh test.
- **Known before writing:** the 27 outcomes, the crowd-level signs and the held-out result (act 5/5, try small 9/18,
  wait 0/4).
- **Unknown before the run:** how often small groups' doubt reproduces that sort.
- **Seen during the controls below:** agreement between a small group's sign 4 and the crowd's. It does not depend on
  outcomes: 67% at k = 1, 73% at k = 3, 77% at k = 5.

**Data.** Hoogeveen, Sarafoglou & Wagenmakers 2020, public at osf.io/wg9s3, copies in
`data/hoogeveen2020/` (sha256):
- `Data_MTurk_AN.csv` `a2f67a08…a918a7f`, `Data_Social_Media_AN.csv` `0ad59855…2aaa5ecc`,
  `Data_UvA_Lab_AN.csv` `2b8731bb…2f3c4e33`: one row per participant, a yes/no prediction per finding.
- `heldout_signs.csv` `08750083…68ba28eb4` (frozen in the held-out test) and `DescriptionAndStatisticsPerStudy.csv`
  `7fba68c0…1661240f40`.
- `lay_shares_description_only.csv` `c28265ca…52d52949482`: the crowd shares the held-out test used.

**Raters.** Same exclusions as the held-out test:
- description-only condition;
- finished the survey;
- no PhD in psychology;
- not familiar with ML2 or SSRP;
- a rater's answer on a finding counts only if they understood that description.

That leaves 127 raters, 90 of whom have a usable answer on all 27 findings. The script refuses to run unless the
rebuilt raters reproduce the held-out test's frozen shares and counts exactly; they do (checked during the controls).

**Sign 4 from a small group.** For a group of k raters (k odd, so no ties), sign 4 fires when fewer than half of them
predict the finding will replicate. This is the crowd rule applied to k people. Signs 1 (weak pass) and 2 (interaction)
stay as frozen. Actions: 0 signs → act, 1 → try small, 2–3 → wait.

**Draws.**
- **k:** 1 (a reader alone), 3 and 5.
- **Draws:** 10,000 per k, seed 20261027 + k.
- **Primary design (panel):** the same k raters judge all 27 findings, drawn without replacement from the 90 with
  complete answers. This is a reader asking the same few people.
- **Sensitivity design (independent):** k raters drawn afresh for each finding from everyone who answered it.
  Reported only.

**Per-draw rule.** The held-out test's own rule: act − wait ≥ 20 points, and act ≥ try small ≥ wait. A draw with an
empty act or wait group fails.

**Verdict per k (primary design).**
- **SMALL GROUP KEEPS THE SORT** at k if the per-draw rule holds in at least 75% of draws and the median act − wait gap
  is at least 20 points.
- **NOT KEPT AT k** otherwise.

**Reported for every k and both designs (no pass/fail):**
- pooled hold rates by action and the mean group sizes;
- the median gap with its 2.5th–97.5th percentile range;
- accuracy and coverage of the act/wait calls;
- the accuracy of the k-group's doubt alone;
- agreement with the crowd's sign 4.

## What the piece does with each result (committed now)

The smallest k that keeps the sort sets sign 4's wording on the card.
- **k = 1:**
  - The card asks the reader alone: "imagine hearing it plainly, for the first time: would you doubt it would hold
    up?"
  - The piece reports k = 3 and 5 beside it.
  - The card adds "asking two or three others is steadier" only if the rule share rises from k = 1 to k = 3.
- **k = 3 (not 1):**
  - The card says "describe it plainly to three people who haven't heard of it; if most doubt it would hold up, count
    the sign".
  - The piece says one person alone did not keep the sort.
- **k = 5 (not 1 or 3):** as for k = 3, with five people.
- **None:**
  - Sign 4 stays on the card, labelled "tested only as a crowd's judgment (about 125 people)".
  - The piece says a handful of people did not reproduce the sort.
  - §7's willpower verdict is presented as resting on that untested step.
- **Every outcome:**
  - The piece states that the 2015 odds used a different measure of the same sign: OSC's "surprising result", the
    average of at least three volunteers naive to the results (OSC codebook).
  - The worked cases used the author's own judgment of surprise, locked before the verdicts were read.

## Controls (run before this file was written; outcome column replaced, real outcomes never computed)

- `--dummy` (outcomes permuted): NOT KEPT at every k. Rule share 7% / 3% / 2%; median gaps −5 / −10 / −12.
- `--plant` (outcome = the crowd's majority call): KEPT at every k. Rule share 90% / 96% / 98%; median gaps +31 / +38 /
  +43.
- Outputs: `smallgroup-check-results.dummy.json` and `.plant.json`.

## Scoring

- **Script:** `smallgroup_check.py`, sha256
  `6f74bbeea4d235d8b98a2da719bfd50e0bc885b150d12c6491df12353e798d89`.
- **Order:** lock → `verify` → hash check → real run (`python3 smallgroup_check.py`).
- **Results:** appended to RESULTS — survives-a-recheck — FReD two cues — 2026-10-26 under their own heading.