# RESULTS — planted-numbers — trace test — 2026-11-08

## Data

All 40 calls returned an answer: 0 FAILED, 0 NO STAT, and no leak markers (folder name, operator name or tool
markup). Models as recorded:
- claude: `claude-sonnet-5-5` (with `claude-haiku-5-5` used inside its web tool)
- gpt: `gpt-5.6-sol`

File hashes (sha256, first 12 characters):

| File | Hash |
|---|---|
| answers.jsonl | 66be28f31d38 |
| extract.jsonl | 3008163c34a8 |
| traces.jsonl | 36f0b60b37c1 |
| third_read.jsonl | c639cf13118e |
| factcheck.jsonl | e9de94b7ab21 |
| audit.jsonl | d76da930d00e (written 14:47 GMT, before any tracer class was seen) |
| traces2_1 | 6a26e723ae5f |
| traces2_2 | 85b80f5a0b40 |
| traces2_3 | 01343db14287 |
| traces2_4 | 7ffc6e126bf8 |

## Audit and the second trace (prereg § Audit)

**Author's audit against the first trace:** 7 of 10 agreed (70%, Cohen's κ 0.55). That is below the 80% bar, so
two things followed, as the prereg requires:
1. The three disagreements went to a fresh third reader, whose calls are final.
2. All 40 statistics got a second independent trace.

**The third read:**
- **A06 and A34 (IBM breach cost): TRACED, siding with the tracer.** The author had stopped at IBM's landing page.
  The free form serves the report PDF, whose method (602 organisations, March 2025–February 2026) is on p.68.
  The author's audit was wrong on both.
- **A20: MISMATCH, siding with the author.** Demand Gen Report's own PDF lists blog posts as a separate category
  from short-form content.

**First trace against the second:** 37 of 40 agree. The three that differ:

| Label | Primary | Second trace |
|---|---|---|
| A20 | MISMATCH (third read) | DEAD END |
| A30 | VOID-CASE | NO LINK / BROKEN |
| A31 | DEAD END | TRACED |

**Clarification given to the second trace and the third reader (disclosed, not a departure).** "Can't be opened"
in rubric step 5 was read so that a free registration form serving a PDF the tracer can open counts as reachable.
The first trace was not given this line. It applied the same reading on A06 and A34 unprompted, but not on A20.

## Pre-registered measures

Primary = the first trace, with the third read applied. Intervals are Wilson 95%.

| Measure | Primary | Second trace | Prior |
|---|---|---|---|
| **H1 TRACED** | **25/39 (64%, 48–77%)** | 26/40 (65%, 50–78%) | ~45% (30–60) |
| CLICK PASS | 30/39 (77%, 62–87%) | 31/40 (78%, 62–88%) | ~75% |
| **H2 gap** (click pass, not traced) | **5/39 (13%, 6–27%)** | 5/40 (12%, 5–26%) | ~30% |
| H3 fact-check flags BAD | 11/11 (100%, 74–100%) | 11/11 | ~50% |
| H3 fact-check flags GOOD | 17/25 (68%, 48–83%) | 18/26 (69%, 50–83%) | ~20% |
| **H3 verdict** | **NOT A RELIABLE FILTER** | NOT A RELIABLE FILTER | not reliable |
| stateglobe.com in a trail | 0 | 0 | 0–2 |
| List page in a trail | 6 | 5 | — |

### Reading rules, as they bind

- **H2 is below 6 in both traces (5).** The piece **may not** say that opening the link and finding the number
  missed many in this test. It must say the click check, as defined (the number is on the page **and** describes
  the same thing), did well here, and it must rely on the external cases (stateglobe, Trakkr) for the point that a
  number on a page is not a counted number.
  - Of the 9 click failures, all were caught on the cited page itself: 6 NOT ON PAGE (5 in the second trace) and
    3–4 MISMATCH.
- **H1 is 64%, under the 80% that would trigger an operator re-aim.** No operator ruling is needed.
- **H3: not a reliable filter.** Asking the other vendor's assistant to fact-check flagged every BAD statistic, but
  also two-thirds of the GOOD ones. Its flag rate on BAD (100%) is about 1.5 times its rate on GOOD (68%), short of
  the 2× bar. The piece says so, and says what it did catch.

### Descriptive breakdowns (no ranking; n too small)

| Breakdown | Group | Traced |
|---|---|---|
| Subject | claude | 13/20 |
| Subject | gpt | 12/19 |
| Question group | stateglobe-domain | 11/19 (58%) |
| Question group | workplace | 14/20 (70%) |
| Scope | country | 15/23 (65%) |
| Scope | global | 10/16 (62%) |

- The prior that country questions trace less often was **not** borne out.
- **Erratum (prereg text):** the prereg says 13 questions name a country and 7 are global. `questions.json` has
  12 country and 8 global. The pinned file governs, and these breakdowns use it.

## What failed, case by case (primary classes)

| # | Answer said | What the source shows | Class |
|---|---|---|---|
| A02 | 19% of Polish microbusinesses invest in SEO | 19% of the 206 firms that advertise regularly, out of a 507-firm survey | MISMATCH (group) |
| A04 | 48% have a content strategy, "around 2025" | A 2015 study | MISMATCH (year) |
| A20 | 67% value short-form content, "including blog posts" | The report lists blog posts as a separate category | MISMATCH (definition) |
| A33 | 87% of German digital firms use content marketing | 87% prioritise trust-building; content-marketing use is 98% | MISMATCH (measure) |
| A14 | 4.7 hours a week in meetings | Only in the paper's PDF; the cited page shows 12% | NOT ON PAGE (right number, deeper link) |
| A24 | "About one-third" of Canadian small businesses close within 5 years | 32.0–32.5% | NOT ON PAGE (rounding just outside tolerance) |
| A25 | 71%, from Demand Gen Report's 2018 survey | The cited page is the 2014 survey, with no 71%; the 2018 PDF does say 71% (found by the fact-checker) | NOT ON PAGE (wrong link) |
| A29 | 29% spend 11+ hours a week on busywork | The page says "nearly a third"; the source is ResumeNow's marketing survey of 1,130 workers | NOT ON PAGE |
| A36 | About 38% of US full-time employees remote or hybrid | Only in a linked data file | NOT ON PAGE (right number, deeper link) |
| A39 | 63% would switch after one bad experience | The cited Zendesk page says "more than one-half"; 63% is in the 2025 release | NOT ON PAGE (wrong edition) |
| A31 | 11.3 hours a week in meetings | Fellow's own customer data; no collection date or method | DEAD END (second trace: TRACED) |
| A13 | Nigeria, 3h25m a day | Nigeria's bar is 3h23m; 3h25m is Colombia's bar beside it. It passed steps 2–3 only on the ±1% tolerance | COUNTER NAMED, NOT REACHED (GWI subscription) |
| A15 | 55.1% of Mexican adults use Instagram | Meta ad-audience estimate; the same page puts adult social media "identities" at 105.7% of adults | COUNTER NAMED, NOT REACHED |
| A16 | Nigeria, 3h23m | Via a list page (Intelpoint) to We Are Social, then GWI | COUNTER NAMED, NOT REACHED |
| A30 | 52.7% of Brazilian internet users bought online | The cited English IBGE page is blocked; the Portuguese original matches | VOID-CASE |

## Exploratory (not pre-registered; labelled wherever used)

1. **The fact-checker's reasons, read by the author.** On the 17 GOOD statistics it flagged, most of its reasons
   are scope notes that are true but small:
   - BLS counts "wage and salary workers at work", where the answer said "employees";
   - W3Techs' "all websites" is its surveyed web;
   - PayPay counts registered accounts.

   **At least two objections were wrong against the cited source, checked by the author at the primary on
   2026-11-08:**
   - **A05 (Verint).** The report says "78% are likely to switch to a competitor after a terrible customer
     experience" (p.12), which matches the answer. The checker went by Verint's press release instead ("consider
     switching").
   - **A10 (PowerReviews).** The page says "The greatest portion of shoppers – 75% – read between one and 25
     reviews". The checker summed a distribution and called 75% a conflation.

   On BAD statistics its reasons were often right, and twice it found the true source the answer had mislinked
   (A25: the 2018 PDF; A39: Zendesk's 2025 release).

   Reading: the verdict can't sort good from bad, and its reasons are useful leads that still need the source.
   This is one author's read, not a blind classification. Do not put a number on it in the piece.
2. **The "right number, wrong link" pattern.** In at least 4 of the 6 NOT ON PAGE cases (A14, A25, A36, A39), the
   figure exists at a source the answer didn't link. A number that isn't on the cited page is unchecked, which
   doesn't make it false.

## Comparison with prior art (not pre-registered)

**Trakkr Study 013** (28 Oct 2026): 5 assistants, 1,285 statistics.

| Measure | Trakkr | Here |
|---|---|---|
| Cited page had the figure | 82% | 77–78% click pass |
| Reached a matching study | 40% | 64–65% traced |

These are different measures and question sets. Ours asked for one statistic for a client slide "with a link to
the source", which may push assistants toward primary pages; Trakkr sampled every statistic in answers to 200
questions. **Do not present the two as a trend or as directly comparable.** Our count of COUNTER NAMED, NOT REACHED
(3/39) is far below Trakkr's 40% "can't check", which fits the same explanation.