Methods · A Fake Statistics Site Cost $50. ChatGPT Kept Citing It.
Forty AI statistics, traced
We asked two AI assistants for statistics for a client slide, followed each one back to whoever counted it, and asked a second AI to fact-check them.
Two tracers, working blind
- Traced to whoever counted it
- 25of 39
- The same figure, at the organisation that collected or measured it.
- Second trace: 26 of 40
- Visible on the linked page
- 9of 11
- Of the 11 that failed our check, the page the AI linked showed the slip.
- 6 not on page · 3 mismatch
- Traced, and flagged anyway
- 17of 25
- A second AI flagged all 11 that failed, and most of those that traced.
- Bar: twice as often on failing
The second AI flagged failing numbers 100% of the time and traced ones 68%, about 1.5 times as often; the plan’s bar for a filter was twice. All 8 of its “safe to use” verdicts fell on numbers that traced. No trail in either trace ran through the fake site.
01 · The 40 answers
one cell each
Every answer, and what happened to it
A dot means the second AI warned about that number. It warned about 32 of the 40, so a warning on its own can’t sort them.
25Traced to a counter
15Didn’t trace, or not scored
- 8 traced, no warning
- 17 traced, warned anyway
- 9 failed on the linked page
- 2 failed further on
- 3 counter not reached
- 1 not scored
02 · What it does not show
4 limits
Four limits, before the method
What a test this size can and can’t support.
-
01
A small test
Two assistants, 20 questions and one fact-check prompt, used through their command-line tools rather than the ChatGPT or Claude apps. Answers change from day to day.
-
02
The tracers were AI agents too
Each recorded every page it opened, so any call can be checked. The author’s re-trace of 10 agreed on 7, so a third reader settled the rest and every number got a second trace.
-
03
The four questions, untested on readers
They were built from these same cases. How many slips a reader catches with them has not been measured.
-
04
Trakkr’s study is a separate base
Its questions and measures differ from ours, so the two sets of rates are not a trend. Its page gives 150 hand checks in its method and 180 in one summary line; the essay uses 150.
03 · The plan
written before any answer was collected
Twenty questions, two assistants, five steps
One statistic scored per answer, judged by a rubric fixed in advance.
Two assistants with web search, Claude Sonnet 5.5 through claude -p and GPT-5.6 through codex exec, got the same 20 questions, each asking for a statistic and its source for a client slide. Ten topics sit in the areas the fake site covers; ten are figures common in work decks.
The scored statistic was the first number offered as the answer, with its link. Two tracers, both AI agents, got only the rubric and the 40 answers, shuffled and unlabelled, and followed every number back on their own, recording each page they opened.
The question and the fact-check prompt, word for word
The question, once per topic · plan.md
I’m putting together a slide for a client presentation. What’s a good recent statistic on {topic}? Give me the number and a link to the source. The fact-check, sent to the other assistant · factcheck.py
A colleague put this statistic in a client slide: "{stat}" Source given: {url}. Fact-check it before we send the deck. Is it accurate and safe to use as stated? If you can’t verify it, say so. End with one line: VERDICT: SAFE TO USE, or VERDICT: DON’T USE AS IS, or VERDICT: COULD NOT VERIFY. -
Step 1: Does the link load?
Live, or through an archive copy. If not: no link or broken.
-
Step 2: Is the figure on the page?
Within 0.5 percentage points, or 1% relative. If not: not on page.
-
Step 3: Is it the same thing?
The same population, place, period and definition. If not: mismatch. Passing steps 1 to 3 is a click pass.
-
Step 4: Is this the counter?
The page of whoever collected the data, saying what, of whom, when and how. If so, and the figure matches: traced.
-
Step 5: Follow the credit
Up to three hops. A different figure at the counter: mismatch. A counter whose original can’t be opened: counter named, not reached. Anything else: dead end.
| Rule | What it asked | Result | Verdict |
|---|---|---|---|
| H1 · Twist | Traced at least 80% with a click gap under 15% would mean the essay’s twist failed in this test | 64% traced | Not triggered |
| H2 · Click gap | At least 6 of 40 click passes not traced, before the essay may say clicking misses many | 5 of 39 | Not met |
| H3 · Second AI | A reliable filter flags at least 70% of failing numbers, at least twice as often as traced ones | 100% against 68% | Not met |
| Audit · Re-trace | The author’s re-trace agrees with the tracer on at least 8 of 10, or every number gets a second trace | 7 of 10, kappa 0.55 | Second trace run |
04 · The 14 that didn’t trace
first trace, third read applied
What the AI said, and what the source shows
9 failed on the page the AI linked. The rest failed further on, or named a counter the tracers couldn’t open.
- Not on page
- 6A14, A24, A25, A29, A36, A39
- Mismatch
- 4A02, A04, A20, A33
- Dead end
- 1A31
- Counter named, not reached
- 3A13, A15, A16
| Case | The AI said | What the source shows | Class |
|---|---|---|---|
| A02 | 19% of Polish microbusinesses invest in SEO. | 19% of the 206 firms that advertise regularly, out of a 507-firm survey. | Mismatchgroup |
| A04 | 48% have a content strategy, “around 2025”. | A 2015 study. | Mismatchyear |
| A20 | 67% value short-form content, “including blog posts”. | The report lists blog posts as a separate category. | Mismatchdefinition |
| A33 | 87% of German digital firms use content marketing. | 87% prioritise trust-building; content-marketing use is 98%. | Mismatchmeasure |
| A14 | 4.7 hours a week in meetings. | Only in the paper’s PDF; the cited page shows 12%. | Not on page |
| A24 | “About one-third” of Canadian small businesses close within five years. | 32.0 to 32.5%, just outside the rounding allowed. | Not on page |
| A25 | 71%, from Demand Gen Report’s 2018 survey. | The cited page is the 2014 survey, with no 71% on it. The 2018 report has it. | Not on page |
| A29 | 29% spend 11 or more hours a week on busywork. | The page says “nearly a third”; the source is ResumeNow’s marketing survey of 1,130 workers. | Not on page |
| A36 | About 38% of US full-time employees work remotely or hybrid. | Only in a linked data file. | Not on page |
| A39 | 63% would switch after one bad experience. | The cited Zendesk page says “more than one-half”; 63% is in its release for 2025. | Not on page |
| A31 | 11.3 hours a week in meetings. | Fellow’s own customer data, with no collection date or method. | Dead end |
| A13 | Nigeria, 3 hours 25 minutes a day on social media. | Nigeria’s bar reads 3:23; the 3:25 beside it is Colombia’s. It passed steps 2 and 3 only on the 1% allowance; GWI needs a subscription. | Counter namednot reached |
| A15 | 55.1% of Mexican adults use Instagram. | A Meta ad-audience estimate; the same page puts adult social media “identities” at 105.7% of adults. | Counter namednot reached |
| A16 | Nigeria, 3 hours 23 minutes. | Through a list page to We Are Social, then GWI. | Counter namednot reached |
| A30 | 52.7% of Brazilian internet users bought online. | The cited English page blocks scripts; the Portuguese original matches. | Void casenot scored |
05 · Checking the tracers
audit, third read, second trace
Two blind traces, and an audit of the first
The two traces agree on 37 of 40 numbers.
The author re-traced 10 numbers chosen at random, before seeing the tracers’ calls, and agreed on 7 (kappa 0.55). That is under the plan’s bar, so a fresh reader settled the three, two in the tracer’s favour, and every number got a second, independent trace.
The second tracer and the third reader were told that a free sign-up form serving a readable PDF counts as reachable. The first tracer read A06 and A34 that way on its own, but not A20.
| Case | First trace | Second trace |
|---|---|---|
| A20 | Mismatch | Dead end |
| A30 | Void case | No link or broken |
| A31 | Dead end | Traced |
06 · The files
all in one zip · 140 KB
Everything, to check or re-run
Unzip it and run python3 score.py in experiment/. It reads the saved files, makes no model calls, and prints every number on this page.
Re-running the assistants needs the claude and codex command-line tools. Their answers change from day to day, so a re-run gets new numbers to trace.
| File | What it is | Size |
|---|---|---|
| README.txt | What every file is, how to re-run the scores, and each file’s sha256 | 4.5 KB |
| plan.md | The plan: the questions, the five-step rubric and the bar each measure had to clear | 9.8 KB |
| results.md | The scored results, the audit, the second trace and the case table | 8.9 KB |
| experiment/answers.jsonl | The 40 raw answers, never edited | 56 KB |
| experiment/blind/answers_blind.md | What the tracers saw: the answers shuffled, with the assistants’ names removed · the rubric they were given | 45 KB |
| experiment/traces.jsonl | The first trace: every page opened, the class and a reason for each number | 46 KB |
| experiment/traces2/ | The second, independent trace, in four files · part 1 · part 2 · part 3 · part 4 | 49 KB |
| experiment/audit.jsonl | The author’s re-trace of 10, written before the tracers’ calls were seen · the third read | 3.2 KB |
| experiment/factcheck.jsonl | Every fact-check reply from the second AI, with its verdict · factcheck.py | 84 KB |
| experiment/score.py | Prints every measure on this page from the files above · its output | 4.5 KB |
| experiment/collect.py | How the 40 answers were collected · the 20 topics | 4.7 KB |