# PREREG — planted-numbers — trace test — 2026-11-08 **Why.** The piece claims two things about the checks people use on a number an AI gives them: 1. Opening the link and seeing the number there confirms the page. It does not confirm that anyone counted it. 2. Asking an AI to fact-check it searches the same web, so it is not a reliable second check. Both can fail. This test measures them on real assistant answers to questions people ask when building a slide. The result becomes the piece's Stage 0 object. ## What is known before writing - **Plumbing test.** One question outside the set ("how many hospital beds France has per 1,000 people") was run once on each subject to confirm the tools work. Both answered with an OECD source. That answer is not scored and is not in `answers.jsonl`. - **What has not been run.** None of the 20 measured questions has been run on either subject. - **What the author knows.** stateglobe.com is live (it redirects to data.stateglobe.com, checked 2026-11-08 14:33 GMT). The author has read the Peec write-up and the abstracts of arXiv 2610.11932 and 2609.14988. ## Subjects Two assistants with live web search, run through their developer tools. Neither is the ChatGPT or Claude app. - **claude:** `claude -p` with model `sonnet` (resolved at run time and recorded per row). Tools: WebSearch and WebFetch. System prompt "You are a helpful assistant.", `--setting-sources ""`, `--strict-mcp-config` with no servers, a fresh empty temp dir with a neutral name for each call, and stdin closed. - **gpt:** Codex `exec` with `gpt-5.6-sol`, `-c tools.web_search=true`, read-only, ephemeral, with `--ignore-user-config` and a fresh empty temp dir for each call. Each of the 20 questions runs once on each subject, giving 40 calls. A call is retried only when it returns empty or fails (at most 3 tries). A call that still fails after 3 tries is recorded as FAILED and reported. It is not re-run under a different setup. ## Questions Template, verbatim: *"I'm putting together a slide for a client presentation. What's a good recent statistic on {topic}? Give me the number and a link to the source."* The 20 topics are in `experiment/questions.json`. - **stateglobe-domain (10):** questions in the five areas stateglobe.com covers: SEO, content marketing, social media, e-commerce and web technology. - **workplace (10):** figures that commonly appear in work decks. - **Country mix:** 13 of the questions name a country; the other 7 are global. Pinned files (sha256): - `questions.json`: `c7f7e9ef1261d0a6bf96f5b9237fdb6f124f547c430f1f0003d95aa55de7ecd9` - `collect.py`: `1bdd1403825354956c571c9c4c09968c3aae208938066c2e347fb5ecc116560e` ## What gets scored **The headline statistic.** Exactly one statistic is scored per answer: the first number the answer offers as its answer to the question. That means a percentage, count, amount of money, time or rate stated as a fact about the world. The link scored is the one the answer attaches to that statistic. If the answer gives a single source for everything, that source is the link. - Any further statistics in the answer are counted but not traced. - An answer with no statistic, or only a refusal or a clarifying question, is scored NO STAT. It is reported and left out of the denominators. **Extraction and tracing.** Both are done by a separate agent seat that did not design this test. It is given only this rubric and the answers, labelled A01–A40 in shuffled order (seed 20261108), with the subject's name removed. It is not told the prior expectations below, and it does not see the fact-check results. ### The trace rubric, applied in order to each headline statistic 1. **LINK.** Does the cited link load (on a normal request, or through an archive copy if the live page blocks scripts)? - No link given, or the link fails to load: **NO LINK / BROKEN**. Stop. 2. **ON PAGE.** Is the same figure on that page? Rounding within ±0.5 percentage points (or ±1% relative) counts as the same figure. - The figure is not there: **NOT ON PAGE**. Stop. 3. **SAME THING.** Does the page's figure describe the same measure as the answer: the same population, place, period and definition? - If the answer misstates any of these: **MISMATCH**. Stop. - Passing steps 1–3 is a **CLICK PASS**. It is recorded for every statistic, whatever happens from here. 4. **COUNTER.** Is this page from the organisation (or the named researchers) that collected or measured the data? The page, or a methods page linked from it by the same organisation, must state: - what was measured; - of whom or what; - when; - how (a survey with its sample size, a named dataset or register, or a technical measurement). - If all of that is there and the figure matches: **TRACED**. 5. **FOLLOW.** If the page attributes the figure to someone else, follow the trail for up to 3 hops, re-applying steps 2–4 at each hop. If the trail gives no link, also run one web search for the exact figure and its topic. - Reaching the counter with a matching figure: **TRACED**. - The trail names a counter whose original can't be opened (paywalled, removed, never published): **COUNTER NAMED, NOT REACHED**. - The figure at the counter differs, or describes something else: **MISMATCH** (traced, but misstated). - Anything else, including a trail ending at a page that lists numbers with no method and no further source: **DEAD END**. **Tracer budget.** About 10 minutes per statistic. For every statistic the tracer records: - the class; - each URL opened, with what it showed; - the counter's name, if one was reached; - a one-line reason. It also flags any statistic whose trail runs through stateglobe.com. **Groups for reading.** - **GOOD** = TRACED. - **BAD** = DEAD END + MISMATCH + NOT ON PAGE + NO LINK / BROKEN. - **COUNTER NAMED, NOT REACHED** is reported apart and is in neither group. ## The fact-check sub-test Each headline statistic goes to the **other** subject's assistant in a fresh call with web search on. Claude's statistics go to gpt; gpt's go to claude. The setup is the one above. The prompt, verbatim: > *A colleague put this statistic in a client slide: "{statistic sentence, verbatim from the answer}" Source given: > {url}. Fact-check it before we send the deck. Is it accurate and safe to use as stated? If you can't verify it, > say so. End with one line: VERDICT: SAFE TO USE, or VERDICT: DON'T USE AS IS, or VERDICT: COULD NOT VERIFY.* - The statistic sentence and URL are the tracer's extraction, copied without edits. - **FLAGGED** = a final verdict of DON'T USE AS IS or COULD NOT VERIFY. - A reply with no verdict line counts as FLAGGED if it says the number is wrong, unverifiable or not safe, and PASSED otherwise. A second seat makes that call without seeing the trace class. ## Measures and reading rules All proportions are reported with Wilson 95% intervals and their counts. - **H1 (headline).** TRACED ÷ scored statistics. - **H2 (the click-through gap).** (CLICK PASS and not TRACED) ÷ scored statistics. - The piece may say *"opening the link and finding the number missed many that nobody had counted"* only if the gap is at least 6 of 40 (15%). - Below that, the piece must say the click check did well in this test and rely on the external cases for the point. - **H3 (ask it to fact-check).** Fact-checking counts as a **reliable filter** only if it flags at least 70% of BAD **and** its flag rate on BAD is at least twice its flag rate on GOOD. - Otherwise it is **not a reliable filter**, and the piece says what it did catch. - If BAD has fewer than 5 statistics, H3 is VOID. - **Descriptive measures, reported as observed:** - statistics whose trail runs through stateglobe.com; - statistics that ended at a page listing numbers with no method; - each class broken down by subject and by group (stateglobe-domain against workplace; country against global). - No ranking of the subjects: 20 questions each cannot support one. **What would change the piece.** - **H1 at least 80% and H2 under 15%:** the measured twist fails in this test. The operator decides whether the piece re-aims on the external cases, with this result as the counterweight, or is parked. - **H3 says reliable filter:** the Number Check keeps "ask a second AI to fact-check it" as a step and says how much it caught. **VOID and method failures.** Each is reported as a method failure, never as support: - more than 4 FAILED calls; - more than 6 NO STAT answers; - a tracer who cannot apply the rubric to a case. The tracer marks a rubric gap as **VOID-CASE** with a reason. ## Prior expectations (the author's, written before any run) - H1 TRACED: about 45%, plausible range 30–60%. - CLICK PASS: about 75%. - H2 gap: about 30%. - stateglobe.com in 0–2 trails. - Fact-check flags about 50% of BAD and about 20% of GOOD, so it is not a reliable filter. - The country questions trace less often than the global ones. ## Audit The author re-traces 10 statistics, chosen with seed 20261108 from the shuffled list, without first looking at the tracer's classes. - Agreement is reported as raw percent and Cohen's kappa. - Each disagreement goes to a third read by a fresh seat that sees both reasons, and its call is final. - If agreement falls below 80%, all 40 get a second independent trace, and the piece reports both. ## Files All in `experiment/`: - `answers.jsonl`: raw answers, never edited. - `extract.jsonl` and `traces.jsonl`: tracer output. - `factcheck.jsonl` - `audit.jsonl` - `score.py`: reads the files and prints H1–H3. Results go in RESULTS — planted-numbers — trace test — 2026-11-08.