METHODS: A Fake Statistics Site Cost $50. ChatGPT Kept Citing It.
The Durability Curve

What is here
  plan.md      the written plan: the questions, the five-step rubric, the measures and the bar each had to clear.
               It was written before any measured answer was collected.
  results.md   the scored results, the audit, the second trace and the case-by-case table.
  experiment/  everything the scorer reads, in the layout it expects:
    questions.json          the 20 topics (the question template is in plan.md)
    answers.jsonl           the 40 raw answers, never edited
    collect.py              how the answers were collected (Claude Code CLI and Codex CLI with web search)
    make_blind.py           shuffles the answers and removes which assistant gave them (seed 20261108)
    blind/RUBRIC.md         the rubric the tracers were given, and nothing else about the test
    blind/answers_blind.md  the shuffled, unlabelled answers the tracers saw
    blind/key.json          which assistant gave which label
    extract.jsonl           the statistic and link the first tracer took from each answer
    traces.jsonl            the first trace (merged from traces/traces_1..4.jsonl)
    traces2/                the second, independent trace
    audit.jsonl             the author's re-trace of 10, written before the tracers' classes were seen
    blind/third_read_input.json, third_read.jsonl   what the third reader saw, and its final calls
    factcheck.py, factcheck.jsonl   the fact-check prompt sent to the other assistant, and every reply
    score.py                prints every measure in the essay and in results.md
    score-output.txt        what score.py printed when this bundle was built

How to run
  From experiment/: python3 score.py  (Python 3, standard library only). It should print score-output.txt.
  collect.py and factcheck.py call the two assistants again; their answers will differ from day to day.

sha256 of every file in this folder (README.txt itself excepted)
66be28f31d3849a8fdd31037ac87142107324491c800504b34ec667f7d15e6a4  experiment/answers.jsonl
d76da930d00ebb7a48604b694451b69b93390e5bb7b6bd68d1bf45148c7357c4  experiment/audit.jsonl
62167c0737af6e2b8181ea2800b96996273b0de7c00fd589c7c2d622c287b29f  experiment/blind/RUBRIC.md
7cc5947f112a54d23f82d414687304e3d0aaac289130db8d809872914f2700bd  experiment/blind/answers_blind.md
755e05c3af00099d323a9f2ff0bdc908571478f5e84bf513e451bcfea4524ef9  experiment/blind/audit_labels.txt
e38acb60bd62c1b77a03849014d9a26136815314ec654796e39b846140cc7871  experiment/blind/key.json
2a074a0a86034d312c676718795dc8b27a9bb8efc46ea220f3306b10864ea699  experiment/blind/third_read_input.json
1bdd1403825354956c571c9c4c09968c3aae208938066c2e347fb5ecc116560e  experiment/collect.py
3008163c34a85835f613693a2801f0944d00bc9f616337ee562c0cd0f9029109  experiment/extract.jsonl
e9de94b7ab21712d7063b20c9b58472809abc85863c76392cf0f2b5989b6e3de  experiment/factcheck.jsonl
1e13e274ac35098bed06ea2921bc2ae27fea32f9a5041b4cfc33d1e5b0884d00  experiment/factcheck.py
322d44b7630348225d3aaaa2c300d4affa1e8c6eb1d7556d330198b22987a8d0  experiment/make_blind.py
c7f7e9ef1261d0a6bf96f5b9237fdb6f124f547c430f1f0003d95aa55de7ecd9  experiment/questions.json
a67d8bfb1f8d086c813a21a4e7e7bfd7a0bce67621ce13db82d1eca7bc615e79  experiment/score-output.txt
7965f49d4eff4b06e02eb7c07ee2dae709e9b3b30dea8ff65fae535d56052e20  experiment/score.py
c639cf13118e950a945a42aaf454d817bb4af98bddefd7875262a9af4f62abf3  experiment/third_read.jsonl
1b789bb8384a8d5c50d327835722ed43de13c3099222b6a2129c05f55fc73dc0  experiment/traces/traces_1.jsonl
34271671a2fe5b1e1f1740d5b2789891e4703978e5d350ece41a8ed712249ea5  experiment/traces/traces_2.jsonl
f8297d5414ed5e0fa0a7d639838b03a6e16a9877ebc34f15c8010a5e4b18a7e2  experiment/traces/traces_3.jsonl
f16ccced5d2a6f340e71decef0bb7852d846591d69db3060779a2e4a77bbecf9  experiment/traces/traces_4.jsonl
36f0b60b37c1bbcf7fda8a0bd1228c04b1be115ddc5cb354b2a1292a08613d1c  experiment/traces.jsonl
6a26e723ae5f73167a15e20ab0938d04fb8f5211458e3e307dab7b08dc6cede5  experiment/traces2/traces2_1.jsonl
85b80f5a0b40264769fcb488a73fdcb161eba3198f007740bda4bc7186d4cbeb  experiment/traces2/traces2_2.jsonl
01343db14287e33930e78c55e9c41983a33219f79d08e9ba15e894576b150eb4  experiment/traces2/traces2_3.jsonl
7ffc6e126bf824c5f5d2d8867390532509eebeecf2db608043743819f178e25c  experiment/traces2/traces2_4.jsonl
5b67559713bdb19e734be8d45f2476088b88da613b0be47b0a4008e7a706f56e  plan.md
305b8eb1e8e120279a3b91249e2c4f9e8df63fe67e3bba640ed1640cbec4e2c4  results.md