{
 "ledger": {
  "title": "Claim Ledger: The JSON parsed. Is it right?",
  "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/",
  "essay": {
   "title": "The JSON parsed. Is it right?",
   "url": "https://durabilitycurve.com/blog/the-json-parsed-is-it-right/",
   "published": "2026-11-15"
  },
  "last_checked": "2026-11-15",
  "format": "current",
  "counts": {
   "claims": 71,
   "by_status": {
    "EXECUTED": 52,
    "VERIFIED": 18,
    "REPORTED": 1
   },
   "verified_at_primary": 18,
   "removed": 2
  },
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "attribution": "The Durability Curve (Harry Floyd), https://durabilitycurve.com/claims/the-json-parsed-is-it-right/"
 },
 "rows": [
  {
   "id": "C01",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C01",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Our grader piece sets out a five-step check on twenty items with a scoring sheet, counts false passes and false fails separately, and says to check the key before you blame the grader.",
   "quote": "Here is a five-step check on twenty items for yours. … five steps, a scoring sheet … Count false passes and false fails separately. … check the key before you blame the grader",
   "removal_reason": "",
   "source": "The Durability Curve, Your AI Grader Is Only as Good as Its Answer Key (first-party)",
   "source_url": "https://durabilitycurve.com/blog/your-ai-grader-answer-key/",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C02",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C02",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "On 20 Federal Register notices with a strict schema, all three LLM runs returned schema-valid JSON for all 20 documents, and none passed the bar.",
   "quote": "HEADLINE: 3 of 3 LLM runs schema-valid on 20 of 20, 0 of 3 passed",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C03",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C03",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The Federal Register's own structured data passed on the three fields we could score it on (on our key).",
   "quote": "federal_register_api_metadata overall: PASS",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C04",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C04",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "One run per setup on 8 October 2026; an AI agent built the answer key from the Federal Register text; a second agent checked all 120 cells on 10 November and found no disagreement; no person has checked them.",
   "quote": "I read the preamble … of all 20 documents and compared all 120 cells … 0 disagreements",
   "removal_reason": "",
   "source": "our own independent check (first-party)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C05",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C05",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The documents are ten Rules and ten Proposed Rules from 28 and 29 September 2026, via the public Federal Register API.",
   "quote": "C05 OK types Rule 10, Proposed Rule 10; dates 28 Sep (15), 29 Sep (5)",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C06",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C06",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Ten are FAA airworthiness directives on one template.",
   "quote": "C06 OK 10 titles start 'Airworthiness Directives'",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C07",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C07",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Two are correction notices, and one of those has no AGENCY: or ACTION: line.",
   "quote": "C07 OK IRS C1-2026-18574 has no AGENCY:/ACTION: line",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C08",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C08",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "A HUD fee announcement calls its date an applicability date.",
   "quote": "DATES: Applicability date: October 28, 2026.",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package); text of FR 2026-19743",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C09",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C09",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "An EPA proposal lists three deadlines in its DATES section.",
   "quote": "C09 OK EPA 2026-19774 DATES names 12 Nov (comments), 28 Oct (OMB/PRA), 5 Oct (hearing request)",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C10",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C10",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Prompted for JSON, gemini-2.5-flash-lite wrapped all twenty replies in a code fence, so json.loads failed twenty times out of twenty.",
   "quote": "C10 OK naive Gemini: fenced 20/20, json.loads failed 20/20",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C11",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C11",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "With the fences stripped, thirteen of twenty still fail a typed schema, every one on a date; the model wrote October 28, 2026 in one row and 2026-10-28 in another.",
   "quote": "C11 OK after stripping fences 13/20 fail the schema, all on dates; 19745='October 28, 2026', 19741='2026-10-28'",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C12",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C12",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "gpt-5-nano with the same prompt: all twenty parsed, nine failed the schema, all on dates; on the IRS correction it put October 26, 2026 in both date fields.",
   "quote": "C12 OK naive nano: 20/20 parsed, 9 failed the schema (dates only); IRS both dates 'October 26, 2026'",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C13",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C13",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "A MARAD correction's DATES: line is a bare September 28, 2026; gpt-5-nano returned it as the effective date, valid and wrong by our key, which leaves it empty.",
   "quote": "C13 OK MARAD: naive nano passed with effective_date 2026-09-28; key empty; DATES line bare",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C14",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C14",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "OpenAI's docs say Structured Outputs constrains completed, non-refusal model output to your supplied JSON Schema.",
   "quote": "Structured Outputs constrains completed, non-refusal model output to your supplied JSON Schema.",
   "removal_reason": "",
   "source": "OpenAI API docs, Structured model outputs",
   "source_url": "https://developers.openai.com/api/docs/guides/structured-outputs",
   "locator": "introduction; a statement about the documentation, not executable (service-side API, no key used)",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C15",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C15",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Anthropic's docs say structured outputs guarantee schema-compliant responses through constrained decoding.",
   "quote": "guarantee schema-compliant responses through constrained decoding",
   "removal_reason": "",
   "source": "Anthropic (Claude) docs, Structured outputs",
   "source_url": "https://platform.claude.com/docs/en/build-with-claude/structured-outputs",
   "locator": "introduction; not executable (service-side API, no key used)",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C16",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C16",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Both list exceptions: a refusal, or a reply cut off at the token limit, can break the shape.",
   "quote": "OpenAI: 'in the case of a refusal … or if for example you reach a max tokens limit and the response is incomplete'. Anthropic: 'guarantee schema compliance in most cases … Refusals … Token limit reached'",
   "removal_reason": "",
   "source": "OpenAI and Anthropic structured-output docs",
   "source_url": "https://developers.openai.com/api/docs/guides/structured-outputs",
   "locator": "OpenAI: handling edge cases; Anthropic: Invalid outputs section; not executable (service-side API, no key used)",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C17",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C17",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Google's Gemini docs say to always validate values, and to handle schema-compliant but semantically incorrect outputs.",
   "quote": "While output is syntactically correct JSON, always validate values in your application. … schema-compliant but semantically incorrect outputs",
   "removal_reason": "",
   "source": "Google Gemini docs, Structured output",
   "source_url": "https://aistudio.google.com/docs/structured-output",
   "locator": "Best practices",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C18",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C18",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Syntax: OpenAI takes response_format with type json_schema and strict true; Anthropic takes the schema in output_config.format or a Pydantic model via client.messages.parse(); Gemini takes response_format with type text, mime_type application/json and a schema.",
   "quote": "OpenAI: response_format: { type: \"json_schema\", json_schema: {\"strict\": true, \"schema\": ...} }. Anthropic: JSON outputs ( output_config.format ) … client.messages.parse( . Gemini: response_format={ \"type\": \"text\", \"mime_type\": \"application/json\", \"schema\": ...}",
   "removal_reason": "",
   "source": "the three vendors' structured-output docs",
   "source_url": "https://developers.openai.com/api/docs/guides/structured-outputs",
   "locator": "code samples on each page; not executable here (service-side API, no key used)",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C19",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C19",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Pydantic's date type rejects November 2, 2026 and accepts 1124 as a year.",
   "quote": "C19 OK pydantic date rejects 'November 2, 2026' and accepts '1124-06-12'",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package) (pydantic 2.14.0)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C20",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C20",
   "status": "STRUCK",
   "status_label": "Struck",
   "removed": true,
   "claim": "C20 The schema descriptions printed in the guide are the exact ones the runs sent.",
   "quote": "",
   "removal_reason": "(narrowed; see the replacement row)",
   "source": "",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C21",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C21",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Document 2026-19757's lines: docket FAA-2026-11786, RIN 2120-AA64, effective October 13, 2026, comments by November 12, 2026.",
   "quote": "C21 OK 19757 text carries docket FAA-2026-11786, RIN 2120-AA64, effective October 13, 2026, comments by November 12, 2026",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C22",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C22",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "gemini-2.5-flash-lite's output for 2026-19757 is as printed; five fields match the key exactly and agency does not.",
   "quote": "C22 OK Gemini 19757 output matches the printed JSON; exact vs key on 5 fields (agency not exact)",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C23",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C23",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Simon Willison's workshop handout: Crucially, did it get everything right? … Spot-checking is crucial.",
   "quote": "Crucially, did it get everything right? We used a very inexpensive model for this. Spot-checking is crucial.",
   "removal_reason": "",
   "source": "Simon Willison, Building software on top of LLMs (PyCon 2025 handout), Structured data extraction",
   "source_url": "https://building-with-llms-pycon-2025.readthedocs.io/en/latest/structured-data-extraction.html",
   "locator": "after the llm logs example",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C24",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C24",
   "status": "STRUCK",
   "status_label": "Struck",
   "removed": true,
   "claim": "C24 The guide's outcome categories reproduce the MUC-5 evaluation categories (Chinchor and Sundheim, 1993), down to the word spurious.",
   "quote": "",
   "removal_reason": "(narrowed; see the replacement row)",
   "source": "",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C25",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C25",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "The Strands Evals structured-output evaluator requires zero false alarms when there was nothing to find.",
   "quote": "When there was nothing to find, it requires fa to be zero, which catches an invention.",
   "removal_reason": "",
   "source": "Strands Agents docs, Structured output evaluator",
   "source_url": "https://strandsagents.com/docs/user-guide/evals-sdk/evaluators/structured_output_evaluator/",
   "locator": "evaluator description; not executable here (no package run)",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C26",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C26",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The results table: all 27 scored cells (rows right and values filled where the key is empty) for the five tools; verdicts FAIL, FAIL, FAIL, PASS, PASS.",
   "quote": "C26 OK results table: 27/27 cells reproduce",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C27",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C27",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "A bar of 18 with the zero rule kept changes no verdict; 18 without the zero rule passes two of the three LLM runs.",
   "quote": "min_right=18 max_spur=0: all LLM runs FAIL; min_right=18 max_spur=99: gemini PASS, gpt-5-nano FAIL, no-date-format PASS",
   "removal_reason": "",
   "source": "our own independent re-score (first-party)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C28",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C28",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Every value filled where the key is empty sits on the two correction notices; the IRS correction alone fails each run under the zero rule.",
   "quote": "C28 OK spurious values only on 2026-19786, C1-2026-18574; C28b OK the IRS correction alone puts a spurious value in agency/action in each LLM run",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C29",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C29",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The regex scored 20 on every field, but was written while looking at these documents.",
   "quote": "regex_baseline: 20/20 on all six fields, overall PASS; code/regex_baseline.py written on the 20 documents (SOURCES.md)",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C30",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C30",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "With format: date, gpt-5-nano wrote 1124-06-12 for a 12 November deadline and 1102-11-02 for a 2 November effective date; 8 rows had a year other than 2026, all valid to the schema and Pydantic; without format: date every date was in 2026.",
   "quote": "C30 OK format: date run 8 of 22 dates not 2026 (incl. 1124-06-12, 1102-11-02); no-format run 23 dates, 0 not 2026",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C31",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C31",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "On the IRS correction: Gemini filled DEPARTMENT OF THE TREASURY and the title plus Correction; gpt-5-nano DEPARTMENT OF THE TREASURY and Correction; the re-run Internal Revenue Service and Correction; all three took its corrected date; MARAD's date became a comment deadline (Gemini), an effective date (gpt-5-nano) or both (re-run).",
   "quote": "C31 Gemini IRS agency='DEPARTMENT OF THE TREASURY' due=2026-10-26, MARAD due=2026-09-28; nano IRS 'Correction' eff=2026-10-26, MARAD eff=2026-09-28; re-run IRS 'Internal Revenue Service' eff=2026-10-26, MARAD eff and due 2026-09-28",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C32",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C32",
   "status": "REPORTED",
   "status_label": "Reported",
   "removed": false,
   "claim": "Our first answer key broke the null rule on the IRS correction; we fixed the key and re-scored everything.",
   "quote": "All score files were regenerated after the C1-2026-18574 answer-key fix",
   "removal_reason": "",
   "source": "the Desk's own run notes (SOURCES.md line 62) and draft v3 line 222; the earlier key file is not in git",
   "source_url": "",
   "locator": "Products/extraction-guide-v5-2026-11-13/SOURCES.md L62",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C33",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C33",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Cleanlab found every public structured-output benchmark it examined full of wrong answers in the key.",
   "quote": "every public benchmark we examined was full of erroneous and inconsistent ground-truth outputs",
   "removal_reason": "",
   "source": "Cleanlab blog (Goh, Mueller), structured output benchmark",
   "source_url": "https://cleanlab.ai/blog/structured-output-benchmark/",
   "locator": "findings paragraph",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C34",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C34",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "On agency, Gemini matched 15 rows exactly and 4 more after clean-up.",
   "quote": "C33 OK Gemini agency exact 15, normalised 4",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C35",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C35",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The Federal Register API missed two dates: the HUD applicability date, and the comment deadline for IRS proposal 2026-19888 (30 November 2026 in the text).",
   "quote": "C34 OK FR API: no effective_on for HUD 2026-19743, no comments_close_on for 2026-19888 (text says November 30, 2026)",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C36",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C36",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Keyed the other way on MARAD's bare date, the Federal Register's effective_date drops to 18 of 20 and fails, while Gemini and the regex still pass it.",
   "quote": "MARAD bare date keyed as effective_date: federal_register_api_metadata 18/20 FAIL; gemini 19/20 PASS; regex 19/20 PASS",
   "removal_reason": "",
   "source": "our own independent re-score (first-party)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C37",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C37",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Routing dates to the Federal Register API sends 11 rows in each date field to a person, because 10 are blank by design.",
   "quote": "C37 OK FR API blank on 11 effective_date rows and 11 comments_due rows; C41 OK answer key null on 10/20 each",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C38",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C38",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "score.py exits non-zero when a field misses the bar, so it can block a deploy.",
   "quote": "C38 OK score.py exits 1 on a FAIL and 0 on a PASS",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C39",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C39",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Anthropic's structured outputs don't support minimum or minLength, and cap a request at 16 parameters that use union types.",
   "quote": "Numerical constraints (such as minimum , maximum , multipleOf ) String constraints ( minLength , maxLength ) … Parameters with union types 16 Total parameters that use anyOf or type arrays",
   "removal_reason": "",
   "source": "Anthropic (Claude) docs, Structured outputs",
   "source_url": "https://platform.claude.com/docs/en/build-with-claude/structured-outputs",
   "locator": "JSON Schema limitations; service-side limit, not executable here",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C40",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C40",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Gemini supports a subset of the JSON Schema specification.",
   "quote": "Gemini's structured output mode supports a subset of the JSON Schema specification.",
   "removal_reason": "",
   "source": "Google Gemini docs, Structured output",
   "source_url": "https://aistudio.google.com/docs/structured-output",
   "locator": "JSON Schema support; service-side, not executable here",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C41",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C41",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Both date fields are null on half the rows here (10 of 20 each).",
   "quote": "C41 OK answer key: effective_date null on 10/20, comments_due null on 10/20",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C42",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C42",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "OpenAI retires gpt-5-nano from its API on 11 December 2026 and names gpt-6-luna as the replacement.",
   "quote": "Dec 11, 2026 gpt-5-nano , gpt-5-nano-2025-08-07 Deprecated gpt-6-luna Jun 11, 2026",
   "removal_reason": "",
   "source": "OpenAI API docs, Deprecations",
   "source_url": "https://developers.openai.com/api/docs/deprecations",
   "locator": "Upcoming deprecations table; service-side schedule, not executable",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C43",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C43",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "A 2026 benchmark of 21 models found near-perfect schema compliance, and the best exact match on values from text was 83.0%.",
   "quote": "We evaluate 21 frontier and open-weight models … models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text",
   "removal_reason": "",
   "source": "Singh, Khurdula, Khemlani, Agarwal, The Structured Output Benchmark (arXiv 2604.25359, 28 Apr 2026)",
   "source_url": "https://arxiv.org/abs/2604.25359",
   "locator": "abstract",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C44",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C44",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "filing-extraction-benchmark scores rules, an LLM and a hybrid that asks the model only where the rules abstain, all with one scorer.",
   "quote": "takes the same inputs, produces the same output schema and is scored by the same scorer … hybrid, model asked only where rules abstained",
   "removal_reason": "",
   "source": "filing-extraction-benchmark (GitHub README)",
   "source_url": "https://github.com/JAYANSHUBADLANI/filing-extraction-benchmark",
   "locator": "README, method and results table; not executable here (repo not run)",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C45",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C45",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Scoring needs no API key; re-scoring our saved runs exits 1 because our LLM runs fail the bar; with no key the extraction scripts stop at the key message and write nothing.",
   "quote": "clean-unzip reader test: score exit 1 (8 files); extract and naive_extract print 'Set OPENAI_API_KEY (or AI_GATEWAY_API_KEY) first.' exit 1; data/outputs unchanged (19 files)",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C46",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C46",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The Metric Validity Audit's tagline is: Your metric is green. Should you believe it?",
   "quote": "Your metric is green. Should you believe it?",
   "removal_reason": "",
   "source": "The Durability Curve, Metric Validity Audit (first-party)",
   "source_url": "https://durabilitycurve.com/tools/metric-validity-audit/",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C47",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C47",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "DeepLearning.AI's Function-calling and data extraction with LLMs is taught by Jiantao Jiao and Venkat Srinivasan of Nexusflow, with a Structured Extraction lesson on an open model.",
   "quote": "Instructors: Jiantao Jiao, Venkat Srinivasan Nexusflow … Structured Extraction … NexusRavenV2-13B, an open source model",
   "removal_reason": "",
   "source": "DeepLearning.AI course page",
   "source_url": "https://www.deeplearning.ai/courses/function-calling-and-data-extraction-with-llms",
   "locator": "instructors and syllabus",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C48",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C48",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Willison's handout does schema extraction from a real FEMA briefing PDF.",
   "quote": "FEMA's Daily Operation Briefing PDF is a fascinating document.",
   "removal_reason": "",
   "source": "Simon Willison, PyCon 2025 handout",
   "source_url": "https://building-with-llms-pycon-2025.readthedocs.io/en/latest/structured-data-extraction.html",
   "locator": "Something a bit more impressive",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C49",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C49",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "score.py scores any tool's output written as one JSON line per document with document_number and a parsed object (or null for a failed call).",
   "quote": "rowfmt.jsonl (the regex output rewritten as document_number + parsed only): 20 rows, 0 failed or absent, overall PASS, exit 0",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C20b",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C20b",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "C20b The schema descriptions printed in the guide are the exact ones the first two runs sent; the third run appends 'Format YYYY-MM-DD.' to both date descriptions and drops format: date.",
   "quote": "6 of 6 descriptions identical to code/schema.py; extract.py plain_dates() pops 'format' and adds ' Format YYYY-MM-DD.' to effective_date and comments_due",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C24b",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C24b",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "C24b Most of the guide's outcomes line up with the MUC-5 evaluation categories (Chinchor and Sundheim, 1993), down to the word spurious; the guide splits exact from normalised and adds failed.",
   "quote": "If the response has a fill which has no corresponding fill in the key, the category is spurious (SPU).",
   "removal_reason": "",
   "source": "Nancy Chinchor, Beth Sundheim, MUC-5 Evaluation Metrics (ACL Anthology M93-1007)",
   "source_url": "https://aclanthology.org/M93-1007/",
   "locator": "PDF, scoring categories COR, PAR, INC, MIS, SPU, NON",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C50",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C50",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The Federal Register's agency, action and docket fields follow different conventions from the key: agencies list the parent department first, actions keep the full stop, docket_ids carry 'Docket No.' and extra identifiers.",
   "quote": "2026-19757 FR agencies ['Transportation Department', 'Federal Aviation Administration']; action 'Final rule; request for comments.'; docket_ids ['Docket No. FAA-2026-11786', 'Project Identifier …'] vs key 'Federal Aviation Administration' / 'Final rule; request for comments' / 'FAA-2026-11786'",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C51",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C51",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The key's checker_note column carries 8 judgement calls, among them the two corrections, the HUD date and which EPA deadline counts.",
   "quote": "checker_note rows: 8 (incl. C1-2026-18574, 2026-19786, 2026-19743, 2026-19774)",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C52",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C52",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Without the two correction notices (18 documents), Gemini and the run without format: date get every field right; gpt-5-nano still fails on its garbled dates; the publisher's data misses one date in each date field.",
   "quote": "gemini 18/18 all fields; nano effective_date 17/18, comments_due 11/18; no-date-format 18/18 all fields; FR effective_date 17/18, comments_due 17/18; regex 18/18",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C53",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C53",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "A plausibility check (reject any date more than a year either side of the document's publication date) rejects all 8 non-2026 dates in our runs and no correct date.",
   "quote": "bad years 8, caught by ±1y window 8; correct dates rejected 0; key dates outside window 0",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C54",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C54",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The zero rule bites only where the key is empty: in this set agency and action are empty in 1 row each, docket_id in none, RIN in 2, and each date field in 10.",
   "quote": "agency empty 1; action 1; docket_id 0; rin 2; effective_date 10; comments_due 10",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C55",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C55",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "score.py scores a row missing a field's key as failed, and skips a field only when every row declares it not_scored; removing three keys from the Gemini run fails those fields and exits 1.",
   "quote": "probe_dropall: agency/action/comments_due 0/20 failed 20, overall FAIL, exit 1; probe_dropodd: 18/20 failed 2, exit 1; saved verdicts unchanged",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C56",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C56",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "score.py's normalisation drops the parent department after the first comma, which is how '(FAA), DOT' counts as right for agency.",
   "quote": "norm_agency: norm_text(re.sub(r'\\([^)]*\\)', '', s).split(',')[0]); Gemini agency 15 exact + 4 normalised",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C57",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C57",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "On the IRS correction, 26 October 2026 is a date it corrects in another document's DATES section.",
   "quote": "In proposed rule document 2026-18574 … in the second line of the DATES section, `October 16, 2026'' should read `October 26, 2026''.",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package); text of FR C1-2026-18574",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C58",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C58",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "We used an OpenAI-compatible gateway, so switching model meant changing one string.",
   "quote": "gateway.py posts to <base>/chat/completions with the body's model string; extract.py takes the model name as its only argument",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C59",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C59",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Gemini's client.interactions.create() takes response_format with type text, mime_type application/json and a schema.",
   "quote": "interaction = client.interactions.create( model=\"gemini-4-hydrogen\", input=prompt, response_format={ \"type\": \"text\", \"mime_type\": \"application/json\", \"schema\": Recipe.model_json_schema() }",
   "removal_reason": "",
   "source": "Google Gemini docs, Structured output",
   "source_url": "https://aistudio.google.com/docs/structured-output",
   "locator": "Python sample; not executable here (service-side API, no key used)",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C60",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C60",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Running the scripts overwrites the matching files in data/outputs/.",
   "quote": "extract.py and metadata_as_output.py open their output with mode \"w\"",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C61",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C61",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The bar scales with the row count, rounded up, so under 20 rows it means every row.",
   "quote": "min_right = -(-MIN_RIGHT_OF_20 * len(truth) // 20): 19 of 20, 18 of 18, 10 of 10",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C62",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C62",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "extract.py cuts each document at 8,000 characters, and a field past the cut usually scores missing.",
   "quote": "MAX_CHARS = 8000; text[:MAX_CHARS] sent to the model; no keyed value sits past 8,000 characters in this set",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C63",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C63",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "plausible_dates.py drops all eight wrong-year dates from the format: date run and no date from any other run; the checked run then scores those fields failed.",
   "quote": "8 date(s) outside ±365 days (gpt-5-nano); 0 for gemini, no-date-format, regex; checked run effective_date 1 failed, comments_due 7 failed",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C64",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C64",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Some rules take effect years after they are published (e.g. FR 2026-11800, published 11 June 2026, effective 30 September 2028).",
   "quote": "\"document_number\": \"2026-11800\", … \"publication_date\": \"2026-06-11\", \"effective_on\": \"2028-09-30\"",
   "removal_reason": "",
   "source": "Federal Register API, document 2026-11800",
   "source_url": "https://www.federalregister.gov/api/v1/documents/2026-11800.json",
   "locator": "fields publication_date, effective_on",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C65",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C65",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The answer-key fix on the IRS correction moved no overall verdict: each LLM run also fails on its date fields, whatever the agency key says.",
   "quote": "gemini comments_due 18/20 (2 spurious); gpt-5-nano effective_date 17/20 (2), comments_due 13/20; no-format effective_date 18/20 (2): FAIL on dates alone",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-13",
   "in_essay": ""
  },
  {
   "id": "C66",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C66",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "score.py says how many fields it scored and exits non-zero when a field was skipped, unless --allow-skip is passed.",
   "quote": "federal_register_api_metadata: 'overall: PASS: 3 of 3 fields pass; 3 of 6 fields scored (skipped: agency, action, docket_id)', exit 1; with --allow-skip, exit 0",
   "removal_reason": "",
   "source": "own run (first-party, extraction-guide-v5 package)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-15",
   "in_essay": ""
  },
  {
   "id": "C67",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C67",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Gemini (one of three LLM runs) returned valid JSON for all 20 notices and got 116 of the 120 values right, and still failed: the bar allows no value in a field that should be empty, and it filled four, all on two correction notices, each real text from the document; on the IRS correction (which changes a date in document 2026-18574 from 16 to 26 October) it returned 26 October as comments_due.",
   "quote": "gemini: agency 19, action 19, docket_id 20, rin 20, effective_date 20, comments_due 18 = 116/120; spurious 1+1+0+0+0+2 = 4 (C1-2026-18574 agency, action, comments_due='2026-10-26'; 2026-19786 comments_due); C1-2026-18574 text: 'In proposed rule document 2026-18574 … `October 16, 2026'' should read `October 26, 2026'''",
   "removal_reason": "",
   "source": "own run + FR document C1-2026-18574",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-15",
   "in_essay": ""
  },
  {
   "id": "C68",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C68",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Two more models, GPT-5.6 and Claude Opus 5.5, keyed all 120 cells again in fresh sessions without seeing our key, and both matched it on every cell.",
   "quote": "score.py on each blind file: every field 20/20, exact + null ok, 0 normalised, 0 missing/wrong/spurious/failed ('overall: PASS: 6 of 6 fields pass')",
   "removal_reason": "",
   "source": "own run (first-party; blind re-keys)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-15",
   "in_essay": ""
  },
  {
   "id": "C69",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C69",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "The Federal Register's own data (API metadata, no model involved) agrees with our key on 113 of 120 cells; the seven differences are four naming conventions, two dates the publisher left blank, and one rule call (the IRS correction's agency).",
   "quote": "comparison table in data/key_checks/README.md: agency 16, action 20, docket_id 19, rin 20, effective_date 19, comments_due 19 = 113 of 120 cells; differences: 2026-19743/19860/19899 agency naming, 2026-19786 'Docket Number' prefix, 2026-19743 effective_date blank, 2026-19888 comments_due blank, C1-2026-18574 agency",
   "removal_reason": "",
   "source": "own run (first-party comparison script): Federal Register API metadata (data/fr_metadata.json, fetched 13 Nov 2026) vs data/answer_key.csv",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-15",
   "in_essay": ""
  },
  {
   "id": "C70",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C70",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "Counted across all six fields, the run without format: date got 115 of 120 right and gpt-5-nano 108; each run filled four or five values where the key is empty, every one on the two correction notices.",
   "quote": "no-date-format: 19+19+20+20+18+19 = 115, spurious 1+1+0+0+2+1 = 5; gpt-5-nano: 19+19+20+20+17+13 = 108, spurious 1+1+0+0+2+0 = 4; all spurious rows on C1-2026-18574 or 2026-19786",
   "removal_reason": "",
   "source": "own run (first-party)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-15",
   "in_essay": ""
  },
  {
   "id": "C71",
   "url": "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C71",
   "status": "EXECUTED",
   "status_label": "Executed",
   "removed": false,
   "claim": "On the three fields the publisher's data is scored on (rin, effective_date, comments_due), it and Gemini each got 58 of 60 right; the publisher's two misses were blanks, Gemini's two were values filled where the key is empty, and only the publisher passed.",
   "quote": "gemini: rin 20/20 PASS, effective_date 20/20 PASS, comments_due 18/20 (2 spurious) FAIL = 58/60; federal_register_api_metadata: rin 20/20 PASS, effective_date 19/20 (1 missing) PASS, comments_due 19/20 (1 missing) PASS = 58/60",
   "removal_reason": "",
   "source": "own run (first-party)",
   "source_url": "",
   "locator": "",
   "date_read": "2026-11-15",
   "in_essay": ""
  }
 ]
}