---
title: "Claim Ledger: The JSON parsed. Is it right?"
description: "Structured outputs all but guarantee the shape of an extraction. Whether the answer is right is yours to score. How to score an LLM, a regex and the publisher's data, then keep what passes per field."
author: "Harry Floyd"
publication: "The Durability Curve"
canonical: "https://durabilitycurve.com/claims/the-json-parsed-is-it-right/"
essay: "https://durabilitycurve.com/blog/the-json-parsed-is-it-right/"
published: "2026-11-15"
ledger_date: "2026-11-15"
claims: 71
removed_before_publication: 2
---

# Claim Ledger: The JSON parsed. Is it right?

*Structured outputs all but guarantee the shape of an extraction. Whether the answer is right is yours to score. How to score an LLM, a regex and the publisher's data, then keep what passes per field.*

Essay: https://durabilitycurve.com/blog/the-json-parsed-is-it-right/  
Ledger (canonical, cite this): https://durabilitycurve.com/claims/the-json-parsed-is-it-right/  
Published 2026-11-15 · last verified 2026-11-15

## What the essay claims

Before: "Structured outputs fixed extraction. If the JSON validates against my schema, the data is good. I spot-check a few rows and ship."

After: "The schema guarantees shape. Validation checks the value is usable. Nothing guarantees it's true, so I score it: 20 checked rows, a bar set before I look, spurious values counted on their own. On our run, three schema-valid runs all failed, the publisher's data passed, and the odd documents caused almost every failure. I keep the 20 rows and the script; the model is the part I expect to change."

## The claim ladder

| # | Claim | Evidence | Whose behaviour it measures | What it does not reach (scope) |
|---|---|---|---|---|
| 1 | Asked for JSON in the prompt, gemini-2.5-flash-lite fenced 20 of 20 replies, so json.loads failed 20 times; after stripping fences 13 of 20 failed a typed schema, all on dates. | saved naive run + log | one model, one prompt, these 20 documents, 8 Oct 2026 | other prompts, other models, today's model versions |
| 2 | gpt-5-nano with the same prompt: 20 parsed, 9 failed the schema, all on dates; a MARAD row passed with a bare date as its effective date. | saved naive run | same | same |
| 3 | With the schema enforced, all three LLM runs were schema-valid on 20 of 20 and none passed the bar. | verify_object.py (Object Record) | three runs of two small models through a gateway | larger models; any claim that LLMs "can't" extract |
| 4 | Every value filled where the key is empty sits on the two correction notices; the IRS correction alone fails each run. | score.py problems lists | same | documents without corrections |
| 5 | With format: date, gpt-5-nano wrote 8 dates with years other than 2026, all valid to the schema and Pydantic; without it, all 23 dates were 2026. | saved runs, check_claims | one run each | a general law about format: date (the piece says "one run of each") |
| 6 | The Federal Register's own data passed the three fields it could be scored on; keyed the other way on MARAD, its effective_date drops to 18 of 20. | score.py; blind check alt_key | the publisher's metadata on these 20 | other publishers; it missed two dates and still passed |
| 7 | A bar of 18 with the zero rule kept changes no verdict; dropping the zero rule passes two of three runs. | bar_variants (blind check) | these runs | — |
| 8 | Vendor statements: OpenAI "constrains completed, non-refusal model output to your supplied JSON Schema"; Anthropic "guarantee schema-compliant responses through constrained decoding"; Gemini "always validate values in your application". Both OpenAI and Anthropic list refusal and token-limit exceptions. | vendor pages, read 13 Nov 2026 | vendor documentation | what the models do in practice |
| 9 | OpenAI retires gpt-5-nano from its API on 11 Dec 2026 and names gpt-6-luna as the replacement. | OpenAI deprecations page, read 13 Nov 2026 | OpenAI's published schedule | — |

## What would make it wrong

If a reader runs the shipped scorer on our saved outputs and gets different per-field counts from the guide's results grid, then the grid is wrong and the piece is pulled until it is fixed.

If a person checking the 120 answer-key cells finds a disagreement that changes a verdict, then the results section changes and the piece says so.

If current models (e.g. gpt-6-luna) pass the bar on the same 20 documents, then the guide's results are dated, and the update says so; the method claim (score it) does not depend on any model failing.

## The evidence, row by row

Labels: VERIFIED = checked against the primary source itself by a checker who did not draft the piece, usually with the exact words, where they sit and the date read · EXECUTED = a run or observation done for the piece (code, a query, a count, or a check made live in an app); the claim is what it returned; run records are not published · CHECKED = checked against its source by a separate checker in the older ledger format, which recorded no quote, locator or date · REPORTED = not verified word for word against a primary source (secondary, not openable in full, or supported only in part); the essay words it accordingly · STRUCK = drafted, checked and removed before publication · EXCLUDED = considered and deliberately left out.

### C01. Our grader piece sets out a five-step check on twenty items with a scoring sheet, counts false passes and false fails separately, and says to check the key before you blame the grader.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C01
- Quote: Here is a five-step check on twenty items for yours. … five steps, a scoring sheet … Count false passes and false fails separately. … check the key before you blame the grader
- Source: The Durability Curve, Your AI Grader Is Only as Good as Its Answer Key (first-party)
- URL: https://durabilitycurve.com/blog/your-ai-grader-answer-key/

### C02. On 20 Federal Register notices with a strict schema, all three LLM runs returned schema-valid JSON for all 20 documents, and none passed the bar.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C02
- Quote: HEADLINE: 3 of 3 LLM runs schema-valid on 20 of 20, 0 of 3 passed
- Source: own run (first-party, extraction-guide-v5 package)

### C03. The Federal Register's own structured data passed on the three fields we could score it on (on our key).

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C03
- Quote: federal_register_api_metadata overall: PASS
- Source: own run (first-party, extraction-guide-v5 package)

### C04. One run per setup on 8 October 2026; an AI agent built the answer key from the Federal Register text; a second agent checked all 120 cells on 10 November and found no disagreement; no person has checked them.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C04
- Quote: I read the preamble … of all 20 documents and compared all 120 cells … 0 disagreements
- Source: our own independent check (first-party)

### C05. The documents are ten Rules and ten Proposed Rules from 28 and 29 September 2026, via the public Federal Register API.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C05
- Quote: C05 OK types Rule 10, Proposed Rule 10; dates 28 Sep (15), 29 Sep (5)
- Source: own run (first-party, extraction-guide-v5 package)

### C06. Ten are FAA airworthiness directives on one template.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C06
- Quote: C06 OK 10 titles start 'Airworthiness Directives'
- Source: own run (first-party, extraction-guide-v5 package)

### C07. Two are correction notices, and one of those has no AGENCY: or ACTION: line.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C07
- Quote: C07 OK IRS C1-2026-18574 has no AGENCY:/ACTION: line
- Source: own run (first-party, extraction-guide-v5 package)

### C08. A HUD fee announcement calls its date an applicability date.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C08
- Quote: DATES: Applicability date: October 28, 2026.
- Source: own run (first-party, extraction-guide-v5 package); text of FR 2026-19743

### C09. An EPA proposal lists three deadlines in its DATES section.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C09
- Quote: C09 OK EPA 2026-19774 DATES names 12 Nov (comments), 28 Oct (OMB/PRA), 5 Oct (hearing request)
- Source: own run (first-party, extraction-guide-v5 package)

### C10. Prompted for JSON, gemini-2.5-flash-lite wrapped all twenty replies in a code fence, so json.loads failed twenty times out of twenty.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C10
- Quote: C10 OK naive Gemini: fenced 20/20, json.loads failed 20/20
- Source: own run (first-party, extraction-guide-v5 package)

### C11. With the fences stripped, thirteen of twenty still fail a typed schema, every one on a date; the model wrote October 28, 2026 in one row and 2026-10-28 in another.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C11
- Quote: C11 OK after stripping fences 13/20 fail the schema, all on dates; 19745='October 28, 2026', 19741='2026-10-28'
- Source: own run (first-party, extraction-guide-v5 package)

### C12. gpt-5-nano with the same prompt: all twenty parsed, nine failed the schema, all on dates; on the IRS correction it put October 26, 2026 in both date fields.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C12
- Quote: C12 OK naive nano: 20/20 parsed, 9 failed the schema (dates only); IRS both dates 'October 26, 2026'
- Source: own run (first-party, extraction-guide-v5 package)

### C13. A MARAD correction's DATES: line is a bare September 28, 2026; gpt-5-nano returned it as the effective date, valid and wrong by our key, which leaves it empty.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C13
- Quote: C13 OK MARAD: naive nano passed with effective_date 2026-09-28; key empty; DATES line bare
- Source: own run (first-party, extraction-guide-v5 package)

### C14. OpenAI's docs say Structured Outputs constrains completed, non-refusal model output to your supplied JSON Schema.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C14
- Quote: Structured Outputs constrains completed, non-refusal model output to your supplied JSON Schema.
- Source: OpenAI API docs, Structured model outputs, introduction; a statement about the documentation, not executable (service-side API, no key used)
- URL: https://developers.openai.com/api/docs/guides/structured-outputs

### C15. Anthropic's docs say structured outputs guarantee schema-compliant responses through constrained decoding.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C15
- Quote: guarantee schema-compliant responses through constrained decoding
- Source: Anthropic (Claude) docs, Structured outputs, introduction; not executable (service-side API, no key used)
- URL: https://platform.claude.com/docs/en/build-with-claude/structured-outputs

### C16. Both list exceptions: a refusal, or a reply cut off at the token limit, can break the shape.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C16
- Quote: OpenAI: 'in the case of a refusal … or if for example you reach a max tokens limit and the response is incomplete'. Anthropic: 'guarantee schema compliance in most cases … Refusals … Token limit reached'
- Source: OpenAI and Anthropic structured-output docs, OpenAI: handling edge cases; Anthropic: Invalid outputs section; not executable (service-side API, no key used)
- URL: https://developers.openai.com/api/docs/guides/structured-outputs

### C17. Google's Gemini docs say to always validate values, and to handle schema-compliant but semantically incorrect outputs.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C17
- Quote: While output is syntactically correct JSON, always validate values in your application. … schema-compliant but semantically incorrect outputs
- Source: Google Gemini docs, Structured output, Best practices
- URL: https://aistudio.google.com/docs/structured-output

### C18. Syntax: OpenAI takes response_format with type json_schema and strict true; Anthropic takes the schema in output_config.format or a Pydantic model via client.messages.parse(); Gemini takes response_format with type text, mime_type application/json and a schema.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C18
- Quote: OpenAI: response_format: { type: "json_schema", json_schema: {"strict": true, "schema": ...} }. Anthropic: JSON outputs ( output_config.format ) … client.messages.parse( . Gemini: response_format={ "type": "text", "mime_type": "application/json", "schema": ...}
- Source: the three vendors' structured-output docs, code samples on each page; not executable here (service-side API, no key used)
- URL: https://developers.openai.com/api/docs/guides/structured-outputs

### C19. Pydantic's date type rejects November 2, 2026 and accepts 1124 as a year.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C19
- Quote: C19 OK pydantic date rejects 'November 2, 2026' and accepts '1124-06-12'
- Source: own run (first-party, extraction-guide-v5 package) (pydantic 2.14.0)

### C21. Document 2026-19757's lines: docket FAA-2026-11786, RIN 2120-AA64, effective October 13, 2026, comments by November 12, 2026.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C21
- Quote: C21 OK 19757 text carries docket FAA-2026-11786, RIN 2120-AA64, effective October 13, 2026, comments by November 12, 2026
- Source: own run (first-party, extraction-guide-v5 package)

### C22. gemini-2.5-flash-lite's output for 2026-19757 is as printed; five fields match the key exactly and agency does not.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C22
- Quote: C22 OK Gemini 19757 output matches the printed JSON; exact vs key on 5 fields (agency not exact)
- Source: own run (first-party, extraction-guide-v5 package)

### C23. Simon Willison's workshop handout: Crucially, did it get everything right? … Spot-checking is crucial.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C23
- Quote: Crucially, did it get everything right? We used a very inexpensive model for this. Spot-checking is crucial.
- Source: Simon Willison, Building software on top of LLMs (PyCon 2025 handout), Structured data extraction, after the llm logs example
- URL: https://building-with-llms-pycon-2025.readthedocs.io/en/latest/structured-data-extraction.html

### C25. The Strands Evals structured-output evaluator requires zero false alarms when there was nothing to find.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C25
- Quote: When there was nothing to find, it requires fa to be zero, which catches an invention.
- Source: Strands Agents docs, Structured output evaluator, evaluator description; not executable here (no package run)
- URL: https://strandsagents.com/docs/user-guide/evals-sdk/evaluators/structured_output_evaluator/

### C26. The results table: all 27 scored cells (rows right and values filled where the key is empty) for the five tools; verdicts FAIL, FAIL, FAIL, PASS, PASS.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C26
- Quote: C26 OK results table: 27/27 cells reproduce
- Source: own run (first-party, extraction-guide-v5 package)

### C27. A bar of 18 with the zero rule kept changes no verdict; 18 without the zero rule passes two of the three LLM runs.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C27
- Quote: min_right=18 max_spur=0: all LLM runs FAIL; min_right=18 max_spur=99: gemini PASS, gpt-5-nano FAIL, no-date-format PASS
- Source: our own independent re-score (first-party)

### C28. Every value filled where the key is empty sits on the two correction notices; the IRS correction alone fails each run under the zero rule.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C28
- Quote: C28 OK spurious values only on 2026-19786, C1-2026-18574; C28b OK the IRS correction alone puts a spurious value in agency/action in each LLM run
- Source: own run (first-party, extraction-guide-v5 package)

### C29. The regex scored 20 on every field, but was written while looking at these documents.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C29
- Quote: regex_baseline: 20/20 on all six fields, overall PASS; code/regex_baseline.py written on the 20 documents (SOURCES.md)
- Source: own run (first-party, extraction-guide-v5 package)

### C30. With format: date, gpt-5-nano wrote 1124-06-12 for a 12 November deadline and 1102-11-02 for a 2 November effective date; 8 rows had a year other than 2026, all valid to the schema and Pydantic; without format: date every date was in 2026.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C30
- Quote: C30 OK format: date run 8 of 22 dates not 2026 (incl. 1124-06-12, 1102-11-02); no-format run 23 dates, 0 not 2026
- Source: own run (first-party, extraction-guide-v5 package)

### C31. On the IRS correction: Gemini filled DEPARTMENT OF THE TREASURY and the title plus Correction; gpt-5-nano DEPARTMENT OF THE TREASURY and Correction; the re-run Internal Revenue Service and Correction; all three took its corrected date; MARAD's date became a comment deadline (Gemini), an effective date (gpt-5-nano) or both (re-run).

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C31
- Quote: C31 Gemini IRS agency='DEPARTMENT OF THE TREASURY' due=2026-10-26, MARAD due=2026-09-28; nano IRS 'Correction' eff=2026-10-26, MARAD eff=2026-09-28; re-run IRS 'Internal Revenue Service' eff=2026-10-26, MARAD eff and due 2026-09-28
- Source: own run (first-party, extraction-guide-v5 package)

### C32. Our first answer key broke the null rule on the IRS correction; we fixed the key and re-scored everything.

- Status: REPORTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C32
- Quote: All score files were regenerated after the C1-2026-18574 answer-key fix
- Source: the Desk's own run notes (SOURCES.md line 62) and draft v3 line 222; the earlier key file is not in git, Products/extraction-guide-v5-2026-11-13/SOURCES.md L62

### C33. Cleanlab found every public structured-output benchmark it examined full of wrong answers in the key.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C33
- Quote: every public benchmark we examined was full of erroneous and inconsistent ground-truth outputs
- Source: Cleanlab blog (Goh, Mueller), structured output benchmark, findings paragraph
- URL: https://cleanlab.ai/blog/structured-output-benchmark/

### C34. On agency, Gemini matched 15 rows exactly and 4 more after clean-up.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C34
- Quote: C33 OK Gemini agency exact 15, normalised 4
- Source: own run (first-party, extraction-guide-v5 package)

### C35. The Federal Register API missed two dates: the HUD applicability date, and the comment deadline for IRS proposal 2026-19888 (30 November 2026 in the text).

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C35
- Quote: C34 OK FR API: no effective_on for HUD 2026-19743, no comments_close_on for 2026-19888 (text says November 30, 2026)
- Source: own run (first-party, extraction-guide-v5 package)

### C36. Keyed the other way on MARAD's bare date, the Federal Register's effective_date drops to 18 of 20 and fails, while Gemini and the regex still pass it.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C36
- Quote: MARAD bare date keyed as effective_date: federal_register_api_metadata 18/20 FAIL; gemini 19/20 PASS; regex 19/20 PASS
- Source: our own independent re-score (first-party)

### C37. Routing dates to the Federal Register API sends 11 rows in each date field to a person, because 10 are blank by design.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C37
- Quote: C37 OK FR API blank on 11 effective_date rows and 11 comments_due rows; C41 OK answer key null on 10/20 each
- Source: own run (first-party, extraction-guide-v5 package)

### C38. score.py exits non-zero when a field misses the bar, so it can block a deploy.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C38
- Quote: C38 OK score.py exits 1 on a FAIL and 0 on a PASS
- Source: own run (first-party, extraction-guide-v5 package)

### C39. Anthropic's structured outputs don't support minimum or minLength, and cap a request at 16 parameters that use union types.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C39
- Quote: Numerical constraints (such as minimum , maximum , multipleOf ) String constraints ( minLength , maxLength ) … Parameters with union types 16 Total parameters that use anyOf or type arrays
- Source: Anthropic (Claude) docs, Structured outputs, JSON Schema limitations; service-side limit, not executable here
- URL: https://platform.claude.com/docs/en/build-with-claude/structured-outputs

### C40. Gemini supports a subset of the JSON Schema specification.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C40
- Quote: Gemini's structured output mode supports a subset of the JSON Schema specification.
- Source: Google Gemini docs, Structured output, JSON Schema support; service-side, not executable here
- URL: https://aistudio.google.com/docs/structured-output

### C41. Both date fields are null on half the rows here (10 of 20 each).

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C41
- Quote: C41 OK answer key: effective_date null on 10/20, comments_due null on 10/20
- Source: own run (first-party, extraction-guide-v5 package)

### C42. OpenAI retires gpt-5-nano from its API on 11 December 2026 and names gpt-6-luna as the replacement.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C42
- Quote: Dec 11, 2026 gpt-5-nano , gpt-5-nano-2025-08-07 Deprecated gpt-6-luna Jun 11, 2026
- Source: OpenAI API docs, Deprecations, Upcoming deprecations table; service-side schedule, not executable
- URL: https://developers.openai.com/api/docs/deprecations

### C43. A 2026 benchmark of 21 models found near-perfect schema compliance, and the best exact match on values from text was 83.0%.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C43
- Quote: We evaluate 21 frontier and open-weight models … models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text
- Source: Singh, Khurdula, Khemlani, Agarwal, The Structured Output Benchmark (arXiv 2604.25359, 28 Apr 2026), abstract
- URL: https://arxiv.org/abs/2604.25359

### C44. filing-extraction-benchmark scores rules, an LLM and a hybrid that asks the model only where the rules abstain, all with one scorer.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C44
- Quote: takes the same inputs, produces the same output schema and is scored by the same scorer … hybrid, model asked only where rules abstained
- Source: filing-extraction-benchmark (GitHub README), README, method and results table; not executable here (repo not run)
- URL: https://github.com/JAYANSHUBADLANI/filing-extraction-benchmark

### C45. Scoring needs no API key; re-scoring our saved runs exits 1 because our LLM runs fail the bar; with no key the extraction scripts stop at the key message and write nothing.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C45
- Quote: clean-unzip reader test: score exit 1 (8 files); extract and naive_extract print 'Set OPENAI_API_KEY (or AI_GATEWAY_API_KEY) first.' exit 1; data/outputs unchanged (19 files)
- Source: own run (first-party, extraction-guide-v5 package)

### C46. The Metric Validity Audit's tagline is: Your metric is green. Should you believe it?

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C46
- Quote: Your metric is green. Should you believe it?
- Source: The Durability Curve, Metric Validity Audit (first-party)
- URL: https://durabilitycurve.com/tools/metric-validity-audit/

### C47. DeepLearning.AI's Function-calling and data extraction with LLMs is taught by Jiantao Jiao and Venkat Srinivasan of Nexusflow, with a Structured Extraction lesson on an open model.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C47
- Quote: Instructors: Jiantao Jiao, Venkat Srinivasan Nexusflow … Structured Extraction … NexusRavenV2-13B, an open source model
- Source: DeepLearning.AI course page, instructors and syllabus
- URL: https://www.deeplearning.ai/courses/function-calling-and-data-extraction-with-llms

### C48. Willison's handout does schema extraction from a real FEMA briefing PDF.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C48
- Quote: FEMA's Daily Operation Briefing PDF is a fascinating document.
- Source: Simon Willison, PyCon 2025 handout, Something a bit more impressive
- URL: https://building-with-llms-pycon-2025.readthedocs.io/en/latest/structured-data-extraction.html

### C49. score.py scores any tool's output written as one JSON line per document with document_number and a parsed object (or null for a failed call).

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C49
- Quote: rowfmt.jsonl (the regex output rewritten as document_number + parsed only): 20 rows, 0 failed or absent, overall PASS, exit 0
- Source: own run (first-party, extraction-guide-v5 package)

### C20b. C20b The schema descriptions printed in the guide are the exact ones the first two runs sent; the third run appends 'Format YYYY-MM-DD.' to both date descriptions and drops format: date.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C20b
- Quote: 6 of 6 descriptions identical to code/schema.py; extract.py plain_dates() pops 'format' and adds ' Format YYYY-MM-DD.' to effective_date and comments_due
- Source: own run (first-party, extraction-guide-v5 package)

### C24b. C24b Most of the guide's outcomes line up with the MUC-5 evaluation categories (Chinchor and Sundheim, 1993), down to the word spurious; the guide splits exact from normalised and adds failed.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C24b
- Quote: If the response has a fill which has no corresponding fill in the key, the category is spurious (SPU).
- Source: Nancy Chinchor, Beth Sundheim, MUC-5 Evaluation Metrics (ACL Anthology M93-1007), PDF, scoring categories COR, PAR, INC, MIS, SPU, NON
- URL: https://aclanthology.org/M93-1007/

### C50. The Federal Register's agency, action and docket fields follow different conventions from the key: agencies list the parent department first, actions keep the full stop, docket_ids carry 'Docket No.' and extra identifiers.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C50
- Quote: 2026-19757 FR agencies ['Transportation Department', 'Federal Aviation Administration']; action 'Final rule; request for comments.'; docket_ids ['Docket No. FAA-2026-11786', 'Project Identifier …'] vs key 'Federal Aviation Administration' / 'Final rule; request for comments' / 'FAA-2026-11786'
- Source: own run (first-party, extraction-guide-v5 package)

### C51. The key's checker_note column carries 8 judgement calls, among them the two corrections, the HUD date and which EPA deadline counts.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C51
- Quote: checker_note rows: 8 (incl. C1-2026-18574, 2026-19786, 2026-19743, 2026-19774)
- Source: own run (first-party, extraction-guide-v5 package)

### C52. Without the two correction notices (18 documents), Gemini and the run without format: date get every field right; gpt-5-nano still fails on its garbled dates; the publisher's data misses one date in each date field.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C52
- Quote: gemini 18/18 all fields; nano effective_date 17/18, comments_due 11/18; no-date-format 18/18 all fields; FR effective_date 17/18, comments_due 17/18; regex 18/18
- Source: own run (first-party, extraction-guide-v5 package)

### C53. A plausibility check (reject any date more than a year either side of the document's publication date) rejects all 8 non-2026 dates in our runs and no correct date.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C53
- Quote: bad years 8, caught by ±1y window 8; correct dates rejected 0; key dates outside window 0
- Source: own run (first-party, extraction-guide-v5 package)

### C54. The zero rule bites only where the key is empty: in this set agency and action are empty in 1 row each, docket_id in none, RIN in 2, and each date field in 10.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C54
- Quote: agency empty 1; action 1; docket_id 0; rin 2; effective_date 10; comments_due 10
- Source: own run (first-party, extraction-guide-v5 package)

### C55. score.py scores a row missing a field's key as failed, and skips a field only when every row declares it not_scored; removing three keys from the Gemini run fails those fields and exits 1.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C55
- Quote: probe_dropall: agency/action/comments_due 0/20 failed 20, overall FAIL, exit 1; probe_dropodd: 18/20 failed 2, exit 1; saved verdicts unchanged
- Source: own run (first-party, extraction-guide-v5 package)

### C56. score.py's normalisation drops the parent department after the first comma, which is how '(FAA), DOT' counts as right for agency.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C56
- Quote: norm_agency: norm_text(re.sub(r'\([^)]*\)', '', s).split(',')[0]); Gemini agency 15 exact + 4 normalised
- Source: own run (first-party, extraction-guide-v5 package)

### C57. On the IRS correction, 26 October 2026 is a date it corrects in another document's DATES section.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C57
- Quote: In proposed rule document 2026-18574 … in the second line of the DATES section, `October 16, 2026'' should read `October 26, 2026''.
- Source: own run (first-party, extraction-guide-v5 package); text of FR C1-2026-18574

### C58. We used an OpenAI-compatible gateway, so switching model meant changing one string.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C58
- Quote: gateway.py posts to <base>/chat/completions with the body's model string; extract.py takes the model name as its only argument
- Source: own run (first-party, extraction-guide-v5 package)

### C59. Gemini's client.interactions.create() takes response_format with type text, mime_type application/json and a schema.

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C59
- Quote: interaction = client.interactions.create( model="gemini-4-hydrogen", input=prompt, response_format={ "type": "text", "mime_type": "application/json", "schema": Recipe.model_json_schema() }
- Source: Google Gemini docs, Structured output, Python sample; not executable here (service-side API, no key used)
- URL: https://aistudio.google.com/docs/structured-output

### C60. Running the scripts overwrites the matching files in data/outputs/.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C60
- Quote: extract.py and metadata_as_output.py open their output with mode "w"
- Source: own run (first-party, extraction-guide-v5 package)

### C61. The bar scales with the row count, rounded up, so under 20 rows it means every row.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C61
- Quote: min_right = -(-MIN_RIGHT_OF_20 * len(truth) // 20): 19 of 20, 18 of 18, 10 of 10
- Source: own run (first-party, extraction-guide-v5 package)

### C62. extract.py cuts each document at 8,000 characters, and a field past the cut usually scores missing.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C62
- Quote: MAX_CHARS = 8000; text[:MAX_CHARS] sent to the model; no keyed value sits past 8,000 characters in this set
- Source: own run (first-party, extraction-guide-v5 package)

### C63. plausible_dates.py drops all eight wrong-year dates from the format: date run and no date from any other run; the checked run then scores those fields failed.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C63
- Quote: 8 date(s) outside ±365 days (gpt-5-nano); 0 for gemini, no-date-format, regex; checked run effective_date 1 failed, comments_due 7 failed
- Source: own run (first-party, extraction-guide-v5 package)

### C64. Some rules take effect years after they are published (e.g. FR 2026-11800, published 11 June 2026, effective 30 September 2028).

- Status: VERIFIED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C64
- Quote: "document_number": "2026-11800", … "publication_date": "2026-06-11", "effective_on": "2028-09-30"
- Source: Federal Register API, document 2026-11800, fields publication_date, effective_on
- URL: https://www.federalregister.gov/api/v1/documents/2026-11800.json

### C65. The answer-key fix on the IRS correction moved no overall verdict: each LLM run also fails on its date fields, whatever the agency key says.

- Status: EXECUTED (read 2026-11-13)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C65
- Quote: gemini comments_due 18/20 (2 spurious); gpt-5-nano effective_date 17/20 (2), comments_due 13/20; no-format effective_date 18/20 (2): FAIL on dates alone
- Source: own run (first-party, extraction-guide-v5 package)

### C66. score.py says how many fields it scored and exits non-zero when a field was skipped, unless --allow-skip is passed.

- Status: EXECUTED (read 2026-11-15)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C66
- Quote: federal_register_api_metadata: 'overall: PASS: 3 of 3 fields pass; 3 of 6 fields scored (skipped: agency, action, docket_id)', exit 1; with --allow-skip, exit 0
- Source: own run (first-party, extraction-guide-v5 package)

### C67. Gemini (one of three LLM runs) returned valid JSON for all 20 notices and got 116 of the 120 values right, and still failed: the bar allows no value in a field that should be empty, and it filled four, all on two correction notices, each real text from the document; on the IRS correction (which changes a date in document 2026-18574 from 16 to 26 October) it returned 26 October as comments_due.

- Status: EXECUTED (read 2026-11-15)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C67
- Quote: gemini: agency 19, action 19, docket_id 20, rin 20, effective_date 20, comments_due 18 = 116/120; spurious 1+1+0+0+0+2 = 4 (C1-2026-18574 agency, action, comments_due='2026-10-26'; 2026-19786 comments_due); C1-2026-18574 text: 'In proposed rule document 2026-18574 … `October 16, 2026'' should read `October 26, 2026'''
- Source: own run + FR document C1-2026-18574

### C68. Two more models, GPT-5.6 and Claude Opus 5.5, keyed all 120 cells again in fresh sessions without seeing our key, and both matched it on every cell.

- Status: EXECUTED (read 2026-11-15)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C68
- Quote: score.py on each blind file: every field 20/20, exact + null ok, 0 normalised, 0 missing/wrong/spurious/failed ('overall: PASS: 6 of 6 fields pass')
- Source: own run (first-party; blind re-keys)

### C69. The Federal Register's own data (API metadata, no model involved) agrees with our key on 113 of 120 cells; the seven differences are four naming conventions, two dates the publisher left blank, and one rule call (the IRS correction's agency).

- Status: EXECUTED (read 2026-11-15)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C69
- Quote: comparison table in data/key_checks/README.md: agency 16, action 20, docket_id 19, rin 20, effective_date 19, comments_due 19 = 113 of 120 cells; differences: 2026-19743/19860/19899 agency naming, 2026-19786 'Docket Number' prefix, 2026-19743 effective_date blank, 2026-19888 comments_due blank, C1-2026-18574 agency
- Source: own run (first-party comparison script): Federal Register API metadata (data/fr_metadata.json, fetched 13 Nov 2026) vs data/answer_key.csv

### C70. Counted across all six fields, the run without format: date got 115 of 120 right and gpt-5-nano 108; each run filled four or five values where the key is empty, every one on the two correction notices.

- Status: EXECUTED (read 2026-11-15)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C70
- Quote: no-date-format: 19+19+20+20+18+19 = 115, spurious 1+1+0+0+2+1 = 5; gpt-5-nano: 19+19+20+20+17+13 = 108, spurious 1+1+0+0+2+0 = 4; all spurious rows on C1-2026-18574 or 2026-19786
- Source: own run (first-party)

### C71. On the three fields the publisher's data is scored on (rin, effective_date, comments_due), it and Gemini each got 58 of 60 right; the publisher's two misses were blanks, Gemini's two were values filled where the key is empty, and only the publisher passed.

- Status: EXECUTED (read 2026-11-15)
- Row link: https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C71
- Quote: gemini: rin 20/20 PASS, effective_date 20/20 PASS, comments_due 18/20 (2 spurious) FAIL = 58/60; federal_register_api_metadata: rin 20/20 PASS, effective_date 19/20 (1 missing) PASS, comments_due 19/20 (1 missing) PASS = 58/60
- Source: own run (first-party)

## Removed before publication

Claims the essay dropped or left out, each with the reason.

- ~~C20 The schema descriptions printed in the guide are the exact ones the runs sent.~~ (STRUCK 2026-11-13): (narrowed; see the replacement row)
- ~~C24 The guide's outcome categories reproduce the MUC-5 evaluation categories (Chinchor and Sundheim, 1993), down to the word spurious.~~ (STRUCK 2026-11-13): (narrowed; see the replacement row)

## Cite

- A claim: name the original source (and its locator) first, then the row it was checked in, e.g. "<source>, <locator>. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row <id>, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-<id>."
- The essay: Floyd, Harry (2026). The JSON parsed. Is it right?. The Durability Curve. https://durabilitycurve.com/blog/the-json-parsed-is-it-right/
- This ledger: Floyd, Harry (2026). Claim Ledger: The JSON parsed. Is it right?. The Durability Curve. https://durabilitycurve.com/claims/the-json-parsed-is-it-right/
- Downloads: https://durabilitycurve.com/claims/the-json-parsed-is-it-right.csv · https://durabilitycurve.com/claims/the-json-parsed-is-it-right.json

Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Reuse it, quote it or train on it, with credit to The Durability Curve and a link. Open to AI (robots.txt: search=yes, ai-input=yes, ai-train=yes).