Skip to content

The Durability Curve · free by email

Get the Weekly every Saturday, with the calls we grade, plus each new piece.

107Pieces, dated
1245Claims verified at the primary source
7Instruments, no login

Your address goes to Substack, where you confirm it. Nothing is stored on this site.

The latest issue · Anthropic Says It Has Met Its Own AI Research Threshold. NVIDIA's New Chief Challenged Huang's Bets. Read it first

The JSON parsed. Is it right?

Structured outputs all but guarantee the shape of an extraction. Whether the answer is right is yours to score. How to score an LLM, a regex and the publisher's data, then keep what passes per field.

Claims
71
Verified at the primary source
18
Removed before publication
2
Last checked

71 claims: 18 verified, 52 executed, 1 reported. Plus 2 removed before publication, listed at the end.

Download this ledger: CSV · JSON · Markdown · CC BY 4.0 · Query it from Claude or ChatGPT

What the essay claims

Before: "Structured outputs fixed extraction. If the JSON validates against my schema, the data is good. I spot-check a few rows and ship."

After: "The schema guarantees shape. Validation checks the value is usable. Nothing guarantees it's true, so I score it: 20 checked rows, a bar set before I look, spurious values counted on their own. On our run, three schema-valid runs all failed, the publisher's data passed, and the odd documents caused almost every failure. I keep the 20 rows and the script; the model is the part I expect to change."

The claim ladder

#ClaimEvidenceWhose behaviour it measuresWhat it does not reach (scope)
1Asked for JSON in the prompt, gemini-2.5-flash-lite fenced 20 of 20 replies, so json.loads failed 20 times; after stripping fences 13 of 20 failed a typed schema, all on dates.saved naive run + logone model, one prompt, these 20 documents, 8 Oct 2026other prompts, other models, today's model versions
2gpt-5-nano with the same prompt: 20 parsed, 9 failed the schema, all on dates; a MARAD row passed with a bare date as its effective date.saved naive runsamesame
3With the schema enforced, all three LLM runs were schema-valid on 20 of 20 and none passed the bar.verify_object.py (Object Record)three runs of two small models through a gatewaylarger models; any claim that LLMs "can't" extract
4Every value filled where the key is empty sits on the two correction notices; the IRS correction alone fails each run.score.py problems listssamedocuments without corrections
5With format: date, gpt-5-nano wrote 8 dates with years other than 2026, all valid to the schema and Pydantic; without it, all 23 dates were 2026.saved runs, check_claimsone run eacha general law about format: date (the piece says "one run of each")
6The Federal Register's own data passed the three fields it could be scored on; keyed the other way on MARAD, its effective_date drops to 18 of 20.score.py; blind check alt_keythe publisher's metadata on these 20other publishers; it missed two dates and still passed
7A bar of 18 with the zero rule kept changes no verdict; dropping the zero rule passes two of three runs.bar_variants (blind check)these runs—
8Vendor statements: OpenAI "constrains completed, non-refusal model output to your supplied JSON Schema"; Anthropic "guarantee schema-compliant responses through constrained decoding"; Gemini "always validate values in your application". Both OpenAI and Anthropic list refusal and token-limit exceptions.vendor pages, read 13 Nov 2026vendor documentationwhat the models do in practice
9OpenAI retires gpt-5-nano from its API on 11 Dec 2026 and names gpt-6-luna as the replacement.OpenAI deprecations page, read 13 Nov 2026OpenAI's published schedule—

What would make it wrong

If a reader runs the shipped scorer on our saved outputs and gets different per-field counts from the guide's results grid, then the grid is wrong and the piece is pulled until it is fixed.

If a person checking the 120 answer-key cells finds a disagreement that changes a verdict, then the results section changes and the piece says so.

If current models (e.g. gpt-6-luna) pass the bar on the same 20 documents, then the guide's results are dated, and the update says so; the method claim (score it) does not depend on any model failing.

Re-checked 15 Nov 20261 holding2 untestedThe re-check

The evidence, row by row

Each row is a claim the essay makes, then the evidence recorded for it and the source it was checked against. For verified rows that is usually the exact words, where they sit and the day they were read. The label says how far it was checked.

What the labels mean
Verified
Checked against the primary source itself by a checker who did not draft the piece, usually with the exact words, where they sit and the date read.
Executed
A run or observation done for the piece: code, a query, a count, or a check made live in an app. The claim is what it returned. Run records are not published.
Checked
Checked against its source by a separate checker in the older ledger format, which recorded no quote, locator or date.
Reported
Not verified word for word against a primary source: carried from a secondary source, from one that could not be opened in full, or supported only in part. The essay words it accordingly.
Struck
Drafted, checked and removed before publication. Kept here, crossed out, with the reason.
Excluded
Considered and deliberately left out of the piece. Kept here with the reason.
Sources (18)
  1. The Durability Curve 2 rows
  2. own run 47 rows
  3. our own independent check 1 row
  4. OpenAI API docs 2 rows
  5. Anthropic 2 rows
  6. OpenAI and Anthropic structured-output docs 1 row
  7. Google Gemini docs 3 rows
  8. the three vendors' structured-output docs 1 row
  9. Simon Willison 2 rows
  10. Strands Agents docs 1 row
  11. our own independent re-score 2 rows
  12. the Desk's own run notes 1 row
  13. Cleanlab blog 1 row
  14. Singh 1 row
  15. filing-extraction-benchmark 1 row
  16. DeepLearning.AI course page 1 row
  17. Nancy Chinchor 1 row
  18. Federal Register API 1 row
  1. C01 Executed Read

    Our grader piece sets out a five-step check on twenty items with a scoring sheet, counts false passes and false fails separately, and says to check the key before you blame the grader.

    Result: Here is a five-step check on twenty items for yours. … five steps, a scoring sheet … Count false passes and false fails separately. … check the key before you blame the grader

    The Durability Curve, Your AI Grader Is Only as Good as Its Answer Key (first-party) · durabilitycurve.com

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (The Durability Curve, Your AI Grader Is Only as Good as Its Answer Key (first-party)), https://durabilitycurve.com/blog/your-ai-grader-answer-key/. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C01, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C01 (source read 2026-11-13).

    Report an error
  2. C02 Executed Read

    On 20 Federal Register notices with a strict schema, all three LLM runs returned schema-valid JSON for all 20 documents, and none passed the bar.

    Result: HEADLINE: 3 of 3 LLM runs schema-valid on 20 of 20, 0 of 3 passed

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C02, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C02 (source read 2026-11-13).

    Report an error
  3. C03 Executed Read

    The Federal Register's own structured data passed on the three fields we could score it on (on our key).

    Result: federal_register_api_metadata overall: PASS

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C03, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C03 (source read 2026-11-13).

    Report an error
  4. C04 Executed Read

    One run per setup on 8 October 2026; an AI agent built the answer key from the Federal Register text; a second agent checked all 120 cells on 10 November and found no disagreement; no person has checked them.

    Result: I read the preamble … of all 20 documents and compared all 120 cells … 0 disagreements

    our own independent check (first-party)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (our own independent check (first-party)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C04, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C04 (source read 2026-11-13).

    Report an error
  5. C05 Executed Read

    The documents are ten Rules and ten Proposed Rules from 28 and 29 September 2026, via the public Federal Register API.

    Result: C05 OK types Rule 10, Proposed Rule 10; dates 28 Sep (15), 29 Sep (5)

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C05, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C05 (source read 2026-11-13).

    Report an error
  6. C06 Executed Read

    Ten are FAA airworthiness directives on one template.

    Result: C06 OK 10 titles start 'Airworthiness Directives'

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C06, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C06 (source read 2026-11-13).

    Report an error
  7. C07 Executed Read

    Two are correction notices, and one of those has no AGENCY: or ACTION: line.

    Result: C07 OK IRS C1-2026-18574 has no AGENCY:/ACTION: line

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C07, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C07 (source read 2026-11-13).

    Report an error
  8. C08 Executed Read

    A HUD fee announcement calls its date an applicability date.

    Result: DATES: Applicability date: October 28, 2026.

    own run (first-party, extraction-guide-v5 package); text of FR 2026-19743

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package); text of FR 2026-19743). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C08, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C08 (source read 2026-11-13).

    Report an error
  9. C09 Executed Read

    An EPA proposal lists three deadlines in its DATES section.

    Result: C09 OK EPA 2026-19774 DATES names 12 Nov (comments), 28 Oct (OMB/PRA), 5 Oct (hearing request)

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C09, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C09 (source read 2026-11-13).

    Report an error
  10. C10 Executed Read

    Prompted for JSON, gemini-2.5-flash-lite wrapped all twenty replies in a code fence, so json.loads failed twenty times out of twenty.

    Result: C10 OK naive Gemini: fenced 20/20, json.loads failed 20/20

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C10, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C10 (source read 2026-11-13).

    Report an error
  11. C11 Executed Read

    With the fences stripped, thirteen of twenty still fail a typed schema, every one on a date; the model wrote October 28, 2026 in one row and 2026-10-28 in another.

    Result: C11 OK after stripping fences 13/20 fail the schema, all on dates; 19745='October 28, 2026', 19741='2026-10-28'

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C11, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C11 (source read 2026-11-13).

    Report an error
  12. C12 Executed Read

    gpt-5-nano with the same prompt: all twenty parsed, nine failed the schema, all on dates; on the IRS correction it put October 26, 2026 in both date fields.

    Result: C12 OK naive nano: 20/20 parsed, 9 failed the schema (dates only); IRS both dates 'October 26, 2026'

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C12, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C12 (source read 2026-11-13).

    Report an error
  13. C13 Executed Read

    A MARAD correction's DATES: line is a bare September 28, 2026; gpt-5-nano returned it as the effective date, valid and wrong by our key, which leaves it empty.

    Result: C13 OK MARAD: naive nano passed with effective_date 2026-09-28; key empty; DATES line bare

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C13, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C13 (source read 2026-11-13).

    Report an error
  14. C14 Verified Read

    OpenAI's docs say Structured Outputs constrains completed, non-refusal model output to your supplied JSON Schema.

    Structured Outputs constrains completed, non-refusal model output to your supplied JSON Schema.

    OpenAI API docs, Structured model outputs · introduction; a statement about the documentation, not executable (service-side API, no key used) · developers.openai.com

    Link
    Cite

    OpenAI API docs, Structured model outputs, introduction; a statement about the documentation, not executable (service-side API, no key used), https://developers.openai.com/api/docs/guides/structured-outputs. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C14, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C14 (source read 2026-11-13).

    Report an error
  15. C15 Verified Read

    Anthropic's docs say structured outputs guarantee schema-compliant responses through constrained decoding.

    guarantee schema-compliant responses through constrained decoding

    Anthropic (Claude) docs, Structured outputs · introduction; not executable (service-side API, no key used) · platform.claude.com

    Link
    Cite

    Anthropic (Claude) docs, Structured outputs, introduction; not executable (service-side API, no key used), https://platform.claude.com/docs/en/build-with-claude/structured-outputs. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C15, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C15 (source read 2026-11-13).

    Report an error
  16. C16 Verified Read

    Both list exceptions: a refusal, or a reply cut off at the token limit, can break the shape.

    OpenAI: 'in the case of a refusal … or if for example you reach a max tokens limit and the response is incomplete'. Anthropic: 'guarantee schema compliance in most cases … Refusals … Token limit reached'

    OpenAI and Anthropic structured-output docs · OpenAI: handling edge cases; Anthropic: Invalid outputs section; not executable (service-side API, no key used) · developers.openai.com

    Link
    Cite

    OpenAI and Anthropic structured-output docs, OpenAI: handling edge cases; Anthropic: Invalid outputs section; not executable (service-side API, no key used), https://developers.openai.com/api/docs/guides/structured-outputs. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C16, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C16 (source read 2026-11-13).

    Report an error
  17. C17 Verified Read

    Google's Gemini docs say to always validate values, and to handle schema-compliant but semantically incorrect outputs.

    While output is syntactically correct JSON, always validate values in your application. … schema-compliant but semantically incorrect outputs

    Google Gemini docs, Structured output · Best practices · aistudio.google.com

    Link
    Cite

    Google Gemini docs, Structured output, Best practices, https://aistudio.google.com/docs/structured-output. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C17, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C17 (source read 2026-11-13).

    Report an error
  18. C18 Verified Read

    Syntax: OpenAI takes response_format with type json_schema and strict true; Anthropic takes the schema in output_config.format or a Pydantic model via client.messages.parse(); Gemini takes response_format with type text, mime_type application/json and a schema.

    OpenAI: response_format: { type: "json_schema", json_schema: {"strict": true, "schema": ...} }. Anthropic: JSON outputs ( output_config.format ) … client.messages.parse( . Gemini: response_format={ "type": "text", "mime_type": "application/json", "schema": ...}

    the three vendors' structured-output docs · code samples on each page; not executable here (service-side API, no key used) · developers.openai.com

    Link
    Cite

    the three vendors' structured-output docs, code samples on each page; not executable here (service-side API, no key used), https://developers.openai.com/api/docs/guides/structured-outputs. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C18, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C18 (source read 2026-11-13).

    Report an error
  19. C19 Executed Read

    Pydantic's date type rejects November 2, 2026 and accepts 1124 as a year.

    Result: C19 OK pydantic date rejects 'November 2, 2026' and accepts '1124-06-12'

    own run (first-party, extraction-guide-v5 package) (pydantic 2.14.0)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package) (pydantic 2.14.0)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C19, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C19 (source read 2026-11-13).

    Report an error
  20. C21 Executed Read

    Document 2026-19757's lines: docket FAA-2026-11786, RIN 2120-AA64, effective October 13, 2026, comments by November 12, 2026.

    Result: C21 OK 19757 text carries docket FAA-2026-11786, RIN 2120-AA64, effective October 13, 2026, comments by November 12, 2026

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C21, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C21 (source read 2026-11-13).

    Report an error
  21. C22 Executed Read

    gemini-2.5-flash-lite's output for 2026-19757 is as printed; five fields match the key exactly and agency does not.

    Result: C22 OK Gemini 19757 output matches the printed JSON; exact vs key on 5 fields (agency not exact)

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C22, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C22 (source read 2026-11-13).

    Report an error
  22. C23 Verified Read

    Simon Willison's workshop handout: Crucially, did it get everything right? … Spot-checking is crucial.

    Crucially, did it get everything right? We used a very inexpensive model for this. Spot-checking is crucial.

    Simon Willison, Building software on top of LLMs (PyCon 2025 handout), Structured data extraction · after the llm logs example · building-with-llms-pycon-2025.readthedocs.io

    Link
    Cite

    Simon Willison, Building software on top of LLMs (PyCon 2025 handout), Structured data extraction, after the llm logs example, https://building-with-llms-pycon-2025.readthedocs.io/en/latest/structured-data-extraction.html. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C23, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C23 (source read 2026-11-13).

    Report an error
  23. C25 Verified Read

    The Strands Evals structured-output evaluator requires zero false alarms when there was nothing to find.

    When there was nothing to find, it requires fa to be zero, which catches an invention.

    Strands Agents docs, Structured output evaluator · evaluator description; not executable here (no package run) · strandsagents.com

    Link
    Cite

    Strands Agents docs, Structured output evaluator, evaluator description; not executable here (no package run), https://strandsagents.com/docs/user-guide/evals-sdk/evaluators/structured_output_evaluator/. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C25, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C25 (source read 2026-11-13).

    Report an error
  24. C26 Executed Read

    The results table: all 27 scored cells (rows right and values filled where the key is empty) for the five tools; verdicts FAIL, FAIL, FAIL, PASS, PASS.

    Result: C26 OK results table: 27/27 cells reproduce

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C26, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C26 (source read 2026-11-13).

    Report an error
  25. C27 Executed Read

    A bar of 18 with the zero rule kept changes no verdict; 18 without the zero rule passes two of the three LLM runs.

    Result: min_right=18 max_spur=0: all LLM runs FAIL; min_right=18 max_spur=99: gemini PASS, gpt-5-nano FAIL, no-date-format PASS

    our own independent re-score (first-party)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (our own independent re-score (first-party)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C27, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C27 (source read 2026-11-13).

    Report an error
  26. C28 Executed Read

    Every value filled where the key is empty sits on the two correction notices; the IRS correction alone fails each run under the zero rule.

    Result: C28 OK spurious values only on 2026-19786, C1-2026-18574; C28b OK the IRS correction alone puts a spurious value in agency/action in each LLM run

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C28, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C28 (source read 2026-11-13).

    Report an error
  27. C29 Executed Read

    The regex scored 20 on every field, but was written while looking at these documents.

    Result: regex_baseline: 20/20 on all six fields, overall PASS; code/regex_baseline.py written on the 20 documents (SOURCES.md)

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C29, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C29 (source read 2026-11-13).

    Report an error
  28. C30 Executed Read

    With format: date, gpt-5-nano wrote 1124-06-12 for a 12 November deadline and 1102-11-02 for a 2 November effective date; 8 rows had a year other than 2026, all valid to the schema and Pydantic; without format: date every date was in 2026.

    Result: C30 OK format: date run 8 of 22 dates not 2026 (incl. 1124-06-12, 1102-11-02); no-format run 23 dates, 0 not 2026

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C30, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C30 (source read 2026-11-13).

    Report an error
  29. C31 Executed Read

    On the IRS correction: Gemini filled DEPARTMENT OF THE TREASURY and the title plus Correction; gpt-5-nano DEPARTMENT OF THE TREASURY and Correction; the re-run Internal Revenue Service and Correction; all three took its corrected date; MARAD's date became a comment deadline (Gemini), an effective date (gpt-5-nano) or both (re-run).

    Result: C31 Gemini IRS agency='DEPARTMENT OF THE TREASURY' due=2026-10-26, MARAD due=2026-09-28; nano IRS 'Correction' eff=2026-10-26, MARAD eff=2026-09-28; re-run IRS 'Internal Revenue Service' eff=2026-10-26, MARAD eff and due 2026-09-28

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C31, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C31 (source read 2026-11-13).

    Report an error
  30. C32 Reported Read

    Our first answer key broke the null rule on the IRS correction; we fixed the key and re-scored everything.

    All score files were regenerated after the C1-2026-18574 answer-key fix

    the Desk's own run notes (SOURCES.md line 62) and draft v3 line 222; the earlier key file is not in git · Products/extraction-guide-v5-2026-11-13/SOURCES.md L62

    Link
    Cite

    the Desk's own run notes (SOURCES.md line 62) and draft v3 line 222; the earlier key file is not in git, Products/extraction-guide-v5-2026-11-13/SOURCES.md L62. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C32, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C32 (source read 2026-11-13).

    Report an error
  31. C33 Verified Read

    Cleanlab found every public structured-output benchmark it examined full of wrong answers in the key.

    every public benchmark we examined was full of erroneous and inconsistent ground-truth outputs

    Cleanlab blog (Goh, Mueller), structured output benchmark · findings paragraph · cleanlab.ai

    Link
    Cite

    Cleanlab blog (Goh, Mueller), structured output benchmark, findings paragraph, https://cleanlab.ai/blog/structured-output-benchmark/. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C33, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C33 (source read 2026-11-13).

    Report an error
  32. C34 Executed Read

    On agency, Gemini matched 15 rows exactly and 4 more after clean-up.

    Result: C33 OK Gemini agency exact 15, normalised 4

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C34, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C34 (source read 2026-11-13).

    Report an error
  33. C35 Executed Read

    The Federal Register API missed two dates: the HUD applicability date, and the comment deadline for IRS proposal 2026-19888 (30 November 2026 in the text).

    Result: C34 OK FR API: no effective_on for HUD 2026-19743, no comments_close_on for 2026-19888 (text says November 30, 2026)

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C35, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C35 (source read 2026-11-13).

    Report an error
  34. C36 Executed Read

    Keyed the other way on MARAD's bare date, the Federal Register's effective_date drops to 18 of 20 and fails, while Gemini and the regex still pass it.

    Result: MARAD bare date keyed as effective_date: federal_register_api_metadata 18/20 FAIL; gemini 19/20 PASS; regex 19/20 PASS

    our own independent re-score (first-party)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (our own independent re-score (first-party)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C36, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C36 (source read 2026-11-13).

    Report an error
  35. C37 Executed Read

    Routing dates to the Federal Register API sends 11 rows in each date field to a person, because 10 are blank by design.

    Result: C37 OK FR API blank on 11 effective_date rows and 11 comments_due rows; C41 OK answer key null on 10/20 each

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C37, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C37 (source read 2026-11-13).

    Report an error
  36. C38 Executed Read

    score.py exits non-zero when a field misses the bar, so it can block a deploy.

    Result: C38 OK score.py exits 1 on a FAIL and 0 on a PASS

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C38, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C38 (source read 2026-11-13).

    Report an error
  37. C39 Verified Read

    Anthropic's structured outputs don't support minimum or minLength, and cap a request at 16 parameters that use union types.

    Numerical constraints (such as minimum , maximum , multipleOf ) String constraints ( minLength , maxLength ) … Parameters with union types 16 Total parameters that use anyOf or type arrays

    Anthropic (Claude) docs, Structured outputs · JSON Schema limitations; service-side limit, not executable here · platform.claude.com

    Link
    Cite

    Anthropic (Claude) docs, Structured outputs, JSON Schema limitations; service-side limit, not executable here, https://platform.claude.com/docs/en/build-with-claude/structured-outputs. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C39, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C39 (source read 2026-11-13).

    Report an error
  38. C40 Verified Read

    Gemini supports a subset of the JSON Schema specification.

    Gemini's structured output mode supports a subset of the JSON Schema specification.

    Google Gemini docs, Structured output · JSON Schema support; service-side, not executable here · aistudio.google.com

    Link
    Cite

    Google Gemini docs, Structured output, JSON Schema support; service-side, not executable here, https://aistudio.google.com/docs/structured-output. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C40, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C40 (source read 2026-11-13).

    Report an error
  39. C41 Executed Read

    Both date fields are null on half the rows here (10 of 20 each).

    Result: C41 OK answer key: effective_date null on 10/20, comments_due null on 10/20

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C41, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C41 (source read 2026-11-13).

    Report an error
  40. C42 Verified Read

    OpenAI retires gpt-5-nano from its API on 11 December 2026 and names gpt-6-luna as the replacement.

    Dec 11, 2026 gpt-5-nano , gpt-5-nano-2025-08-07 Deprecated gpt-6-luna Jun 11, 2026

    OpenAI API docs, Deprecations · Upcoming deprecations table; service-side schedule, not executable · developers.openai.com

    Link
    Cite

    OpenAI API docs, Deprecations, Upcoming deprecations table; service-side schedule, not executable, https://developers.openai.com/api/docs/deprecations. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C42, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C42 (source read 2026-11-13).

    Report an error
  41. C43 Verified Read

    A 2026 benchmark of 21 models found near-perfect schema compliance, and the best exact match on values from text was 83.0%.

    We evaluate 21 frontier and open-weight models … models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text

    Singh, Khurdula, Khemlani, Agarwal, The Structured Output Benchmark (arXiv 2604.25359, 28 Apr 2026) · abstract · arxiv.org

    Link
    Cite

    Singh, Khurdula, Khemlani, Agarwal, The Structured Output Benchmark (arXiv 2604.25359, 28 Apr 2026), abstract, https://arxiv.org/abs/2604.25359. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C43, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C43 (source read 2026-11-13).

    Report an error
  42. C44 Verified Read

    filing-extraction-benchmark scores rules, an LLM and a hybrid that asks the model only where the rules abstain, all with one scorer.

    takes the same inputs, produces the same output schema and is scored by the same scorer … hybrid, model asked only where rules abstained

    filing-extraction-benchmark (GitHub README) · README, method and results table; not executable here (repo not run) · github.com

    Link
    Cite

    filing-extraction-benchmark (GitHub README), README, method and results table; not executable here (repo not run), https://github.com/JAYANSHUBADLANI/filing-extraction-benchmark. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C44, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C44 (source read 2026-11-13).

    Report an error
  43. C45 Executed Read

    Scoring needs no API key; re-scoring our saved runs exits 1 because our LLM runs fail the bar; with no key the extraction scripts stop at the key message and write nothing.

    Result: clean-unzip reader test: score exit 1 (8 files); extract and naive_extract print 'Set OPENAI_API_KEY (or AI_GATEWAY_API_KEY) first.' exit 1; data/outputs unchanged (19 files)

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C45, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C45 (source read 2026-11-13).

    Report an error
  44. C46 Executed Read

    The Metric Validity Audit's tagline is: Your metric is green. Should you believe it?

    Result: Your metric is green. Should you believe it?

    The Durability Curve, Metric Validity Audit (first-party) · durabilitycurve.com

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (The Durability Curve, Metric Validity Audit (first-party)), https://durabilitycurve.com/tools/metric-validity-audit/. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C46, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C46 (source read 2026-11-13).

    Report an error
  45. C47 Verified Read

    DeepLearning.AI's Function-calling and data extraction with LLMs is taught by Jiantao Jiao and Venkat Srinivasan of Nexusflow, with a Structured Extraction lesson on an open model.

    Instructors: Jiantao Jiao, Venkat Srinivasan Nexusflow … Structured Extraction … NexusRavenV2-13B, an open source model

    DeepLearning.AI course page · instructors and syllabus · deeplearning.ai

    Link
    Cite

    DeepLearning.AI course page, instructors and syllabus, https://www.deeplearning.ai/courses/function-calling-and-data-extraction-with-llms. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C47, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C47 (source read 2026-11-13).

    Report an error
  46. C48 Verified Read

    Willison's handout does schema extraction from a real FEMA briefing PDF.

    FEMA's Daily Operation Briefing PDF is a fascinating document.

    Simon Willison, PyCon 2025 handout · Something a bit more impressive · building-with-llms-pycon-2025.readthedocs.io

    Link
    Cite

    Simon Willison, PyCon 2025 handout, Something a bit more impressive, https://building-with-llms-pycon-2025.readthedocs.io/en/latest/structured-data-extraction.html. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C48, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C48 (source read 2026-11-13).

    Report an error
  47. C49 Executed Read

    score.py scores any tool's output written as one JSON line per document with document_number and a parsed object (or null for a failed call).

    Result: rowfmt.jsonl (the regex output rewritten as document_number + parsed only): 20 rows, 0 failed or absent, overall PASS, exit 0

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C49, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C49 (source read 2026-11-13).

    Report an error
  48. C20b Executed Read

    C20b The schema descriptions printed in the guide are the exact ones the first two runs sent; the third run appends 'Format YYYY-MM-DD.' to both date descriptions and drops format: date.

    Result: 6 of 6 descriptions identical to code/schema.py; extract.py plain_dates() pops 'format' and adds ' Format YYYY-MM-DD.' to effective_date and comments_due

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C20b, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C20b (source read 2026-11-13).

    Report an error
  49. C24b Verified Read

    C24b Most of the guide's outcomes line up with the MUC-5 evaluation categories (Chinchor and Sundheim, 1993), down to the word spurious; the guide splits exact from normalised and adds failed.

    If the response has a fill which has no corresponding fill in the key, the category is spurious (SPU).

    Nancy Chinchor, Beth Sundheim, MUC-5 Evaluation Metrics (ACL Anthology M93-1007) · PDF, scoring categories COR, PAR, INC, MIS, SPU, NON · aclanthology.org

    Link
    Cite

    Nancy Chinchor, Beth Sundheim, MUC-5 Evaluation Metrics (ACL Anthology M93-1007), PDF, scoring categories COR, PAR, INC, MIS, SPU, NON, https://aclanthology.org/M93-1007/. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C24b, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C24b (source read 2026-11-13).

    Report an error
  50. C50 Executed Read

    The Federal Register's agency, action and docket fields follow different conventions from the key: agencies list the parent department first, actions keep the full stop, docket_ids carry 'Docket No.' and extra identifiers.

    Result: 2026-19757 FR agencies ['Transportation Department', 'Federal Aviation Administration']; action 'Final rule; request for comments.'; docket_ids ['Docket No. FAA-2026-11786', 'Project Identifier …'] vs key 'Federal Aviation Administration' / 'Final rule; request for comments' / 'FAA-2026-11786'

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C50, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C50 (source read 2026-11-13).

    Report an error
  51. C51 Executed Read

    The key's checker_note column carries 8 judgement calls, among them the two corrections, the HUD date and which EPA deadline counts.

    Result: checker_note rows: 8 (incl. C1-2026-18574, 2026-19786, 2026-19743, 2026-19774)

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C51, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C51 (source read 2026-11-13).

    Report an error
  52. C52 Executed Read

    Without the two correction notices (18 documents), Gemini and the run without format: date get every field right; gpt-5-nano still fails on its garbled dates; the publisher's data misses one date in each date field.

    Result: gemini 18/18 all fields; nano effective_date 17/18, comments_due 11/18; no-date-format 18/18 all fields; FR effective_date 17/18, comments_due 17/18; regex 18/18

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C52, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C52 (source read 2026-11-13).

    Report an error
  53. C53 Executed Read

    A plausibility check (reject any date more than a year either side of the document's publication date) rejects all 8 non-2026 dates in our runs and no correct date.

    Result: bad years 8, caught by ±1y window 8; correct dates rejected 0; key dates outside window 0

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C53, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C53 (source read 2026-11-13).

    Report an error
  54. C54 Executed Read

    The zero rule bites only where the key is empty: in this set agency and action are empty in 1 row each, docket_id in none, RIN in 2, and each date field in 10.

    Result: agency empty 1; action 1; docket_id 0; rin 2; effective_date 10; comments_due 10

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C54, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C54 (source read 2026-11-13).

    Report an error
  55. C55 Executed Read

    score.py scores a row missing a field's key as failed, and skips a field only when every row declares it not_scored; removing three keys from the Gemini run fails those fields and exits 1.

    Result: probe_dropall: agency/action/comments_due 0/20 failed 20, overall FAIL, exit 1; probe_dropodd: 18/20 failed 2, exit 1; saved verdicts unchanged

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C55, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C55 (source read 2026-11-13).

    Report an error
  56. C56 Executed Read

    score.py's normalisation drops the parent department after the first comma, which is how '(FAA), DOT' counts as right for agency.

    Result: norm_agency: norm_text(re.sub(r'\([^)]*\)', '', s).split(',')[0]); Gemini agency 15 exact + 4 normalised

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C56, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C56 (source read 2026-11-13).

    Report an error
  57. C57 Executed Read

    On the IRS correction, 26 October 2026 is a date it corrects in another document's DATES section.

    Result: In proposed rule document 2026-18574 … in the second line of the DATES section, `October 16, 2026'' should read `October 26, 2026''.

    own run (first-party, extraction-guide-v5 package); text of FR C1-2026-18574

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package); text of FR C1-2026-18574). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C57, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C57 (source read 2026-11-13).

    Report an error
  58. C58 Executed Read

    We used an OpenAI-compatible gateway, so switching model meant changing one string.

    Result: gateway.py posts to <base>/chat/completions with the body's model string; extract.py takes the model name as its only argument

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C58, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C58 (source read 2026-11-13).

    Report an error
  59. C59 Verified Read

    Gemini's client.interactions.create() takes response_format with type text, mime_type application/json and a schema.

    interaction = client.interactions.create( model="gemini-4-hydrogen", input=prompt, response_format={ "type": "text", "mime_type": "application/json", "schema": Recipe.model_json_schema() }

    Google Gemini docs, Structured output · Python sample; not executable here (service-side API, no key used) · aistudio.google.com

    Link
    Cite

    Google Gemini docs, Structured output, Python sample; not executable here (service-side API, no key used), https://aistudio.google.com/docs/structured-output. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C59, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C59 (source read 2026-11-13).

    Report an error
  60. C60 Executed Read

    Running the scripts overwrites the matching files in data/outputs/.

    Result: extract.py and metadata_as_output.py open their output with mode "w"

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C60, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C60 (source read 2026-11-13).

    Report an error
  61. C61 Executed Read

    The bar scales with the row count, rounded up, so under 20 rows it means every row.

    Result: min_right = -(-MIN_RIGHT_OF_20 * len(truth) // 20): 19 of 20, 18 of 18, 10 of 10

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C61, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C61 (source read 2026-11-13).

    Report an error
  62. C62 Executed Read

    extract.py cuts each document at 8,000 characters, and a field past the cut usually scores missing.

    Result: MAX_CHARS = 8000; text[:MAX_CHARS] sent to the model; no keyed value sits past 8,000 characters in this set

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C62, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C62 (source read 2026-11-13).

    Report an error
  63. C63 Executed Read

    plausible_dates.py drops all eight wrong-year dates from the format: date run and no date from any other run; the checked run then scores those fields failed.

    Result: 8 date(s) outside ±365 days (gpt-5-nano); 0 for gemini, no-date-format, regex; checked run effective_date 1 failed, comments_due 7 failed

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C63, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C63 (source read 2026-11-13).

    Report an error
  64. C64 Verified Read

    Some rules take effect years after they are published (e.g. FR 2026-11800, published 11 June 2026, effective 30 September 2028).

    "document_number": "2026-11800", … "publication_date": "2026-06-11", "effective_on": "2028-09-30"

    Federal Register API, document 2026-11800 · fields publication_date, effective_on · federalregister.gov

    Link
    Cite

    Federal Register API, document 2026-11800, fields publication_date, effective_on, https://www.federalregister.gov/api/v1/documents/2026-11800.json. Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C64, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C64 (source read 2026-11-13).

    Report an error
  65. C65 Executed Read

    The answer-key fix on the IRS correction moved no overall verdict: each LLM run also fails on its date fields, whatever the agency key says.

    Result: gemini comments_due 18/20 (2 spurious); gpt-5-nano effective_date 17/20 (2), comments_due 13/20; no-format effective_date 18/20 (2): FAIL on dates alone

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C65, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C65 (source read 2026-11-13).

    Report an error
  66. C66 Executed Read

    score.py says how many fields it scored and exits non-zero when a field was skipped, unless --allow-skip is passed.

    Result: federal_register_api_metadata: 'overall: PASS: 3 of 3 fields pass; 3 of 6 fields scored (skipped: agency, action, docket_id)', exit 1; with --allow-skip, exit 0

    own run (first-party, extraction-guide-v5 package)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party, extraction-guide-v5 package)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C66, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C66 (source read 2026-11-15).

    Report an error
  67. C67 Executed Read

    Gemini (one of three LLM runs) returned valid JSON for all 20 notices and got 116 of the 120 values right, and still failed: the bar allows no value in a field that should be empty, and it filled four, all on two correction notices, each real text from the document; on the IRS correction (which changes a date in document 2026-18574 from 16 to 26 October) it returned 26 October as comments_due.

    Result: gemini: agency 19, action 19, docket_id 20, rin 20, effective_date 20, comments_due 18 = 116/120; spurious 1+1+0+0+0+2 = 4 (C1-2026-18574 agency, action, comments_due='2026-10-26'; 2026-19786 comments_due); C1-2026-18574 text: 'In proposed rule document 2026-18574 … `October 16, 2026'' should read `October 26, 2026'''

    own run + FR document C1-2026-18574

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run + FR document C1-2026-18574). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C67, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C67 (source read 2026-11-15).

    Report an error
  68. C68 Executed Read

    Two more models, GPT-5.6 and Claude Opus 5.5, keyed all 120 cells again in fresh sessions without seeing our key, and both matched it on every cell.

    Result: score.py on each blind file: every field 20/20, exact + null ok, 0 normalised, 0 missing/wrong/spurious/failed ('overall: PASS: 6 of 6 fields pass')

    own run (first-party; blind re-keys)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party; blind re-keys)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C68, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C68 (source read 2026-11-15).

    Report an error
  69. C69 Executed Read

    The Federal Register's own data (API metadata, no model involved) agrees with our key on 113 of 120 cells; the seven differences are four naming conventions, two dates the publisher left blank, and one rule call (the IRS correction's agency).

    Result: comparison table in data/key_checks/README.md: agency 16, action 20, docket_id 19, rin 20, effective_date 19, comments_due 19 = 113 of 120 cells; differences: 2026-19743/19860/19899 agency naming, 2026-19786 'Docket Number' prefix, 2026-19743 effective_date blank, 2026-19888 comments_due blank, C1-2026-18574 agency

    own run (first-party comparison script): Federal Register API metadata (data/fr_metadata.json, fetched 13 Nov 2026) vs data/answer_key.csv

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party comparison script): Federal Register API metadata (data/fr_metadata.json, fetched 13 Nov 2026) vs data/answer_key.csv). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C69, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C69 (source read 2026-11-15).

    Report an error
  70. C70 Executed Read

    Counted across all six fields, the run without format: date got 115 of 120 right and gpt-5-nano 108; each run filled four or five values where the key is empty, every one on the two correction notices.

    Result: no-date-format: 19+19+20+20+18+19 = 115, spurious 1+1+0+0+2+1 = 5; gpt-5-nano: 19+19+20+20+17+13 = 108, spurious 1+1+0+0+2+0 = 4; all spurious rows on C1-2026-18574 or 2026-19786

    own run (first-party)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C70, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C70 (source read 2026-11-15).

    Report an error
  71. C71 Executed Read

    On the three fields the publisher's data is scored on (rin, effective_date, comments_due), it and Gemini each got 58 of 60 right; the publisher's two misses were blanks, Gemini's two were values filled where the key is empty, and only the publisher passed.

    Result: gemini: rin 20/20 PASS, effective_date 20/20 PASS, comments_due 18/20 (2 spurious) FAIL = 58/60; federal_register_api_metadata: rin 20/20 PASS, effective_date 19/20 (1 missing) PASS, comments_due 19/20 (1 missing) PASS = 58/60

    own run (first-party)

    A run or observation done for this piece. The run record is not published.

    Link
    Cite

    A run or observation by The Durability Curve (own run (first-party)). Checked in: Claim Ledger, "The JSON parsed. Is it right?", The Durability Curve, row C71, https://durabilitycurve.com/claims/the-json-parsed-is-it-right/#c-C71 (source read 2026-11-15).

    Report an error

Removed before publication

Claims that were drafted, checked and taken out, or considered and left out. The essay does not make them; they stay here with the reason.

  1. C20 Struck

    C20 The schema descriptions printed in the guide are the exact ones the runs sent.

    Why: (narrowed; see the replacement row)

  2. C24 Struck

    C24 The guide's outcome categories reproduce the MUC-5 evaluation categories (Chinchor and Sundheim, 1993), down to the word spurious.

    Why: (narrowed; see the replacement row)

Cite and reuse

A single claim
Use the row's Cite button. It names the original source first, then this row, which carries its own link.
This ledger
Floyd, Harry (2026). Claim Ledger: The JSON parsed. Is it right? The Durability Curve. https://durabilitycurve.com/claims/the-json-parsed-is-it-right/
The essay
Floyd, Harry (2026). The JSON parsed. Is it right? The Durability Curve. https://durabilitycurve.com/blog/the-json-parsed-is-it-right/

The ledger data is licensed CC BY 4.0: reuse it, quote it or train on it, with credit to The Durability Curve and a link. Read the licence. Machine copies: JSON, CSV, Markdown.

The newsletter · Free

Subscribe to get each new piece in your inbox, with something to use in every one.

Several essays a week, plus Notes through the week.

Signup completes on Substack. Unsubscribe from any email.