Claim ledger

The Seven-Layer Agent Audit

Your agent is starved on one layer of seven. It is rarely the harness everyone argues about.

Read the essay On Substack Published Tests Law I: Bottleneck Migration

Claims
8
Verified at the primary source
8
Removed before publication
0
Last checked

8 claims: 8 verified.

Download this ledger: CSV · JSON · Markdown · CC BY 4.0 · Query it from Claude or ChatGPT

The evidence, row by row

Each row is a claim the essay makes, then the evidence recorded for it and the source it was checked against. For verified rows that is usually the exact words, where they sit and the day they were read. The label says how far it was checked.

What the labels mean
Verified
Checked against the primary source itself by a checker who did not draft the piece, usually with the exact words, where they sit and the date read.
Executed
A run or observation done for the piece: code, a query, a count, or a check made live in an app. The claim is what it returned. Run records are not published.
Checked
Checked against its source by a separate checker in the older ledger format, which recorded no quote, locator or date.
Reported
Not verified word for word against a primary source: carried from a secondary source, from one that could not be opened in full, or supported only in part. The essay words it accordingly.
Struck
Drafted, checked and removed before publication. Kept here, crossed out, with the reason.
Excluded
Considered and deliberately left out of the piece. Kept here with the reason.
  1. SL1 Verified Read

    SWE-agent was built by a team at Princeton Language and Intelligence, Princeton University, and was published in the NeurIPS 2024 Main Conference Track.

    Princeton Language and Intelligence, Princeton University … Advances in Neural Information Processing Systems 37 (NeurIPS 2024) Main Conference Track

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan and Ofir Press, SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793) · Author affiliations, title block (arXiv HTML v3) · NeurIPS 2024 proceedings page, citation line · arxiv.org

    Link
    Cite

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan and Ofir Press, SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793), Author affiliations, title block (arXiv HTML v3) · NeurIPS 2024 proceedings page, citation line, https://arxiv.org/abs/2405.15793. Checked in: Claim Ledger, "The Seven-Layer Agent Audit", The Durability Curve, row SL1, https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL1 (source read 2026-10-06).

    Report an error
  2. SL2 Verified Read

    Using GPT-4 Turbo, SWE-agent resolved 12.47% of the 2,294 tasks in the SWE-bench test set, against a previous best of 3.8% by a non-interactive, retrieval-augmented system.

    SWE-agent solves 12.47% of the 2,294 SWE-bench test tasks … previous best resolve rate of 3.8% by a non-interactive, retrieval-augmented system

    John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793) · §1 Introduction · §5 Results (286 of 2,294) · arxiv.org

    Link
    Cite

    John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793), §1 Introduction · §5 Results (286 of 2,294), https://arxiv.org/abs/2405.15793. Checked in: Claim Ledger, "The Seven-Layer Agent Audit", The Durability Curve, row SL2, https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL2 (source read 2026-10-06).

    Report an error
  3. SL3 Verified Read

    The 3.8% previous best was the retrieval-augmented baseline running on Claude 3 Opus (3.79% of the full test set); the same baseline on GPT-4 Turbo resolved 1.31%.

    RAG w/ GPT-4 Turbo 1.31 … w/ Claude 3 Opus 3.79

    John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793) · Table 1, SWE-bench column, % Resolved (same values in v1, v2 and v3) · arxiv.org

    Link
    Cite

    John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793), Table 1, SWE-bench column, % Resolved (same values in v1, v2 and v3), https://arxiv.org/abs/2405.15793. Checked in: Claim Ledger, "The Seven-Layer Agent Audit", The Durability Curve, row SL3, https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL3 (source read 2026-10-06).

    Report an error
  4. SL4 Verified Read

    With the model held at GPT-4 Turbo on the 300-task SWE-bench Lite split, SWE-agent resolved 18.00% of tasks against 11.00% for an agent working through the plain Linux shell, a 64% relative increase.

    An LM-friendly ACI's value is confirmed by SWE-agent's 64% relative increase compared to Shell-only, both with GPT-4 Turbo.

    John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793) · §5 Results, first paragraph · Table 1, SWE-bench Lite column (Shell-only agent w/ GPT-4 Turbo 11.00; SWE-agent w/ GPT-4 Turbo 18.00) · arxiv.org

    Link
    Cite

    John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793), §5 Results, first paragraph · Table 1, SWE-bench Lite column (Shell-only agent w/ GPT-4 Turbo 11.00; SWE-agent w/ GPT-4 Turbo 18.00), https://arxiv.org/abs/2405.15793. Checked in: Claim Ledger, "The Seven-Layer Agent Audit", The Durability Curve, row SL4, https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL4 (source read 2026-10-06).

    Report an error
  5. SL5 Verified Read

    SWE-agent's interface changed what the agent could see and do: commands for searching, viewing and editing files, guardrails against common mistakes, and concise feedback on each command's effects.

    a small set of simple actions for viewing, searching through and editing files … guardrails to prevent common mistakes … concise feedback about a command's effects

    John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793) · §1 Introduction · §3, opening paragraph (search/navigation, file viewer, file editor, context management) · arxiv.org

    Link
    Cite

    John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793), §1 Introduction · §3, opening paragraph (search/navigation, file viewer, file editor, context management), https://arxiv.org/abs/2405.15793. Checked in: Claim Ledger, "The Seven-Layer Agent Audit", The Durability Curve, row SL5, https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL5 (source read 2026-10-06).

    Report an error
  6. SL6 Verified Read

    In SkillsBench, curated skills raised the average pass rate from 33.9% to 50.5%, a gain of 16.6 percentage points across 87 tasks and 18 model-harness configurations; versions 1 to 3 of the paper reported 16.2 points on 84 evaluated tasks (of 86).

    Curated Skills raise the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain)

    Xiangyi Li et al., SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv:2602.12670 · Abstract (v4, 14 June 2026) · v1–v3 abstract: "Curated Skills raise average pass rate by 16.2 percentage points" · v1 §3.5 Metrics, "Pass Rate": "a fixed denominator of 84 (the number of evaluated tasks)" · arxiv.org

    Link
    Cite

    Xiangyi Li et al., SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv:2602.12670, Abstract (v4, 14 June 2026) · v1–v3 abstract: "Curated Skills raise average pass rate by 16.2 percentage points" · v1 §3.5 Metrics, "Pass Rate": "a fixed denominator of 84 (the number of evaluated tasks)", https://arxiv.org/abs/2602.12670. Checked in: Claim Ledger, "The Seven-Layer Agent Audit", The Durability Curve, row SL6, https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL6 (source read 2026-10-11).

    Report an error
  7. SL7 Verified Read

    Skills that a model wrote for itself gave no average benefit. In versions 1 to 3 they averaged 1.3 points below the no-skills baseline; in version 4 they landed below it on all three configurations tested, by 8.1 to 11.5 points.

    Self-generated Skills land below the no-Skills baseline on all three configurations

    Xiangyi Li et al., SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv:2602.12670 · v4 §5.1.1, paragraph "Self-generated Skills do not substitute for curated ones" (−8.1, −11.3 and −11.5 pp) · v1 §4.1.1, Finding 3 (–1.3pp on average) · arxiv.org

    Link
    Cite

    Xiangyi Li et al., SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv:2602.12670, v4 §5.1.1, paragraph "Self-generated Skills do not substitute for curated ones" (−8.1, −11.3 and −11.5 pp) · v1 §4.1.1, Finding 3 (–1.3pp on average), https://arxiv.org/abs/2602.12670. Checked in: Claim Ledger, "The Seven-Layer Agent Audit", The Durability Curve, row SL7, https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL7 (source read 2026-10-06).

    Report an error
  8. SL8 Verified Read

    Language models use long contexts unevenly: performance is often highest when the relevant information is at the beginning or end of the input, and degrades significantly when it sits in the middle.

    often highest when relevant information occurs at the beginning or end … significantly degrades when models must access relevant information in the middle of long contexts

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang, Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics 12, 2024, pp. 157–173 · Abstract · aclanthology.org

    Link
    Cite

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang, Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics 12, 2024, pp. 157–173, Abstract, https://aclanthology.org/2024.tacl-1.9/. Checked in: Claim Ledger, "The Seven-Layer Agent Audit", The Durability Curve, row SL8, https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL8 (source read 2026-10-06).

    Report an error

Cite and reuse

A single claim
Use the row's Cite button. It names the original source first, then this row, which carries its own link.
This ledger
Floyd, Harry (2026). Claim Ledger: The Seven-Layer Agent Audit. The Durability Curve. https://durabilitycurve.com/claims/the-seven-layer-agent-audit/
The essay
Floyd, Harry (2026). The Seven-Layer Agent Audit. The Durability Curve. https://durabilitycurve.com/blog/the-seven-layer-agent-audit/

The ledger data is licensed CC BY 4.0: reuse it, quote it or train on it, with credit to The Durability Curve and a link. Read the licence. Machine copies: JSON, CSV, Markdown.