--- title: "Claim Ledger: The Seven-Layer Agent Audit" description: "Your agent is starved on one layer of seven. It is rarely the harness everyone argues about." author: "Harry Floyd" publication: "The Durability Curve" canonical: "https://durabilitycurve.com/claims/the-seven-layer-agent-audit/" essay: "https://durabilitycurve.com/blog/the-seven-layer-agent-audit/" substack: "https://harryfloyd.substack.com/p/the-seven-layer-agent-audit" published: "2026-06-17" ledger_date: "2026-10-11" law: "Law I" claims: 8 removed_before_publication: 0 --- # Claim Ledger: The Seven-Layer Agent Audit *Your agent is starved on one layer of seven. It is rarely the harness everyone argues about.* Essay: https://durabilitycurve.com/blog/the-seven-layer-agent-audit/ Ledger (canonical, cite this): https://durabilitycurve.com/claims/the-seven-layer-agent-audit/ Published 2026-06-17 · last verified 2026-10-11 ## The evidence, row by row Labels: VERIFIED = checked against the primary source itself by a checker who did not draft the piece, usually with the exact words, where they sit and the date read · EXECUTED = a run or observation done for the piece (code, a query, a count, or a check made live in an app); the claim is what it returned; run records are not published · CHECKED = checked against its source by a separate checker in the older ledger format, which recorded no quote, locator or date · REPORTED = not verified word for word against a primary source (secondary, not openable in full, or supported only in part); the essay words it accordingly · STRUCK = drafted, checked and removed before publication · EXCLUDED = considered and deliberately left out. ### SL1. SWE-agent was built by a team at Princeton Language and Intelligence, Princeton University, and was published in the NeurIPS 2024 Main Conference Track. - Status: VERIFIED (read 2026-10-06) - Row link: https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL1 - Quote: Princeton Language and Intelligence, Princeton University … Advances in Neural Information Processing Systems 37 (NeurIPS 2024) Main Conference Track - Source: John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan and Ofir Press, SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793), Author affiliations, title block (arXiv HTML v3) · NeurIPS 2024 proceedings page, citation line - URL: https://arxiv.org/abs/2405.15793 ### SL2. Using GPT-4 Turbo, SWE-agent resolved 12.47% of the 2,294 tasks in the SWE-bench test set, against a previous best of 3.8% by a non-interactive, retrieval-augmented system. - Status: VERIFIED (read 2026-10-06) - Row link: https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL2 - Quote: SWE-agent solves 12.47% of the 2,294 SWE-bench test tasks … previous best resolve rate of 3.8% by a non-interactive, retrieval-augmented system - Source: John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793), §1 Introduction · §5 Results (286 of 2,294) - URL: https://arxiv.org/abs/2405.15793 ### SL3. The 3.8% previous best was the retrieval-augmented baseline running on Claude 3 Opus (3.79% of the full test set); the same baseline on GPT-4 Turbo resolved 1.31%. - Status: VERIFIED (read 2026-10-06) - Row link: https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL3 - Quote: RAG w/ GPT-4 Turbo 1.31 … w/ Claude 3 Opus 3.79 - Source: John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793), Table 1, SWE-bench column, % Resolved (same values in v1, v2 and v3) - URL: https://arxiv.org/abs/2405.15793 ### SL4. With the model held at GPT-4 Turbo on the 300-task SWE-bench Lite split, SWE-agent resolved 18.00% of tasks against 11.00% for an agent working through the plain Linux shell, a 64% relative increase. - Status: VERIFIED (read 2026-10-06) - Row link: https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL4 - Quote: An LM-friendly ACI's value is confirmed by SWE-agent's 64% relative increase compared to Shell-only, both with GPT-4 Turbo. - Source: John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793), §5 Results, first paragraph · Table 1, SWE-bench Lite column (Shell-only agent w/ GPT-4 Turbo 11.00; SWE-agent w/ GPT-4 Turbo 18.00) - URL: https://arxiv.org/abs/2405.15793 ### SL5. SWE-agent's interface changed what the agent could see and do: commands for searching, viewing and editing files, guardrails against common mistakes, and concise feedback on each command's effects. - Status: VERIFIED (read 2026-10-06) - Row link: https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL5 - Quote: a small set of simple actions for viewing, searching through and editing files … guardrails to prevent common mistakes … concise feedback about a command's effects - Source: John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793), §1 Introduction · §3, opening paragraph (search/navigation, file viewer, file editor, context management) - URL: https://arxiv.org/abs/2405.15793 ### SL6. In SkillsBench, curated skills raised the average pass rate from 33.9% to 50.5%, a gain of 16.6 percentage points across 87 tasks and 18 model-harness configurations; versions 1 to 3 of the paper reported 16.2 points on 84 evaluated tasks (of 86). - Status: VERIFIED (read 2026-10-11) - Row link: https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL6 - Quote: Curated Skills raise the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain) - Source: Xiangyi Li et al., SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv:2602.12670, Abstract (v4, 14 June 2026) · v1–v3 abstract: "Curated Skills raise average pass rate by 16.2 percentage points" · v1 §3.5 Metrics, "Pass Rate": "a fixed denominator of 84 (the number of evaluated tasks)" - URL: https://arxiv.org/abs/2602.12670 ### SL7. Skills that a model wrote for itself gave no average benefit. In versions 1 to 3 they averaged 1.3 points below the no-skills baseline; in version 4 they landed below it on all three configurations tested, by 8.1 to 11.5 points. - Status: VERIFIED (read 2026-10-06) - Row link: https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL7 - Quote: Self-generated Skills land below the no-Skills baseline on all three configurations - Source: Xiangyi Li et al., SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv:2602.12670, v4 §5.1.1, paragraph "Self-generated Skills do not substitute for curated ones" (−8.1, −11.3 and −11.5 pp) · v1 §4.1.1, Finding 3 (–1.3pp on average) - URL: https://arxiv.org/abs/2602.12670 ### SL8. Language models use long contexts unevenly: performance is often highest when the relevant information is at the beginning or end of the input, and degrades significantly when it sits in the middle. - Status: VERIFIED (read 2026-10-06) - Row link: https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL8 - Quote: often highest when relevant information occurs at the beginning or end … significantly degrades when models must access relevant information in the middle of long contexts - Source: Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang, Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics 12, 2024, pp. 157–173, Abstract - URL: https://aclanthology.org/2024.tacl-1.9/ ## Cite - A claim: name the original source (and its locator) first, then the row it was checked in, e.g. ", . Checked in: Claim Ledger, "The Seven-Layer Agent Audit", The Durability Curve, row , https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-." - The essay: Floyd, Harry (2026). The Seven-Layer Agent Audit. The Durability Curve. https://durabilitycurve.com/blog/the-seven-layer-agent-audit/ - This ledger: Floyd, Harry (2026). Claim Ledger: The Seven-Layer Agent Audit. The Durability Curve. https://durabilitycurve.com/claims/the-seven-layer-agent-audit/ - Downloads: https://durabilitycurve.com/claims/the-seven-layer-agent-audit.csv · https://durabilitycurve.com/claims/the-seven-layer-agent-audit.json Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Reuse it, quote it or train on it, with credit to The Durability Curve and a link. Open to AI (robots.txt: search=yes, ai-input=yes, ai-train=yes).