{
 "ledger": {
  "title": "Claim Ledger: The Seven-Layer Agent Audit",
  "url": "https://durabilitycurve.com/claims/the-seven-layer-agent-audit/",
  "essay": {
   "title": "The Seven-Layer Agent Audit",
   "url": "https://durabilitycurve.com/blog/the-seven-layer-agent-audit/",
   "published": "2026-06-17"
  },
  "last_checked": "2026-10-11",
  "format": "current",
  "counts": {
   "claims": 8,
   "by_status": {
    "VERIFIED": 8
   },
   "verified_at_primary": 8,
   "removed": 0
  },
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "attribution": "The Durability Curve (Harry Floyd), https://durabilitycurve.com/claims/the-seven-layer-agent-audit/"
 },
 "rows": [
  {
   "id": "SL1",
   "url": "https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL1",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "SWE-agent was built by a team at Princeton Language and Intelligence, Princeton University, and was published in the NeurIPS 2024 Main Conference Track.",
   "quote": "Princeton Language and Intelligence, Princeton University … Advances in Neural Information Processing Systems 37 (NeurIPS 2024) Main Conference Track",
   "removal_reason": "",
   "source": "John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan and Ofir Press, SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793)",
   "source_url": "https://arxiv.org/abs/2405.15793",
   "locator": "Author affiliations, title block (arXiv HTML v3) · NeurIPS 2024 proceedings page, citation line",
   "date_read": "2026-10-06"
  },
  {
   "id": "SL2",
   "url": "https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL2",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Using GPT-4 Turbo, SWE-agent resolved 12.47% of the 2,294 tasks in the SWE-bench test set, against a previous best of 3.8% by a non-interactive, retrieval-augmented system.",
   "quote": "SWE-agent solves 12.47% of the 2,294 SWE-bench test tasks … previous best resolve rate of 3.8% by a non-interactive, retrieval-augmented system",
   "removal_reason": "",
   "source": "John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793)",
   "source_url": "https://arxiv.org/abs/2405.15793",
   "locator": "§1 Introduction · §5 Results (286 of 2,294)",
   "date_read": "2026-10-06"
  },
  {
   "id": "SL3",
   "url": "https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL3",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "The 3.8% previous best was the retrieval-augmented baseline running on Claude 3 Opus (3.79% of the full test set); the same baseline on GPT-4 Turbo resolved 1.31%.",
   "quote": "RAG w/ GPT-4 Turbo 1.31 … w/ Claude 3 Opus 3.79",
   "removal_reason": "",
   "source": "John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793)",
   "source_url": "https://arxiv.org/abs/2405.15793",
   "locator": "Table 1, SWE-bench column, % Resolved (same values in v1, v2 and v3)",
   "date_read": "2026-10-06"
  },
  {
   "id": "SL4",
   "url": "https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL4",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "With the model held at GPT-4 Turbo on the 300-task SWE-bench Lite split, SWE-agent resolved 18.00% of tasks against 11.00% for an agent working through the plain Linux shell, a 64% relative increase.",
   "quote": "An LM-friendly ACI's value is confirmed by SWE-agent's 64% relative increase compared to Shell-only, both with GPT-4 Turbo.",
   "removal_reason": "",
   "source": "John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793)",
   "source_url": "https://arxiv.org/abs/2405.15793",
   "locator": "§5 Results, first paragraph · Table 1, SWE-bench Lite column (Shell-only agent w/ GPT-4 Turbo 11.00; SWE-agent w/ GPT-4 Turbo 18.00)",
   "date_read": "2026-10-06"
  },
  {
   "id": "SL5",
   "url": "https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL5",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "SWE-agent's interface changed what the agent could see and do: commands for searching, viewing and editing files, guardrails against common mistakes, and concise feedback on each command's effects.",
   "quote": "a small set of simple actions for viewing, searching through and editing files … guardrails to prevent common mistakes … concise feedback about a command's effects",
   "removal_reason": "",
   "source": "John Yang, Carlos E. Jimenez et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024 (arXiv:2405.15793)",
   "source_url": "https://arxiv.org/abs/2405.15793",
   "locator": "§1 Introduction · §3, opening paragraph (search/navigation, file viewer, file editor, context management)",
   "date_read": "2026-10-06"
  },
  {
   "id": "SL6",
   "url": "https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL6",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "In SkillsBench, curated skills raised the average pass rate from 33.9% to 50.5%, a gain of 16.6 percentage points across 87 tasks and 18 model-harness configurations; versions 1 to 3 of the paper reported 16.2 points on 84 evaluated tasks (of 86).",
   "quote": "Curated Skills raise the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain)",
   "removal_reason": "",
   "source": "Xiangyi Li et al., SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv:2602.12670",
   "source_url": "https://arxiv.org/abs/2602.12670",
   "locator": "Abstract (v4, 14 June 2026) · v1–v3 abstract: \"Curated Skills raise average pass rate by 16.2 percentage points\" · v1 §3.5 Metrics, \"Pass Rate\": \"a fixed denominator of 84 (the number of evaluated tasks)\"",
   "date_read": "2026-10-11"
  },
  {
   "id": "SL7",
   "url": "https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL7",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Skills that a model wrote for itself gave no average benefit. In versions 1 to 3 they averaged 1.3 points below the no-skills baseline; in version 4 they landed below it on all three configurations tested, by 8.1 to 11.5 points.",
   "quote": "Self-generated Skills land below the no-Skills baseline on all three configurations",
   "removal_reason": "",
   "source": "Xiangyi Li et al., SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv:2602.12670",
   "source_url": "https://arxiv.org/abs/2602.12670",
   "locator": "v4 §5.1.1, paragraph \"Self-generated Skills do not substitute for curated ones\" (−8.1, −11.3 and −11.5 pp) · v1 §4.1.1, Finding 3 (–1.3pp on average)",
   "date_read": "2026-10-06"
  },
  {
   "id": "SL8",
   "url": "https://durabilitycurve.com/claims/the-seven-layer-agent-audit/#c-SL8",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Language models use long contexts unevenly: performance is often highest when the relevant information is at the beginning or end of the input, and degrades significantly when it sits in the middle.",
   "quote": "often highest when relevant information occurs at the beginning or end … significantly degrades when models must access relevant information in the middle of long contexts",
   "removal_reason": "",
   "source": "Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang, Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics 12, 2024, pp. 157–173",
   "source_url": "https://aclanthology.org/2024.tacl-1.9/",
   "locator": "Abstract",
   "date_read": "2026-10-06"
  }
 ]
}