{
 "ledger": {
  "title": "Claim Ledger: The Guardrail Your Agent Can Reach",
  "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/",
  "essay": {
   "title": "The Guardrail Your Agent Can Reach",
   "url": "https://durabilitycurve.com/blog/the-guardrail-your-agent-can-reach/",
   "published": "2026-08-08"
  },
  "last_checked": "2026-08-07",
  "format": "current",
  "counts": {
   "claims": 20,
   "by_status": {
    "VERIFIED": 20
   },
   "verified_at_primary": 20,
   "removed": 2
  },
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "attribution": "The Durability Curve (Harry Floyd), https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/"
 },
 "rows": [
  {
   "id": "L1",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L1",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Rule-file gains are largely content-independent — random, shuffled and mismatched-domain files all match curated ones",
   "quote": "\"Performance gains are largely content-independent: random, shuffled, mismatched-domain, and unconverted-format rule files all match curated rules, pointing to a context priming mechanism.\"",
   "removal_reason": "",
   "source": "Zhang et al., Guardrails Beat Guidance",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "Abstract, v2",
   "date_read": "2026-08-07"
  },
  {
   "id": "L2",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L2",
   "status": "STRUCK",
   "status_label": "Struck",
   "removed": true,
   "claim": "Random rules match expert-curated ones",
   "quote": "",
   "removal_reason": "\"Same effect size\" reads as two equal measured effects; the paper's result is a NULL (Cochran's Q = 4.70, p = 0.697) and the +13.8pp does not itself reach significance against the no-rule baseline (McNemar p = 0.077).",
   "source": "",
   "source_url": "",
   "locator": "",
   "date_read": "2026-08-07"
  },
  {
   "id": "L3",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L3",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Instructions in context files ARE well followed — the strongest evidence against this piece",
   "quote": "\"while instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful\"",
   "removal_reason": "",
   "source": "Gloaguen et al.",
   "source_url": "https://arxiv.org/abs/2602.11988",
   "locator": "Abstract, v2 (23 Jun 2026)",
   "date_read": "2026-08-07"
  },
  {
   "id": "L4",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L4",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Context files do not generally improve task success, and cost over 20% more",
   "quote": "\"Surprisingly, we find that providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average. This observation holds across different LLMs, coding agents, and for both LLM-generated and developer-committed context files.\"",
   "removal_reason": "",
   "source": "Gloaguen et al.",
   "source_url": "https://arxiv.org/abs/2602.11988",
   "locator": "Abstract, v2",
   "date_read": "2026-08-07"
  },
  {
   "id": "L5",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L5",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Gloaguen's own conclusion is conditional, not dismissive",
   "quote": "\"We conclude that while context files are useful for specifying non-standard coding practices, any attempts to improve performance should be rigorously evaluated before deployment.\"",
   "removal_reason": "",
   "source": "Gloaguen et al.",
   "source_url": "https://arxiv.org/abs/2602.11988",
   "locator": "Abstract, v2",
   "date_read": "2026-08-07"
  },
  {
   "id": "L6",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L6",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Scale of the Zhang study, and of the practice it measures",
   "quote": "\"over 5,000 agent runs of Claude Code with Claude Opus 4.6 on SWE-bench Verified\"; 679 rule files containing 25,532 total rules scraped from GitHub",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "Abstract + method, v2",
   "date_read": "2026-08-07"
  },
  {
   "id": "L7",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L7",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Of the 18-rule curated set: 3 shaping (removal hurts), 4 distorting (removal helps), 11 inert",
   "quote": "\"Starting from the full 18-rule curated set (65.7% pass rate), we remove each rule individually and measure the change (Figure 1b). Using a [ABS-DELTA]>5 pp threshold, of 18 rules, 3 are shaping (removal hurts), 4 are distorting (removal helps), and 11 are inert.\" — [ABS-DELTA] renders the paper's absolute-value notation around delta; the pipe characters are substituted because they break a markdown table cell. Nothing else is altered.",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "§6.4, Experiment 4a",
   "date_read": "2026-08-07"
  },
  {
   "id": "L8",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L8",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Zhang et al. say an already capable coding agent does not need to be told what to do, but benefits from being told what not to do",
   "quote": "\"The base agent already possesses strong coding capabilities; it does not need to be told what to do, but it benefits from being told what not to do.\"",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "§6.4, PBRS reading",
   "date_read": "2026-08-07"
  },
  {
   "id": "L9",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L9",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "14 of 18 rules broke at least 2 previously-solved tasks; worst offenders 4 of 17",
   "quote": "\"As shown in Figure 4a, 14 of 18 rules break at least 2 previously-solved tasks; the worst offenders (\"understand full context,\" \"keep functions concise,\" \"do not install dependencies\") each break 4/17 (24%). Only 4 rules are safe (≤1 task broken).\"",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "§6.4, Experiment 4b",
   "date_read": "2026-08-07"
  },
  {
   "id": "L10",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L10",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "The authors caveat L9 on run-to-run variance",
   "quote": "\"We treat this evidence as suggestive rather than conclusive: with a baseline retest reliability of 88.2%, each task has an ≈11.8% chance of flipping under run-to-run variance alone, so any single rule's break count cannot be read as fully causal.\"",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "§6.4, Experiment 4b",
   "date_read": "2026-08-07"
  },
  {
   "id": "L11",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L11",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Zhang et al. ran a paired comparison on 58 discriminative SWE-bench Verified tasks, and three finer analyses on 35 discriminative tasks with 18 curated rules",
   "quote": "\"We perform three fine-grained analyses on 35 discriminative tasks using a curated set of 18 rules\"; and the study runs \"controlled experiments with a state-of-the-art coding agent on SWE-bench Verified… A paired within-subject design on 58 discriminative tasks\"",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "§6.4 opening; §1",
   "date_read": "2026-08-07"
  },
  {
   "id": "L13",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L13",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "The authors name their own selection bias",
   "quote": "\"Selection bias. By selecting discriminative tasks (30–70% baseline pass rate), we enrich for tasks where rules can produce a measurable effect.\"",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "§ Limitations",
   "date_read": "2026-08-07"
  },
  {
   "id": "L14",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L14",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Individual harm does not compound when rules are stacked",
   "quote": "\"Finding 5: Individual harm does not compound in ensemble. The robust comparison is between any single rule and the ensemble. If individual rules were genuinely as distortive as 4b suggests, stacking 18 of them should compound…\"",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "§6.4, Finding 5",
   "date_read": "2026-08-07"
  },
  {
   "id": "L15",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L15",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "What rules are, in the authors' framing",
   "quote": "Rules \"are persistent (loaded every session), authored by third parties (not the model developer), and intended to shape multi-step tool-using behavior rather than single-turn generation.\"",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "§1",
   "date_read": "2026-08-07"
  },
  {
   "id": "L16",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L16",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Anthropic's own position: an instruction is the wrong tool for a must-not-happen",
   "quote": "\"When there's something that absolutely must not happen, an instruction is the wrong tool. Claude will follow the instruction most of the time, but when under pressure, in a long session or an ambiguous situation, or due to a prompt injection in a file accessed as part of the task, the model can fail to follow a prompted rule. A real guardrail needs to be deterministic, and the enforcement methods are hooks and permissions.\"",
   "removal_reason": "",
   "source": "Michael Segner, Anthropic",
   "source_url": "https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more",
   "locator": "\"Steering Claude Code\", 18 June 2026",
   "date_read": "2026-08-07"
  },
  {
   "id": "L17",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L17",
   "status": "STRUCK",
   "status_label": "Struck",
   "removed": true,
   "claim": "Prompt-only constraint compliance, measured",
   "quote": "",
   "removal_reason": "Benchmark is SWE-bench Lite.",
   "source": "",
   "source_url": "",
   "locator": "",
   "date_read": "2026-08-07"
  },
  {
   "id": "L18",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L18",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Zhang et al.'s affiliations — the provenance the piece discloses",
   "quote": "Author footnotes: \"¹AWS Generative AI Innovation Center ²HSBC Holdings Plc., HSBC Technology Center, China\"",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "Title page, author footnote",
   "date_read": "2026-08-07"
  },
  {
   "id": "L19",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L19",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "The paper's own headline finding is that polarity matters — i.e. content matters at the level where rules are written",
   "quote": "\"in our data every individually beneficial rule is a negative constraint (\"do not refactor unrelated code\"), while every individually harmful one is a positive directive (\"follow code style\")\"; and the principle: \"constrain what agents must not do, rather than prescribing what they should\"",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "Abstract, finding (i)",
   "date_read": "2026-08-07"
  },
  {
   "id": "L22",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L22",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "NIST's reference monitor sets three requirements on an enforcement mechanism, and tamperproof is one of them",
   "quote": "\"A set of design requirements on a reference validation mechanism that, as a key component of an operating system, enforces an access control policy over all subjects and objects.\" Three attributes: \"Always invoked (complete mediation)\", \"Tamperproof\", and \"Small enough to be subject to analysis and tests, with verifiable completeness\"",
   "removal_reason": "",
   "source": "NIST, SP 800-53 Rev. 5",
   "source_url": "https://csrc.nist.gov/glossary/term/reference_monitor",
   "locator": "Glossary entry",
   "date_read": "2026-08-07"
  },
  {
   "id": "L21",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L21",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Anthropic distinguishes deterministic hooks from settings that cannot be overridden locally — the vendor drawing this article's own distinction",
   "quote": "\"Managed settings go further: they are admin-deployed, cannot be overridden by a user's local config, and are the only way to enforce a deterministic, organization-wide guardrail.\" And separately: \"A PreToolUse hook can inspect a call and exit with code 2 to block it.\"",
   "removal_reason": "",
   "source": "Michael Segner, Anthropic",
   "source_url": "https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more",
   "locator": "\"Steering Claude Code\", 18 June 2026",
   "date_read": "2026-08-07"
  },
  {
   "id": "L20",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L20",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Rule count does not accumulate damage",
   "quote": "\"Individual rules often appear harmful in isolation yet do not visibly accumulate damage in ensemble: pass rates remain stable across rule counts from 0 to 50.\"",
   "removal_reason": "",
   "source": "Zhang et al.",
   "source_url": "https://arxiv.org/abs/2604.11088",
   "locator": "Abstract, finding (iii)",
   "date_read": "2026-08-07"
  },
  {
   "id": "L12",
   "url": "https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L12",
   "status": "VERIFIED",
   "status_label": "Verified",
   "removed": false,
   "claim": "Our own Meadows piece ranks \"system prompts as constraints\" in the same cell as \"tool permissions, action gates\"",
   "quote": "\"Rules of the system (incentives, punishments, constraints)\" / \"Tool permissions, action gates, system prompts as constraints\"",
   "removal_reason": "",
   "source": "The Leverage Hierarchy of Agent Engineering (ours)",
   "source_url": "https://harryfloyd.substack.com/p/the-leverage-hierarchy-of-agent-engineering",
   "locator": "The 12-rank table, rank 5",
   "date_read": "2026-08-07"
  }
 ]
}