Claim ledger
The Guardrail Your Agent Can Reach
Most guardrails end up with an escape hatch. Check whether the thing you are constraining can reach yours.
- Claims
- 20
- Verified at the primary source
- 20
- Removed before publication
- 2
- Last checked
20 claims: 20 verified. Plus 2 removed before publication, listed at the end.
Download this ledger: CSV · JSON · Markdown · CC BY 4.0 · Query it from Claude or ChatGPT
What the essay claims
Before: If I write the rule clearly enough, the agent will follow it. When it misbehaves, my instructions were not good enough yet. This is the near-universal working model, and it is why practitioners iterate on prose for months.
After: Instructions are followed as a tendency, never as a floor. A layer made of words cannot refuse, and no amount of rewriting will give it a worst case. The only things that bound behaviour live in the execution layer.
The claim ladder
The rung that stops the essay overclaiming. Every item below is something a reader might think the piece establishes, and does not:
| # | Claim | Rung | Whose behaviour it measures (Checker / Subject) | Evidence | Scope carried in the same sentence |
|---|---|---|---|---|---|
| 1 | Rule files work, and work by priming rather than by instruction-specific compliance | Load-bearing | Zhang et al. / Claude Code on Claude Opus 4.6 | Abstract + §6.1: random, shuffled, mismatched-domain files all match curated | One model, SWE-bench Verified subset |
| 2 | Inside one curated 18-rule set, 11 were inert, 4 harmful, 3 helpful | Supporting | Zhang et al. / the same agent, 35 discriminative tasks | §6.4, Experiment 4 | One ablation, 35 Python bug-fix tasks |
| 3 | Most rules broke something the agent already did reliably | Supporting | Zhang et al. / the same agent, 17 previously-solved tasks | §6.4, Experiment 4b, 14/18 | Mandatory companion: 88.2% retest reliability, authors call it suggestive not conclusive |
| 4 | Instructions are nonetheless well followed | Counter-evidence, carried | Gloaguen et al. / coding agents on real repositories | Abstract | Stated as the strongest case against |
| 5 | Following is not a floor; a tendency has no worst case | The thesis | Nobody. No agent was measured. Reasoning from 1 + 4 | Argument | Marked in the body as reasoning, not measurement |
| 6 | Therefore the ladder's rungs sort into describing and enforcing | The frame | Nobody. No agent was measured. | Argument | Offered as useful, never as proven |
It does not reach the reader's own rule file. Every measured claim is about one model on Python bug-fix tasks. Nothing here licenses a statement about what fraction of your rules is inert, and the body never makes one.
It does not reach non-coding agents. Both sources measure coding agents on code repositories. Whether priming behaves the same for a research or writing agent is unmeasured and the piece says so.
It does not reach "enforcement works better than description." No source compares the two layers head to head. The claim is that they are different kinds of thing, not that one outperforms the other. There is no measurement anywhere in this piece of an enforcement layer's effect.
It does not reach causation for claim 3. At 88.2% retest reliability the break counts cannot be read as fully causal, and the authors say so first.
It does not reach the frame's own usefulness. Whether the describing/enforcing boundary actually helps anyone is untested. It is the thing the next five pieces will test, and the failure-diagnosis section pre-registers what would count as it failing.
The evidence, row by row
Each row is a claim the essay makes, then the evidence recorded for it and the source it was checked against. For verified rows that is usually the exact words, where they sit and the day they were read. The label says how far it was checked.
What the labels mean
- Verified
- Checked against the primary source itself by a checker who did not draft the piece, usually with the exact words, where they sit and the date read.
- Executed
- A run or observation done for the piece: code, a query, a count, or a check made live in an app. The claim is what it returned. Run records are not published.
- Checked
- Checked against its source by a separate checker in the older ledger format, which recorded no quote, locator or date.
- Reported
- Not verified word for word against a primary source: carried from a secondary source, from one that could not be opened in full, or supported only in part. The essay words it accordingly.
- Struck
- Drafted, checked and removed before publication. Kept here, crossed out, with the reason.
- Excluded
- Considered and deliberately left out of the piece. Kept here with the reason.
Sources (5)
- Zhang et al. 13 rows
- Gloaguen et al. 3 rows
- Michael Segner 2 rows
- NIST 1 row
- The Leverage Hierarchy of Agent Engineering 1 row
-
Rule-file gains are largely content-independent — random, shuffled and mismatched-domain files all match curated ones
"Performance gains are largely content-independent: random, shuffled, mismatched-domain, and unconverted-format rule files all match curated rules, pointing to a context priming mechanism."
Zhang et al., Guardrails Beat Guidance · Abstract, v2 · arxiv.org
LinkReport an errorCite
Zhang et al., Guardrails Beat Guidance, Abstract, v2, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L1, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L1 (source read 2026-08-07). -
Instructions in context files ARE well followed — the strongest evidence against this piece
"while instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful"
Gloaguen et al. · Abstract, v2 (23 Jun 2026) · arxiv.org
LinkReport an errorCite
Gloaguen et al., Abstract, v2 (23 Jun 2026), https://arxiv.org/abs/2602.11988. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L3, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L3 (source read 2026-08-07). -
Context files do not generally improve task success, and cost over 20% more
"Surprisingly, we find that providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average. This observation holds across different LLMs, coding agents, and for both LLM-generated and developer-committed context files."
Gloaguen et al. · Abstract, v2 · arxiv.org
LinkReport an errorCite
Gloaguen et al., Abstract, v2, https://arxiv.org/abs/2602.11988. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L4, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L4 (source read 2026-08-07). -
Gloaguen's own conclusion is conditional, not dismissive
"We conclude that while context files are useful for specifying non-standard coding practices, any attempts to improve performance should be rigorously evaluated before deployment."
Gloaguen et al. · Abstract, v2 · arxiv.org
LinkReport an errorCite
Gloaguen et al., Abstract, v2, https://arxiv.org/abs/2602.11988. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L5, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L5 (source read 2026-08-07). -
Scale of the Zhang study, and of the practice it measures
"over 5,000 agent runs of Claude Code with Claude Opus 4.6 on SWE-bench Verified"; 679 rule files containing 25,532 total rules scraped from GitHub
Zhang et al. · Abstract + method, v2 · arxiv.org
LinkReport an errorCite
Zhang et al., Abstract + method, v2, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L6, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L6 (source read 2026-08-07). -
Of the 18-rule curated set: 3 shaping (removal hurts), 4 distorting (removal helps), 11 inert
"Starting from the full 18-rule curated set (65.7% pass rate), we remove each rule individually and measure the change (Figure 1b). Using a [ABS-DELTA]>5 pp threshold, of 18 rules, 3 are shaping (removal hurts), 4 are distorting (removal helps), and 11 are inert." — [ABS-DELTA] renders the paper's absolute-value notation around delta; the pipe characters are substituted because they break a markdown table cell. Nothing else is altered.
Zhang et al. · §6.4, Experiment 4a · arxiv.org
LinkReport an errorCite
Zhang et al., §6.4, Experiment 4a, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L7, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L7 (source read 2026-08-07). -
Zhang et al. say an already capable coding agent does not need to be told what to do, but benefits from being told what not to do
"The base agent already possesses strong coding capabilities; it does not need to be told what to do, but it benefits from being told what not to do."
Zhang et al. · §6.4, PBRS reading · arxiv.org
LinkReport an errorCite
Zhang et al., §6.4, PBRS reading, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L8, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L8 (source read 2026-08-07). -
14 of 18 rules broke at least 2 previously-solved tasks; worst offenders 4 of 17
"As shown in Figure 4a, 14 of 18 rules break at least 2 previously-solved tasks; the worst offenders ("understand full context," "keep functions concise," "do not install dependencies") each break 4/17 (24%). Only 4 rules are safe (≤1 task broken)."
Zhang et al. · §6.4, Experiment 4b · arxiv.org
LinkReport an errorCite
Zhang et al., §6.4, Experiment 4b, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L9, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L9 (source read 2026-08-07). -
The authors caveat L9 on run-to-run variance
"We treat this evidence as suggestive rather than conclusive: with a baseline retest reliability of 88.2%, each task has an ≈11.8% chance of flipping under run-to-run variance alone, so any single rule's break count cannot be read as fully causal."
Zhang et al. · §6.4, Experiment 4b · arxiv.org
LinkReport an errorCite
Zhang et al., §6.4, Experiment 4b, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L10, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L10 (source read 2026-08-07). -
Zhang et al. ran a paired comparison on 58 discriminative SWE-bench Verified tasks, and three finer analyses on 35 discriminative tasks with 18 curated rules
"We perform three fine-grained analyses on 35 discriminative tasks using a curated set of 18 rules"; and the study runs "controlled experiments with a state-of-the-art coding agent on SWE-bench Verified… A paired within-subject design on 58 discriminative tasks"
Zhang et al. · §6.4 opening; §1 · arxiv.org
LinkReport an errorCite
Zhang et al., §6.4 opening; §1, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L11, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L11 (source read 2026-08-07). -
The authors name their own selection bias
"Selection bias. By selecting discriminative tasks (30–70% baseline pass rate), we enrich for tasks where rules can produce a measurable effect."
Zhang et al. · § Limitations · arxiv.org
LinkReport an errorCite
Zhang et al., § Limitations, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L13, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L13 (source read 2026-08-07). -
Individual harm does not compound when rules are stacked
"Finding 5: Individual harm does not compound in ensemble. The robust comparison is between any single rule and the ensemble. If individual rules were genuinely as distortive as 4b suggests, stacking 18 of them should compound…"
Zhang et al. · §6.4, Finding 5 · arxiv.org
LinkReport an errorCite
Zhang et al., §6.4, Finding 5, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L14, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L14 (source read 2026-08-07). -
What rules are, in the authors' framing
Rules "are persistent (loaded every session), authored by third parties (not the model developer), and intended to shape multi-step tool-using behavior rather than single-turn generation."
Zhang et al. · §1 · arxiv.org
LinkReport an errorCite
Zhang et al., §1, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L15, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L15 (source read 2026-08-07). -
Anthropic's own position: an instruction is the wrong tool for a must-not-happen
"When there's something that absolutely must not happen, an instruction is the wrong tool. Claude will follow the instruction most of the time, but when under pressure, in a long session or an ambiguous situation, or due to a prompt injection in a file accessed as part of the task, the model can fail to follow a prompted rule. A real guardrail needs to be deterministic, and the enforcement methods are hooks and permissions."
Michael Segner, Anthropic · "Steering Claude Code", 18 June 2026 · claude.com
LinkReport an errorCite
Michael Segner, Anthropic, "Steering Claude Code", 18 June 2026, https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L16, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L16 (source read 2026-08-07). -
Zhang et al.'s affiliations — the provenance the piece discloses
Author footnotes: "¹AWS Generative AI Innovation Center ²HSBC Holdings Plc., HSBC Technology Center, China"
Zhang et al. · Title page, author footnote · arxiv.org
LinkReport an errorCite
Zhang et al., Title page, author footnote, https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L18, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L18 (source read 2026-08-07). -
The paper's own headline finding is that polarity matters — i.e. content matters at the level where rules are written
"in our data every individually beneficial rule is a negative constraint ("do not refactor unrelated code"), while every individually harmful one is a positive directive ("follow code style")"; and the principle: "constrain what agents must not do, rather than prescribing what they should"
Zhang et al. · Abstract, finding (i) · arxiv.org
LinkReport an errorCite
Zhang et al., Abstract, finding (i), https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L19, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L19 (source read 2026-08-07). -
NIST's reference monitor sets three requirements on an enforcement mechanism, and tamperproof is one of them
"A set of design requirements on a reference validation mechanism that, as a key component of an operating system, enforces an access control policy over all subjects and objects." Three attributes: "Always invoked (complete mediation)", "Tamperproof", and "Small enough to be subject to analysis and tests, with verifiable completeness"
NIST, SP 800-53 Rev. 5 · Glossary entry · csrc.nist.gov
LinkReport an errorCite
NIST, SP 800-53 Rev. 5, Glossary entry, https://csrc.nist.gov/glossary/term/reference_monitor. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L22, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L22 (source read 2026-08-07). -
Anthropic distinguishes deterministic hooks from settings that cannot be overridden locally — the vendor drawing this article's own distinction
"Managed settings go further: they are admin-deployed, cannot be overridden by a user's local config, and are the only way to enforce a deterministic, organization-wide guardrail." And separately: "A PreToolUse hook can inspect a call and exit with code 2 to block it."
Michael Segner, Anthropic · "Steering Claude Code", 18 June 2026 · claude.com
LinkReport an errorCite
Michael Segner, Anthropic, "Steering Claude Code", 18 June 2026, https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L21, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L21 (source read 2026-08-07). -
Rule count does not accumulate damage
"Individual rules often appear harmful in isolation yet do not visibly accumulate damage in ensemble: pass rates remain stable across rule counts from 0 to 50."
Zhang et al. · Abstract, finding (iii) · arxiv.org
LinkReport an errorCite
Zhang et al., Abstract, finding (iii), https://arxiv.org/abs/2604.11088. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L20, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L20 (source read 2026-08-07). -
Our own Meadows piece ranks "system prompts as constraints" in the same cell as "tool permissions, action gates"
"Rules of the system (incentives, punishments, constraints)" / "Tool permissions, action gates, system prompts as constraints"
The Leverage Hierarchy of Agent Engineering (ours) · The 12-rank table, rank 5 · harryfloyd.substack.com
LinkReport an errorCite
The Leverage Hierarchy of Agent Engineering (ours), The 12-rank table, rank 5, https://harryfloyd.substack.com/p/the-leverage-hierarchy-of-agent-engineering. Checked in: Claim Ledger, "The Guardrail Your Agent Can Reach", The Durability Curve, row L12, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/#c-L12 (source read 2026-08-07).
No rows match. Clear the search or pick another label.
Removed before publication
Claims that were drafted, checked and taken out, or considered and left out. The essay does not make them; they stay here with the reason.
- L2 Struck
Random rules match expert-curated onesWhy: "Same effect size" reads as two equal measured effects; the paper's result is a NULL (Cochran's Q = 4.70, p = 0.697) and the +13.8pp does not itself reach significance against the no-rule baseline (McNemar p = 0.077).
- L17 Struck
Prompt-only constraint compliance, measuredWhy: Benchmark is SWE-bench Lite.
Cite and reuse
- A single claim
- Use the row's Cite button. It names the original source first, then this row, which carries its own link.
- This ledger
Floyd, Harry (2026). Claim Ledger: The Guardrail Your Agent Can Reach. The Durability Curve. https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/- The essay
Floyd, Harry (2026). The Guardrail Your Agent Can Reach. The Durability Curve. https://durabilitycurve.com/blog/the-guardrail-your-agent-can-reach/
The ledger data is licensed CC BY 4.0: reuse it, quote it or train on it, with credit to The Durability Curve and a link. Read the licence. Machine copies: JSON, CSV, Markdown.