Claim ledger
The Guardrail Your Agent Can Reach
Most guardrails end up with an escape hatch. Check whether the thing you are constraining can reach yours.
20 claims · 20 verified to primary · 2 struck or excluded
What the essay claims
Before: If I write the rule clearly enough, the agent will follow it. When it misbehaves, my instructions were not good enough yet. This is the near-universal working model, and it is why practitioners iterate on prose for months.
After: Instructions are followed as a tendency, never as a floor. A layer made of words cannot refuse, and no amount of rewriting will give it a worst case. The only things that bound behaviour live in the execution layer.
The claim ladder
The seam gets a visible marker in the body — the reader is told where measurement stops and reasoning starts.
| # | Claim | Rung | Whose behaviour it measures (Checker / Subject) | Evidence | Scope carried in the same sentence |
|---|---|---|---|---|---|
| 1 | Rule files work, and work by priming rather than by instruction-specific compliance | Load-bearing | Zhang et al. / Claude Code on Claude Opus 4.6 | Abstract + §6.1: random, shuffled, mismatched-domain files all match curated | One model, SWE-bench Verified subset |
| 2 | Inside one curated 18-rule set, 11 were inert, 4 harmful, 3 helpful | Supporting | Zhang et al. / the same agent, 35 discriminative tasks | §6.4, Experiment 4 | One ablation, 35 Python bug-fix tasks |
| 3 | Most rules broke something the agent already did reliably | Supporting | Zhang et al. / the same agent, 17 previously-solved tasks | §6.4, Experiment 4b, 14/18 | Mandatory companion: 88.2% retest reliability, authors call it suggestive not conclusive |
| 4 | Instructions are nonetheless well followed | Counter-evidence, carried | Gloaguen et al. / coding agents on real repositories | Abstract | Stated as the strongest case against |
| 5 | Following is not a floor; a tendency has no worst case | The thesis | Nobody. No agent was measured. Reasoning from 1 + 4 | Argument | Marked in the body as reasoning, not measurement |
| 6 | Therefore the ladder's rungs sort into describing and enforcing | The frame | Nobody. No agent was measured. | Argument | Offered as useful, never as proven |
The rung that stops the essay overclaiming. Every item below is something a reader might think the piece establishes, and does not:
It does not reach the reader's own rule file. Every measured claim is about one model on Python bug-fix tasks. Nothing here licenses a statement about what fraction of your rules is inert, and the body never makes one.
It does not reach non-coding agents. Both sources measure coding agents on code repositories. Whether priming behaves the same for a research or writing agent is unmeasured and the piece says so.
It does not reach "enforcement works better than description." No source compares the two layers head to head. The claim is that they are different kinds of thing, not that one outperforms the other. There is no measurement anywhere in this piece of an enforcement layer's effect.
It does not reach causation for claim 3. At 88.2% retest reliability the break counts cannot be read as fully causal, and the authors say so first.
It does not reach the frame's own usefulness. Whether the describing/enforcing boundary actually helps anyone is untested. It is the thing the next five pieces will test, and the failure-diagnosis section pre-registers what would count as it failing.
The evidence, row by row
Each row is a claim the essay makes, the words in the source that support it, where in the source they are, the day the source was read, and the status the check assigned. Verified means the primary was opened and read; Executed means we ran it ourselves; Reported means carried from a source we could not open in full.
-
Rule-file gains are largely content-independent — random, shuffled and mismatched-domain files all match curated ones
"Performance gains are largely content-independent: random, shuffled, mismatched-domain, and unconverted-format rule files all match curated rules, pointing to a context priming mechanism."
Zhang et al., Guardrails Beat Guidance · Abstract, v2 · arxiv.org
-
Instructions in context files ARE well followed — the strongest evidence against this piece
"while instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful"
Gloaguen et al. · Abstract, v2 (23 Jun 2026) · arxiv.org
-
Context files do not generally improve task success, and cost over 20% more
"Surprisingly, we find that providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average. This observation holds across different LLMs, coding agents, and for both LLM-generated and developer-committed context files."
Gloaguen et al. · Abstract, v2 · arxiv.org
-
Gloaguen's own conclusion is conditional, not dismissive
"We conclude that while context files are useful for specifying non-standard coding practices, any attempts to improve performance should be rigorously evaluated before deployment."
Gloaguen et al. · Abstract, v2 · arxiv.org
-
Scale of the Zhang study, and of the practice it measures
"over 5,000 agent runs of Claude Code with Claude Opus 4.6 on SWE-bench Verified"; 679 rule files containing 25,532 total rules scraped from GitHub
Zhang et al. · Abstract + method, v2 · arxiv.org
-
Of the 18-rule curated set: 3 shaping (removal hurts), 4 distorting (removal helps), 11 inert
"Starting from the full 18-rule curated set (65.7% pass rate), we remove each rule individually and measure the change (Figure 1b). Using a [ABS-DELTA]>5 pp threshold, of 18 rules, 3 are shaping (removal hurts), 4 are distorting (removal helps), and 11 are inert." — [ABS-DELTA] renders the paper's absolute-value notation around delta; the pipe characters are substituted because they break a markdown table cell. Nothing else is altered.
Zhang et al. · §6.4, Experiment 4a · arxiv.org
-
The authors' own one-line mechanism
"The base agent already possesses strong coding capabilities; it does not need to be told what to do, but it benefits from being told what not to do."
Zhang et al. · §6.4, PBRS reading · arxiv.org
-
14 of 18 rules broke at least 2 previously-solved tasks; worst offenders 4 of 17
"As shown in Figure 4a, 14 of 18 rules break at least 2 previously-solved tasks; the worst offenders ("understand full context," "keep functions concise," "do not install dependencies") each break 4/17 (24%). Only 4 rules are safe (≤1 task broken)."
Zhang et al. · §6.4, Experiment 4b · arxiv.org
-
The authors caveat L9 on run-to-run variance
"We treat this evidence as suggestive rather than conclusive: with a baseline retest reliability of 88.2%, each task has an ≈11.8% chance of flipping under run-to-run variance alone, so any single rule's break count cannot be read as fully causal."
Zhang et al. · §6.4, Experiment 4b · arxiv.org
-
Study scope
"We perform three fine-grained analyses on 35 discriminative tasks using a curated set of 18 rules"; and the study runs "controlled experiments with a state-of-the-art coding agent on SWE-bench Verified… A paired within-subject design on 58 discriminative tasks"
Zhang et al. · §6.4 opening; §1 · arxiv.org
-
The authors name their own selection bias
"Selection bias. By selecting discriminative tasks (30–70% baseline pass rate), we enrich for tasks where rules can produce a measurable effect."
Zhang et al. · § Limitations · arxiv.org
-
Individual harm does not compound when rules are stacked
"Finding 5: Individual harm does not compound in ensemble. The robust comparison is between any single rule and the ensemble. If individual rules were genuinely as distortive as 4b suggests, stacking 18 of them should compound…"
Zhang et al. · §6.4, Finding 5 · arxiv.org
-
What rules are, in the authors' framing
Rules "are persistent (loaded every session), authored by third parties (not the model developer), and intended to shape multi-step tool-using behavior rather than single-turn generation."
Zhang et al. · §1 · arxiv.org
-
Anthropic's own position: an instruction is the wrong tool for a must-not-happen
"When there's something that absolutely must not happen, an instruction is the wrong tool. Claude will follow the instruction most of the time, but when under pressure, in a long session or an ambiguous situation, or due to a prompt injection in a file accessed as part of the task, the model can fail to follow a prompted rule. A real guardrail needs to be deterministic, and the enforcement methods are hooks and permissions."
Michael Segner, Anthropic · "Steering Claude Code", 18 June 2026 · claude.com
-
Zhang et al.'s affiliations — the provenance the piece discloses
Author footnotes: "¹AWS Generative AI Innovation Center ²HSBC Holdings Plc., HSBC Technology Center, China"
Zhang et al. · Title page, author footnote · arxiv.org
-
The paper's own headline finding is that polarity matters — i.e. content matters at the level where rules are written
"in our data every individually beneficial rule is a negative constraint ("do not refactor unrelated code"), while every individually harmful one is a positive directive ("follow code style")"; and the principle: "constrain what agents must not do, rather than prescribing what they should"
Zhang et al. · Abstract, finding (i) · arxiv.org
-
NIST's reference monitor sets three requirements on an enforcement mechanism, and tamperproof is one of them
"A set of design requirements on a reference validation mechanism that, as a key component of an operating system, enforces an access control policy over all subjects and objects." Three attributes: "Always invoked (complete mediation)", "Tamperproof", and "Small enough to be subject to analysis and tests, with verifiable completeness"
NIST, SP 800-53 Rev. 5 · Glossary entry · csrc.nist.gov
-
Anthropic distinguishes deterministic hooks from settings that cannot be overridden locally — the vendor drawing this article's own distinction
"Managed settings go further: they are admin-deployed, cannot be overridden by a user's local config, and are the only way to enforce a deterministic, organization-wide guardrail." And separately: "A PreToolUse hook can inspect a call and exit with code 2 to block it."
Michael Segner, Anthropic · "Steering Claude Code", 18 June 2026 · claude.com
-
Rule count does not accumulate damage
"Individual rules often appear harmful in isolation yet do not visibly accumulate damage in ensemble: pass rates remain stable across rule counts from 0 to 50."
Zhang et al. · Abstract, finding (iii) · arxiv.org
-
Our own Meadows piece ranks "system prompts as constraints" in the same cell as "tool permissions, action gates"
"Rules of the system (incentives, punishments, constraints)" / "Tool permissions, action gates, system prompts as constraints"
The Leverage Hierarchy of Agent Engineering (ours) · The 12-rank table, rank 5 · harryfloyd.substack.com
Struck or excluded before publication
Claims that were drafted, checked, and removed. They are not claims the essay makes; they are the ones it stopped making, with the reason.
-
Random rules match expert-curated ones"Same effect size" reads as two equal measured effects; the paper's result is a NULL (Cochran's Q = 4.70, p = 0.697) and the +13.8pp does not itself reach significance against the no-rule baseline (McNemar p = 0.077).
-
Prompt-only constraint compliance, measuredBenchmark is SWE-bench Lite.
Cite this
- A single claim
"[claim text]" (Floyd, Harry, 2026, https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/)Replace the bracket with the row's claim text. The page URL carries the source; the row's own source link is in the row.- The essay
Floyd, Harry (2026). The Guardrail Your Agent Can Reach. The Durability Curve. https://durabilitycurve.com/blog/the-guardrail-your-agent-can-reach/- This ledger
Floyd, Harry (2026). Claim Ledger: The Guardrail Your Agent Can Reach [structured claims with sources]. The Durability Curve. https://durabilitycurve.com/claims/the-guardrail-your-agent-can-reach/
Quote with attribution and a link to this page or the essay. Say if you changed the wording. Not licensed for model training. Plain-text copy for machines: /md/claims/the-guardrail-your-agent-can-reach.md.