How Reliable Is Your AI Agent?

For about three weeks I thought my server host was robbing me.

The agent I run, Ghost, is my own instance of Hermes, an open-source framework from Nous Research. It works on a rented box all day with me nowhere near it: research, drafts, scheduled jobs, the unattended autonomy everyone is being sold. In May it started slowing down. A job that took a minute took five. The server’s own dashboard blamed “steal time,” the polite name for a noisy neighbour on shared hardware eating the processor. So I did the normal thing. I complained to support, read forum threads about oversold hosts, priced a migration.

The neighbour was me. The failure had spent three weeks disguised as someone else’s fault, which is exactly what the dangerous ones do.

At the time, my Ghost setup started a fresh container for each chat session and never removed the dead ones.¹ They piled up. When I finally ran the one command that would have told me on day one, the list of dead containers filled the screen and kept scrolling: 748 of them, all but five long dead, and the machine underneath had seized trying to keep track. The host throttled everything. Nothing was broken in the way broken usually looks: no crash, no error, no alert, only a number climbing by one, over and over, for weeks, with nobody reading it.

That counter is what running an agent on your own actually looks like. The demo writes you a poem. The real thing is a number climbing in the dark while the disk fills.

FIG·02 — Nothing broke; a number climbed: 748 containers, all but five long dead, unread for weeks, into the cliff.

Grade it twice

The counter was one way Ghost had fooled me. There was another, and this one you can run on your own agent this week.

I ran a proper audit of its research against primary sources. Twenty-two factual claims about companies and markets, checked one at a time. Twenty came back directionally right, the right company and the right direction and the right thesis. Ninety-one percent. You could sell that number.

Then I graded the same twenty-two strictly. Was every specific right too, the exact figure, the exact date, the exact quarter. Seventeen. Seventy-seven percent.

The record I kept from that audit preserves the total and every miss: seventeen accurate, three approximate and two wrong. A blind re-check two months later, across this audit and the next, agreed with six of eight sampled claims, and both disagreements came from the later audit. The seventeen accurate claims survive only as counts, so I cannot replay that part of the run claim by claim today.

The gap between twenty and seventeen is the entire problem. The claims Ghost got strictly wrong were not inventions. They were the boring kind of miss: a price that was right last week, a date that had drifted, guidance that had since been raised. It had the shape of the world right and its current state wrong, and current state is the part you act on.

So an agent that is confidently, directionally right is more dangerous than one that is obviously wrong. The obviously wrong one you check. The directionally right one earns your trust and then spends it on a stale figure you paste into a memo for someone who acts on it. If you have ever pulled a number from an AI and dropped it into a deck, you have shipped one of these without knowing.

A 91% that hides a 77% is the kind of number that makes people stop checking.

The audit takes an hour and tells you something a benchmark cannot: how your own agent fails on the work you actually use. Take twenty of its claims and grade them twice, once for the shape and once for every specific, current as of today. I call the spread between those scores the blast radius. It shows how much a reassuring top-line accuracy can hide.

FIG·03 — Grade it twice. The 91% that hides a 77%, and the three claims you would have shipped.

It was never the model

Once I had that shape in my eye I saw it everywhere, and never in the model. The scheduler reported success whenever Ghost replied at all, so a job could fail outright, write back that it could not fetch the data, and still get logged green because something had come back. A config change I made was silently overruled by a second file that loaded later and won, pointing the agent’s storage at the wrong place; the system did exactly what the files told it, and nothing reconciled the two. The search index ballooned overnight to a size with no relation to the data inside it, until the disk hit 100 percent, the agent started returning “no space left,” and the work stopped, with nothing watching it grow.

None of this was the model’s fault. Resource leaks, configs that override each other, success signals that lie: these are the oldest problems in running software, and intelligence buys no exemption from them. The model underneath was cheap and fast; a frontier one would have hit all of them the same. A smarter model slips less often, and it still cannot see the slip it makes, because the evidence sits outside it. Every failure started in the same place: the agent did something, and nothing outside it looked at the result. A better model changes the odds. It does not remove the need for something outside the agent to look.

That was the turn for me. I had spent weeks grading Ghost on how clever it was. What decides whether an agent is safe to leave alone is duller and far harder to fake: how much of what it does gets checked by something that is not the agent.

An agent cannot be the thing that confirms its own work.

It reasons from inside its own process, where a sandbox that cannot start a container and a whole host that is down look identical. It will hand you a confident account of which one it is, and the account is worthless, because the fact that would settle it sits on the other side of a wall it cannot see over. An agent that grades its own work can always move the grade, which is why the only gate a self-rewriting agent cannot game is time: the checks that hold are the ones it has no hands on.

Different operators, same answer

For a while I assumed this was specific to my setup. Then I read other Hermes operators and kept finding versions of my own problems in their notes. None of us had compared notes; we were solving different problems, with different tools, in different words. One, tired of trusting the agent’s own edits, built a plugin that stages its proposed self-changes as diffs a human reviews before they go live.² One setup guide I found spends its length on the dull perimeter: what each surface may touch, what runs sandboxed, what a human approves.³ Others come at it from other directions, building evals to close the loop on output.⁴ Hermes itself now ships a Curator that tracks usage, moves long-unused agent-created skills through stale and archived states, and lets operators pin or restore them.⁵ The instinct is the same.

The pattern I kept finding was the same: what most of them had built, in the end, was a verifier.

This finds you whether or not you run a server. It bites anyone who lets an AI do something they then act on: the draft you send without rereading, the figure you quote because it sounded sure, the report you stopped opening because it is always fine. An agent does not have to be autonomous to fool you. It only has to produce something you have stopped checking.

You do not need your own month of this to learn what it teaches.

Where the check has to live

Fixing all of it was the same move every time. Put something outside the agent that can see what it cannot, and have it watch what is actually running, not the agent’s account of it. The form that takes depends on the failure.

The silent pile-ups, the containers and the disk, each had a leading number, one that moves long before the crash: the count of dead containers, the size of the index, the free space left. Those you watch directly. Set the alarm well below the cliff and read it far more often than it can break, every half hour rather than every twelve hours, so the warning fires while the problem is still a number and not yet a wall.

The lying scheduler was harder, because there the agent was the one producing the success signal. If a check can be passed by the thing it is meant to be checking, it proves nothing; what you need is a record the agent cannot paint green just by replying. So the failure log gets written from the real error, by the harness itself, where it fires whether or not the model chooses to cooperate.⁶

The stale figures got the narrowest fix of all. Ghost re-fetches a price or a date the moment it writes one, and never quotes from memory, because memory is where staleness hides. You pay that cost only on the things that decay, the price, the date, the version, the quarter. What does not move, it is allowed to remember.

All of that catches a failure after it happens, which is enough when the damage can be undone. A full disk clears. A stale figure gets corrected. It is not enough for money that has already moved, a post published under your name, or a file deleted. So the sorting key in my setup is reversibility. If an action can be undone, a watcher behind it can be enough. For the irreversible ones, the check sits in front of the action and is a person.

Ghost is barred from the irreversible ones. When it decides one is needed it writes a short request and stops. Other Hermes operators come to the same instinct from the other side: least privilege per surface, so the session you are sitting in front of can touch everything while the job that runs at four in the morning can be restricted to web and files.³⁷ Where you cannot gate an action, make it reversible instead, snapshotting before a change and archiving instead of deleting, so there is always a state to roll back to. The boundary is dumb and absolute on purpose.

A safety rule an agent can argue its way around is not a safety rule.

Look back at those fixes and they share one property. The counter it cannot fake, the log the harness writes instead of it, the date it has to re-fetch, the person standing in front of the irreversible action: the agent has no hands on any of them. That is the whole requirement. A check an agent can reach, it learns to play to, which is how most verification quietly fails: the check stops testing the work and starts testing the performance the agent puts on for the check. The watcher that survives is the one the agent cannot forge, whether it can see the watcher or not.

None of this made Ghost smarter. It made Ghost watched, and watched turned out to be what mattered. Hold the model fixed and the surrounding harness can still move task success, cost and reliability materially, and a month of cleaning up after Ghost is that argument with scorch marks on it. The watchers and the gates and the re-fetches are where an autonomous system’s real competence lives. You will swap the model next month. The instruments stay.

An afternoon with a pen

So the work that matters has nothing to do with grading the output, the one part that was never going to break. It is an afternoon with a pen. Write down everything your agent does while you are not watching: every scheduled job, every file it writes, every figure it fetches, every change it makes to itself. That list is its real reach, and it is often longer than you expect. Then put each line through three questions. Can you undo it, and if not, does a person stand in front of it? What looks at the result, and is it anything other than the agent itself? And the question that catches what the first two miss: could the agent make that check pass without doing the work?

FIG·04 — The verification pass: Reach, Reversibility, Witness, Forgery.

Take the dullest case you like, an assistant that drafts your weekly update and posts it to the team channel on a standing job. A posted message is read before you can take it back, so a person should see it first. Then ask what actually confirms the figures in it, and whether that is anything more than the assistant rereading its own draft. Ask whether it could report “posted, all good” with last week’s numbers inside. Two of those checks are usually empty, and the empty ones are where it bites.

Every action checked by nothing but the agent is one of my 748 containers.

It is not failing yet.

It is adding one to a counter nobody is reading.

Run it on your own setup this week. Which of your agent’s actions is checked by nothing but the agent itself?

The Durability Curve is where I write up what outlasts the model: the harnesses, the checks, the structure that is still standing after you swap the engine underneath. Subscribe for the rest, or start with what survives.

Footnotes

1. Current Hermes behaves differently from the setup described above. Its Docker backend now defaults to one long-lived container shared across sessions and processes; per-session isolation removes its container when the session closes or goes idle, and an orphan reaper sweeps exited Hermes-labelled containers at the next startup. Checked 2 October 2026 against the Hermes configuration docs at commit 50a6abca: https://github.com/NousResearch/hermes-agent/blob/50a6abca7cff83afb0ebfe930a77c67311aa1044/website/docs/user-guide/configuration.md#container-lifecycle ↩

2. Tony Simons (@tonysimons_), announcing Hermes Dreaming (X, May 2026). A plugin that stages an agent’s proposed self-edits as create, diff, validate, then apply or discard, so a human reads the change before it touches live state. ↩

3. zaimiri (@zaimiri), an X article on eight Hermes Agent settings to check before building anything (X, June 2026). Argues for per-surface tool access, sandboxed execution, manual approvals, and a cron setting that stops scheduled jobs approving dangerous actions while you are away. ↩ ↩²

4. Machina (@exm7777), “How to Fix AI Slop” (X, May 2026). Frames inconsistent output as a quality-control gap rather than a prompt problem, and builds the fix as an eval loop: generate, score against a written benchmark, gate, and feed every failure back as a permanent test case. ↩

5. Current Hermes Curator documentation describes a background maintenance pass for agent-created skills that tracks usage, moves long-unused skills through active → stale → archived, skips pinned skills during automatic transitions, and supports restore/rollback. Checked 2 October 2026 at commit 50a6abca: https://github.com/NousResearch/hermes-agent/blob/50a6abca7cff83afb0ebfe930a77c67311aa1044/website/docs/user-guide/features/curator.md ↩

6. Current Hermes hooks documentation exposes pre_tool_call hooks that can block or fail closed before a tool runs, plus post_tool_call hooks that receive the returned result for logging, metrics and success/failure tracking. Checked 2 October 2026 at commit 50a6abca: https://github.com/NousResearch/hermes-agent/blob/50a6abca7cff83afb0ebfe930a77c67311aa1044/website/docs/user-guide/features/hooks.md ↩

7. Current Hermes cron documentation says unattended jobs use the separately configured cron platform toolset, and that a job-level enabled_toolsets list can narrow one run to web and file. Checked 3 October 2026 at commit 50a6abca: https://github.com/NousResearch/hermes-agent/blob/50a6abca7cff83afb0ebfe930a77c67311aa1044/website/docs/user-guide/features/cron.md#toolsets-available-to-cron-jobs ↩

Comments are on the Substack copy. If an essay was worth one, buy me a coffee.

It rests on Law IV, Instruments Over Theory, one of the laws this publication tests against the evidence.

Claude and ChatGPT can search this archive and its claim ledgers once you add the connector.

8 other essays point here

• Never Let Claude Code Tell You It's Done

  If you want the why under this: How Reliable Is Your AI Agent makes the broader case, and The Stable Liar is this exact failure mode up close.

• Start Here: What Survives When The Surface Changes?

  Start with How Reliable Is Your AI Agent?, which is the piece the rest of this publication keeps returning to.

• Ten Lines of Code Scored 100%. One Agent Broke Eight Benchmarks.

  It is the same property that decides whether you can trust a sub-agent at all.

• The Safe Parts of Your Job Are the First to Go

  Think of the analyst who trusts a model all quarter, not noticing it has quietly drifted.

• The Most Expensive AI Errors Are Made of True Numbers

  The agent is the same one from How Reliable Is Your AI Agent?, the reliability piece: a Hermes-framework agent on a deliberately cheap model, running unattended on a rented server, researching markets and AI.

• Your AI Looks Best Where You Can Check It Least

  An agent whose failures compound quietly until the host seizes is a failure you can still recover from.

• Your Research Agent Cites Sources It Never Read

  This discipline is the principle behind an external check your agent cannot talk its way past, the reason an agent cannot verify its own work, and exactly what fails when ten lines of code can score full marks on a benchmark by reading the answer key…

• Your Tests Pass. So What?

  Related: How Reliable Is Your AI Agent makes the broader case, The Stable Liar shows the failure mode up close, and Never Let Claude Code Tell You It's Done builds the test gate this piece audits.