Same Model, Different Product: The Case for Harness Engineering

You can run the same model inside two coding tools and get two different products.

Put Claude Sonnet under Claude Code and under Cursor. Same weights, same context window, same benchmark scores on paper. In practice you get different token burn, different success rates, different cost per task, a different feeling about whether you can leave the thing running. Adam Elkassas, who builds with both, put it plainly.

Same Sonnet underneath Claude Code, Cursor, Cline, and a dozen no-name CLIs, and they feel like completely different products.¹

The size of the cross-harness spread is now on the record. Table 7 of the Meta-Harness paper sets its result beside official leaderboard entries on fixed models: Claude Opus 4.6 scores 58.0% under Claude Code and 76.4% under Meta-Harness; Claude Haiku 4.5 scores 27.5% and 37.6%. Those are cross-harness spreads of 18.4 and 10.1 points. Meta-Harness also beat Terminus-KIRA, one of the two open harnesses its search started from, which the leaderboard puts at 74.7% and 33.7%: by 1.7 points on Opus and 3.9 on Haiku.² Same model. Different surrounding system.

LangChain showed it from the other direction. Their coding agent, deepagents-cli, climbed from 52.8% to 66.5% on Terminal Bench 2.0, from outside the Top 30 into the Top 5, while the model underneath, GPT-5.2-Codex, stayed fixed.³ The score moved nearly 14 points. The model did not move at all.

The variable that moved the score has a name. The harness: the code that decides what the model sees, when it runs again, which tools it can reach, and what happens when it fails. Changing it well has become its own discipline now, with its own name: harness engineering. Most teams spend their attention choosing the model. The performance they are chasing lives one layer out, in code most of them have never opened.

For beginners: what is a harness? The model generates text. The harness is everything around it that turns a raw text predictor into something that can do work: the loop that calls the model again and again, the memory of what happened earlier, the list of tools it is allowed to use, the rules for what to do when a step breaks. Two products can run the identical model and still behave differently, because the harness around each one makes different choices. When people say “Claude Code feels different from Cursor,” the harness is most of what they are feeling.

Where harnesses diverge

Four harness decisions are worth opening before you switch models. None requires changing the weights.

What it keeps

Every harness has to decide what to keep as the conversation grows and what to throw away. A long task fills the context window with resolved debug cycles, completed file edits, and conversational noise. Keep all of it and the model drowns in its own words. Throw away the wrong thing and it forgets why it started.

Different harnesses make different trade-offs. Claude Code’s current documentation makes compaction explicit: /compact summarises the conversation to free context, automatic compaction can trigger near the context limit, and subagents can isolate side-work from the main conversation rather than inheriting its full history.⁴ The implementation changes over time. That is the point: context policy is part of the product, set by whoever builds the harness.

If you are running an agent on recency alone, keeping the last N messages and discarding the rest, you are leaving performance on the table and paying for the privilege in tokens.

What it does when it fails

When a step breaks, the harness has to decide whether to retry, compact, change tools or surface the failure. Claude Code exposes separate controls for failed API-request retries, structured-output retries and compaction. Manus makes the same point from the trace side: it deliberately keeps failed actions and their resulting errors in context because erasing the failure also erases evidence the model can use on its next step.⁴⁵

Collapse every failure into a single blind retry and you throw away information that could have changed the next decision.

A harness that treats a token overflow and a broken tool call as the same event has discarded the one piece of information that would have let the model recover.

What it lets the model touch

A large, poorly constrained action space makes tool choice harder. Manus says user-configurable agents can end up with hundreds of tools and reports that larger action spaces make wrong or inefficient action selection more likely. It keeps tool definitions stable for cache consistency and uses a context-aware state machine plus token masking to constrain which action groups the model can choose at a given state.⁵

The format of a single tool can move the number dramatically in some model-and-task combinations. Can Bölük’s hashline edit format lets the model point at lines by a content hash instead of reproducing the exact text. In his current benchmark, hashline beat patch in 14 of 16 models; Grok Code Fast 1 rose from 6.7% to 68.3%, while Grok 4 Fast’s output tokens fell 61%.⁶ One format did not win for every model. That caveat strengthens the useful point: evaluate the tool interface and the model together.

How the loop is built

The execution loop itself is an engineering decision. This is the layer where LangChain found 13.7 points. On a fixed GPT-5.2-Codex, it changed the harness rather than the model: system prompt, tools, middleware, reasoning allocation and verification behaviour, using traces to identify failure modes. Terminal Bench 2.0 moved from 52.8% to 66.5%.³ A team that can see which step corrupted the run can fix it. A team that only sees the final output cannot.

Why first-party does not automatically win

If harness design moves the number this much, the obvious question is why a first-party model lab does not automatically own the best harness. One possible answer is incentives, but the evidence here cannot prove it. Independent harness builders feel user-facing cost, latency and failure rates directly. Frontier labs optimise across a broader business: models, APIs, consumer products and enormous infrastructure commitments.

Adam Elkassas makes the sharper incentive argument: an independent harness maker has a stronger reason to cut token waste than a frontier lab does.¹ Treat that as a hypothesis about incentives; whether any lab acts on them is an open question. The infrastructure numbers establish only scale. Anthropic announced a $50 billion US computing-infrastructure investment with Fluidstack in November 2025. OpenAI said in September 2025 that Stargate had nearly 7 gigawatts of planned capacity and more than $400 billion of investment over the following three years.¹ Neither statement shows that either company prefers a token-heavy harness. The useful test is empirical: hold the model and task fixed, then compare first- and third-party harnesses on success, tokens, latency and cost.

Why the advantage lasts

It would be easy to read all this as a 2026 quirk that the next model erases. Some of it is. The parts of a harness that exist only to compensate for a model’s current weakness can shrink as the model improves.

The parts that do what a model structurally cannot do are moving the other way. Sandboxing, permissions, observability, memory: a more capable agent needs more of each, not less. The harness is not shrinking as a whole. The shrinking parts and the thickening parts are different parts.

The market has noticed. Menlo Ventures estimated Anthropic at 54% of enterprise coding-model usage and spend at the end of 2025, using survey-reported production API usage weighted by application scale as its basis. That was up from 42% six months earlier, and Menlo said Claude Code drove much of the rise. In February 2026 Anthropic said Claude Code itself had passed $2.5 billion in run-rate revenue.⁷ DeepSeek now publishes an official open-source DeepSeek Harness, dsh, and describes it as an agent harness developed by DeepSeek AI.⁸

When a model-first lab ships its own dedicated harness, the layer above the model is becoming a product in its own right.

For an investor the read is structural. A specific harness optimisation is temporary, and the next model may erase it. What compounds is the discipline of building harnesses, and the tooling and memory a team carries from one model generation to the next. That is the part a team owns. The model underneath it is rented.

The position has a clean exit. If a frontier lab ships a first-party agent that beats every third-party harness on the same model by more than 10% on Terminal Bench, the independent-harness edge is gone. If the open-source ecosystem converges on one dominant design and same-model gaps like the ones above (the 18.4-point cross-harness spread between Claude Code and Meta-Harness on Opus 4.6, and LangChain’s 13.7-point fixed-model gain from its own harness work) shrink below 5 points, the gap was transitional. Watch those two numbers. They are where the thesis dies if it is going to.

The Durability Curve tracks where value is moving in AI and markets, before the consensus reprices it. A new structural breakdown most weeks. [Subscribe free.]

Audit your own harness

You do not need the source code of Claude Code to find out whether your agent is harness-bound. Five questions locate the gap, and a team shipping an internal agent can answer all five about their own setup in an afternoon. The five checks below isolate the harness. If they look healthy and the agent still fails, The Seven-Layer Agent Audit widens the diagnosis to the six layers around it.

Start with what it keeps. Does your harness hold the plan and the current state, or the last N messages? If it is recency, the cheapest gain you have is sitting in the pruning logic, and you have probably never touched it.

Then what it does when it fails. Does a token overflow, a tool error, and a timeout produce three different responses, or one “retry”? If it is one, the model is recovering blind.

Then what it can touch. Does the model meet a curated set of tools, or a flat list of everything? If the list is long, trim it before you change anything else.

Then whether it knows why it is running again. When your loop calls the model back, does it pass the reason, or just “continue”? An agent told only to continue makes its next decision with no idea what just happened.

Then whether you can see inside a run. Can you score the intermediate steps, or only the final output? If step two fails quietly, step five inherits the corruption and you will blame the model for it.

A team that answers these honestly usually finds the same thing: they have changed the model three times and never opened the harness once.

The one-week test

Pick one agent you are running. Do not change the model this week. Instead, find the harness layer you have never touched, almost always context pruning or error recovery, and change that one thing. Measure cost per task and success rate before and after.

Most teams have never opened the part of their stack that decides what the model sees and what happens when it breaks. That is usually where the cheapest improvement is hiding, and you do not have to switch models to find it.

The model you picked is the part everyone can see. The harness is the part doing the work.

THE HARNESS AUDIT · five checks to run on your agent

1 · CONTEXT (what it keeps): the last N messages, or the plan and the live state?
2 · RECOVERY (when a step fails): one blind retry, or a different move per failure?
3 · TOOLS (what it can touch): a flat list of everything, or a curated few?
4 · FEEDBACK (why it reran): just "continue," or the reason it is running again?
5 · MEASUREMENT (inside a run): only the final output, or every step scored?

The first answer in each line is the harness-bound default.

THE MOVE
Take the weakest of the five for your agent. Change that one layer this week. Measure cost and success before and after.

THE LINE TO REMEMBER
Same model, different product. The wrapper is the variable.

---

When you last reached for a better model, was the bottleneck ever the model?

Footnotes

1. Adam Elkassas, pre.dev, “Frontier labs won’t build good harnesses. Their incentives won’t let them,” is the source of the incentive argument; it argues from incentives, user reports and pre.dev’s own harness work, and offers no direct evidence of how any lab prioritises token efficiency. Infrastructure scale is primary-sourced separately: Anthropic announced a $50 billion US computing-infrastructure investment with Fluidstack on 12 November 2025; OpenAI said on 23 September 2025 that Stargate had nearly 7 GW of planned capacity and more than $400 billion of investment over the next three years. https://pre.dev/blog/frontier-labs-wont-build-good-harnesses-their-incentives-wont-let-them/ https://www.anthropic.com/news/anthropic-invests-50-billion-in-american-ai-infrastructure https://openai.com/index/five-new-stargate-sites/ ↩ ↩² ↩³

2. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn, “Meta-Harness: End-to-End Optimization of Model Harnesses,” arXiv:2603.28052. Table 7 sets the authors’ Meta-Harness result beside official leaderboard entries: Claude Opus 4.6 scores 58.0% with Claude Code, 74.7% with Terminus-KIRA (one of the two open harnesses the search started from) and 76.4% with Meta-Harness; Claude Haiku 4.5 scores 27.5%, 33.7% and 37.6% respectively. Meta-Harness’s margins over Terminus-KIRA’s leaderboard scores are 1.7 and 3.9 points. The paper ran search and final evaluation on the same 89 tasks, and calls the resulting harness specialised to that benchmark. https://arxiv.org/abs/2603.28052 ↩

3. LangChain, “Improving Deep Agents with harness engineering”: deepagents-cli rose from 52.8% to 66.5% on Terminal Bench 2.0, from outside the Top 30 into the Top 5, on fixed GPT-5.2-Codex. LangChain says it focused the harness changes on system prompts, tools and middleware, and its account also describes verification guidance and reasoning-budget allocation inside that work. https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering ↩ ↩²

4. Claude Code documents /compact as conversation summarisation for freeing context, automatic compaction near context limits, configurable API and structured-output retry controls, and isolated subagent context. https://code.claude.com/docs/en/commands https://code.claude.com/docs/en/env-vars https://code.claude.com/docs/en/features-overview ↩ ↩²

5. Yichao “Peak” Ji, Manus, “Context Engineering for AI Agents: Lessons from Building Manus,” 18 July 2025. Manus reports that large tool spaces increase wrong or inefficient action selection, uses a context-aware state machine and token masking to constrain actions without changing tool definitions mid-run, and deliberately retains failed actions and errors as evidence for later steps. https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus ↩ ↩²

6. Can Bölük, “We improved 15 LLMs at coding in one afternoon. Only the harness changed,” 12 February 2026. The current benchmark table covers 16 models: hashline beats patch in 14 of 16; Grok Code Fast 1 rises from 6.7% to 68.3%; Grok 4 Fast output tokens fall 61%. https://stencil.so/blog/the-harness-problem ↩

7. Menlo Ventures, “2025: The State of Generative AI in the Enterprise,” estimated Anthropic at 54% of enterprise coding-model usage and spend, up from 42% six months earlier, and said Claude Code drove much of the rise. Its market-share method approximates dollars spent from survey-reported production API usage, weighted by application scale and triangulated with public financials; it does not measure coding-tool sales. Anthropic said in February 2026 that Claude Code’s run-rate revenue had passed $2.5 billion. https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/ https://www.anthropic.com/news/anthropic-raises-30-billion-series-g-funding-380-billion-post-money-valuation ↩

8. DeepSeek AI, DeepSeek Harness. The official repository describes dsh as “an open-source agent harness developed by DeepSeek AI” and currently labels it a developer preview. https://github.com/deepseek-ai/deepseek-harness ↩

Comments are on the Substack copy. If an essay was worth one, buy me a coffee.

It rests on Law IV, Instruments Over Theory, one of the laws this publication tests against the evidence.

Claude and ChatGPT can search this archive and its claim ledgers once you add the connector.

11 other essays point here

• How Long Until Your AI Edge Stops Paying?

  Most companies are not even there yet. Nearly every one runs AI somewhere now, and by McKinsey's 2025 survey almost nine in ten have adopted it while fewer than four in ten can attribute any measurable impact on profit.

• How Reliable Is Your AI Agent?

  Hold the model fixed and the surrounding harness can still move task success, cost and reliability materially, and a month of cleaning up after Ghost is that argument with scorch marks on it.

• One Question for Any AI Model Headline: Can You Use It?

  Even once you find it, the model's name is only half of what you get, and I wrote about the other half in Same Model, Different Product.

• Show It an Example

  I wrote about why the setup you give a model beats the model itself for anyone whose version of this has moved past writing taglines.

• Skills Are Package Management for Your AI

  That is architecture outliving content stated in one sentence, and it is the same structural bet behind why the same model behaves like a different product depending on the code wrapped around it.

• The Cheaper Fix You Keep Skipping

  In trading: do I lack a setup, or the discipline to wait for one? In building with agents: does the model lack capability, or does the harness lack structure? In a hard conversation: do I lack the right words, or can I not hold my state while I say them?…

• The Limit Said 10. The Loop Made 500 Calls.

  All of it is a property of the code you wrapped around the model, which is where the engineering actually lives.

• Your Benchmark Measures a Sprint. Your Agent Runs a Marathon.

  It is the model, the harness around it, the verifier, and the context discipline holding together across hundreds of steps, where any single link can break the run.

• The Seven-Layer Agent Audit

  Harness engineering drives a real share of the gap between agent products, and it is still one layer; the skills layer just below makes the same point from the other side, where a copied skill is a behavioural dependency and curated skills tend to lift…

• You Own an Asset You've Never Scored

  Give two operators the same model and the same task and the outcomes need not be close: one is running it through a specification that has absorbed a hundred hours of hard-won corrections, the other through six lines written once and never revisited.

• Your Multi-Agent System Is an Org Chart

  It is the same case as building the harness around the model instead of swapping the model: the structure around one strong agent does more of the work than the agent count ever will.