53.1 percent. That is how often an agent iterating on its own harness saw its benchmark feedback improve and its actual held-out performance improve too — the same direction, on tasks it had never seen. Flip a coin and you get 50 percent. When six frontier models were asked to build and then self-improve agent infrastructure, the signal that guided their engineering decisions was statistically indistinguishable from noise.
That number comes from HarnessDev (arXiv:2609.01437), published September 1 by a team spanning ByteDance Seed, Singapore University of Technology and Design, Georgia Tech, M-A-P, and TokenWave.AI. It is the first benchmark that does something this blog has argued for since its first post: it scores the harness — the runnable loop logic, tools, context handling, state, recovery, and stopping rules — instead of the answers the model produces inside one. Every other agent benchmark holds the harness fixed and varies the model. HarnessDev freezes the model question and asks a different one: can the model write the infrastructure its successors will run in?
The consensus answer in 2026 is yes — it is on every roadmap, in every keynote, embedded in the phrase "self-improving agents." The first rigorous measurement of that claim says: not yet, and the failure mode is not the one anyone predicted.
What the Benchmark Actually Does
HarnessDev is built around a clean separation: a creator model writes the harness; a separate executor model runs tasks inside it; an evaluator scores the results. That separation is the whole trick. When a harness fails, you can ask whether the harness is bad or whether the executor is weak — and by swapping executors, you can measure how much of a harness's performance is portable engineering versus co-adaptation to one specific model.
The benchmark has two stages. In Creation, the creator starts from a deliberately weak but runnable seed — file, search, and process primitives, no planner, no verifier, no retry logic, no stopping rule — plus a task-family spec and one to three development cases. It has to build a complete execution system from that. In Evolution, the creator starts from its own frozen harness and iterates against execution feedback from a 100-task SWE-bench Pro feedback set and all 89 Terminal-Bench 2.1 tasks, trying to improve downstream performance — with a 630-instance held-out SWE-bench Pro split whose scores are never shown to the creator, so it cannot overfit its version selection to them.
The scale is serious: 2,207 unique downstream instances across five benchmarks (SWE-bench Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, BrowseComp), six creators — Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro — each building three harnesses per benchmark pair, all developed inside Claude Code 2.1.177 (GPT-5.5 used Codex 0.144.3). This is not a prompt-engineering demo. It is a controlled study of harness-writing as a capability.
Creation: Capable, and Eighteen Points Behind
Under self-evaluation — the creator also serves as executor — the best model is Opus 4.8 with an overall average of 67.8. The human-engineered reference systems average 86.2. An 18-point gap is not a rounding error; it is the distance between "generates working infrastructure" and "does infrastructure engineering."
The domain splits are more interesting than the headline:
- Code: Opus 4.8 hit 69.3 on SWE-bench Pro against an 80.0 human reference. GPT-5.5 managed 32.8. Seed 2.0 Pro collapsed to 10.8.
- Terminal: Gemini 3.1 Pro led at 68.8 on Terminal-Bench versus an 88.8 reference.
- Search: the worst gap. GPT-5.5's best BrowseComp score was 52.6 against a 92.2 reference — a 40-point chasm.
- Writing: Opus 4.8 scored 84.6 on EQ-Bench3, above the 83.7 reference.
- ML experimentation: Opus (32.9) and Gemini (32.4) both beat the 24.0 reference medal rate on MLE-bench.
Read that pattern carefully, because it is the paper's most useful finding. Where the harness's job is to get out of the way — writing, ML experimentation, domains where a competent executor plus basic plumbing wins — generated harnesses match or beat human references. Where the harness is the product — search, where the reference systems encode years of iteration on query strategy, result triage, and stopping behavior — the gap is enormous. Models can generate the mechanics of agency. They cannot generate the accumulated judgment.
The failure attribution makes this concrete: 77.8 percent of failed data tasks were attributed to harness defects, not executor capability. The bottleneck was not the model running inside the loop. It was the loop.
The Executor Swap Is the Real Story
Here is the result that should reorganize how you think about the harness market. When HarnessDev re-ran every generated harness under a single fixed executor — Gemini 3.1 Pro — the rankings fell apart.
Opus 4.8's SWE-bench Pro score fell from 69.3 to 33.0. Same harness. Same tasks. Only the model executing it changed. Its duplicate-search rate on retrieval work jumped from 10.1 percent to 88.2 percent, and the audit found the Opus-built system had effectively hard-coded a 120-step limit around its original executor's behavior. Meanwhile Qwen's harness gained 17.6 points on BrowseComp and 12.9 on MLE-bench under the stronger executor. Some harnesses were holding their creators back. Others were built so specifically for their creators that they were unusable by anyone else.
The paper calls this creator co-adaptation. It has an uncomfortable implication for everything the industry is currently shipping. The pitch of managed harnesses — OpenAI's Agents API being last week's headline example — is that the orchestration layer is model-agnostic plumbing you can rent once and run anything through. The HarnessDev data says generated harnesses at least are not plumbing. They are shaped around a specific executor's habits: its typical trajectory lengths, its failure modes, its tolerances. Write a harness with one model and swap the executor, and you may have changed nothing about your system except its performance — by 36 points, in Opus's case.
This is also evidence for something harness engineers have said anecdotally for a year and now have numbers for: harness and model are co-designed whether you intend it or not. The question is whether your co-design is deliberate — tested across the executors you might run — or accidental, discovered the day a model deprecation forces a swap.
Evolution: The Self-Improvement Story Meets a Coin Flip
The Evolution stage is where HarnessDev is at its most brutal, because it tests the exact claim the "self-improving agent" narrative rests on: take execution feedback, improve the system, repeat.
The headline numbers look fine. All five self-runtime creators improved their visible feedback scores. All five declared final versions improved over their starting harness on held-out tasks — gains of +1.43 to +4.44 points, mean +3.11. Opus 4.8 delivered one genuinely impressive diagnosis: it noticed that 99 of 100 runs reported success while only 48 actually passed, traced the gap to premature completion, and added a completion gate to the harness. That is real debugging of control flow.
Then you read the fine print. Across nine lineages producing 73 official versions and 64 adjacent version switches, feedback scores and held-out scores moved in the same direction 34 times — 53.1 percent. Only 2 of 9 declared final versions were the best held-out version. Agents were repeatedly selecting "improved" versions that were worse on tasks they hadn't seen, because the visible feedback signal was too noisy to optimize against. Under the fixed Gemini executor, only Opus improved on held-out tasks at all; the other three lineages regressed, GPT-5.5 by 10.32 points.
In other words: the models can write harness code, and they cannot tell whether their edits helped. Generation is solved-ish. Evaluation — deciding which of your own changes to keep — is not. And the paper adds a detail anyone who has watched an agent churn will recognize: the dedicated trajectory interface for failure diagnosis was called only twice. The models would rather rewrite than read their own execution traces.
Dead Code and the Volume Fallacy
One more finding deserves its own paragraph because it will map onto your production experience. Across 18 generated systems, the models added 17,111 net lines of code — and code volume did not predict quality. Gemini 3.1 Pro added the fewest lines and led Terminal-Bench. Worse, much of the added machinery did nothing: of 108 code component instances, 18 never fired at runtime, all in state and memory handling. In the writing domain, 124 of 587 features were dead code.
Agents write infrastructure the way they write everything else — maximally, defensively, with features that exist to satisfy the plan rather than the execution. A harness component that never runs is worse than no component: it costs context window, misleads future edits, and creates the appearance of capability — recovery logic! memory! — that audit shows is inert. If you are letting an agent build tooling around itself and reviewing diffs by skim, this is your reminder that runtime instrumentation beats code review for this class of system.
What This Means for Harness Engineering
The optimistic read of HarnessDev — models wrote functional harnesses from a weak seed, beat human references in two domains — and the pessimistic read — eighteen points behind, coin-flip self-improvement, near-zero portability — are both true. The synthesis is that harness development is not reducible to code generation, which is precisely why the discipline this benchmark names is safe for now. Writing the loop is the easy part. Diagnosing why the loop failed, predicting whether a change generalizes, and knowing which executor a harness is secretly shaped around — those are engineering judgments, and the benchmark shows current models possess them only inconsistently.
There is a spectrum worth internalizing here, and it runs from "let the model draft the harness" to "let the model own the harness." The data supports the first and demolishes the second. Generated first drafts that a human then audits against runtime traces are already competitive in well-bounded domains. Autonomous evolution loops with noisy selection are, today, a machine for accumulating dead code and overfitting your eval set. The practical architecture is a human in the version-selection loop, not the code-writing loop — you review which changes ship, not which lines get typed. That division of labor is the core of harness engineering as I lay it out in Harness Engineering: The Complete Guide to Agent-First Software Development, and this paper is the first controlled evidence for why the version-selection job can't be delegated yet.
It also settles a Tuesday-afternoon argument you may have had: whether your team's hand-built harness is obsolete now that agents can generate one. HarnessDev says your harness — with its years of encoded failure lessons — is exactly the asset the generated ones can't replicate. The 86.2 reference didn't come from a smarter model. It came from survived production.
What to Do This Week
- Instrument before you generate. If you're letting agents write any part of your harness, add coverage tracking first. The paper's dead-code finding (18 of 108 components never firing) is fully detectable with cheap runtime instrumentation, and invisible in code review. Anything the agent adds that never executes gets deleted, not refactored.
- Test every harness change across two executors. The executor-swap collapse (69.3 → 33.0) is the cheapest insurance policy in this paper. Before merging a harness change, run your eval subset under your current model and your fallback model. If the change only helps one, you've found co-adaptation — decide deliberately whether that's acceptable, don't inherit it accidentally.
- Stop trusting a single feedback number to pick versions. 53.1 percent agreement with reality means single-run feedback is not a selection signal. If an agent is iterating on its own tooling, require it to report held-out or at minimum multi-seed results before you accept a "better" version. Two of nine declared optima should scare you.
- Read the paper's Evolution protocol and copy it. Visible feedback set, disjoint held-out set, frozen official versions, adjacent-switch bookkeeping — this is a production-grade harness CI design disguised as a benchmark methodology. It is more actionable than most vendor whitepapers this month.
The self-improving agent story assumes the agent can judge its own work. HarnessDev just measured that assumption at scale and returned 53.1 percent. Harness engineering as a human discipline survives this benchmark — and for the first time, it has a benchmark of its own.