17,000 lines versus 115,000 lines. Same model, same task, same benchmark, same hardware. The only difference was who decided the work was finished.
On August 27, Factory Research published an experiment on ProgramBench, the cleanroom software-engineering benchmark where an agent must reproduce the observable behavior of a reference program it can run but never read. Factory asked Droid to rebuild GDAL — the geospatial library whose CLI surface alone spans roughly 600 thousand lines of C/C++ — twice. Once as a single agent that implemented, checked its own work, and decided for itself when it was done. And once as a multi-role system where a separate role built an executable standard of completion before implementation began, and the implementer was held to that standard. The single agent shipped 17,000 lines. The system grew to 115,000 lines and reached 90 percent behavioral parity with the reference.
That gap is not a model capability story. Every role in the system condition used the same model at the same reasoning level as its single-agent counterpart. The gap is architectural. And its most uncomfortable finding is buried in the middle of the write-up: every single-agent campaign ended because the agent decided to end it. Not because it hit a context limit. Not because it ran out of budget. Because it judged itself complete.
The single agent didn't lack skill. It lacked a standard of completion.
The Experiment
ProgramBench is a demanding limit case. Each task hands the agent a reference program, fixtures, and partial documentation. The reference is a black-box oracle — it may be executed without limit, but never read, decompiled, or traced. The goal is to reproduce the reference's observable behavior from scratch, graded against a hidden suite of behavioral checks the agent never sees.
Factory selected 24 of the benchmark's hardest tasks and ran each through two conditions with three frontier models: Claude Fable 5, Kimi K3, and GPT-5.6 Sol. In the single-agent condition, the agent investigates, implements, runs checks, judges its own work, and chooses when to stop. In the system condition, three things change. A validator constructs an independent instrument — its own account of what the reimplementation must do and what evidence would prove it — before any implementation starts. The implementer never sees raw results from that instrument; clustered findings cross an information wall. And an orchestrator adjudicates findings, issues directives, and decides when to ship.
The results were lopsided. Fable 5's median score jumped from 56.7 to 89.3 — closing 73 percent of the gap to perfect parity. GPT-5.6 Sol went from 48.6 to 66.2. Task by task, the deltas are stark: 7-Zip from 54 percent parity to 95, DuckDB from 34 to 80, lazygit from 46 to 93, lnav from 38 to 84, pandoc from 19 to 84. These are not marginal gains from a better prompt. They are what happens when the judgment of completion moves outside the agent that did the work.
Why This Isn't a Compute Story
The obvious objection: the system runs were longer and more expensive. For GDAL, the system condition consumed 14 times the credits and 13 times the wall time of the single-agent run. Maybe this is just "more compute wins."
It isn't, and Factory's own data says so. The critical observation is that additional compute does not help an agent that will not spend it. Every single-agent campaign terminated on the agent's own judgment. You could have given the lone Droid ten times the budget; it had already decided it was done. The single agent's GDAL run didn't fail because it ran out of room to write code. It failed because at 17,000 lines it looked at its own work, ran its own checks, and concluded — sincerely, wrongly — that it had covered the behavior space.
What the system condition changed was the judgment of completion. The compute followed from that judgment, not the other way around. This inverts the usual scaling narrative. Most of the 2026 discourse treats agent performance as a function of model capability times tokens spent. Factory's experiment suggests that on long-horizon tasks, the binding constraint is something else entirely: whether anything in the loop can tell the difference between "looks done" and "is done."
There's a second, subtler mechanism worth naming. The validator built its instrument before implementation began — before the implementer's attention had narrowed to whatever partial solution was already on disk. This is a well-known failure mode in human software too: the moment you have 5,000 lines written, your definition of "correct" drifts toward "consistent with what I've built." Agents do the same thing, faster. An agent that writes its own acceptance criteria after the fact is grading its own homework with an answer key it derived from its own answers. Moving the standard's construction to before implementation, and its enforcement outside the implementer's context, is the cheapest correction anyone has demonstrated at this scale.
The Pattern Is Everywhere Once You Look
Once you internalize "agents don't know when to stop," you start seeing it across this month's results.
Prime Intellect's NanoGPT Speedrun Frontier, published August 24, ran 18 models through 153 autonomous optimization tests. The spread was enormous — Claude Opus 5 completed 81.7 percent of the suite, Kimi K3 hit 52.2, and a long tail clustered below 15. But the failure analysis matters more than the leaderboard: struggling models fell into recursive loops or stopped improving too early, a pattern the researchers called premature convergence. The top performers weren't just smarter — they used structured subagent delegation and systematic tool invocation. They externalized the judgment of progress. Same diagnosis, different benchmark.
Or take SWE-Bench ProMax, the refactoring benchmark released August 21 by researchers at Shanghai Jiao Tong University, Peking University, and Douyin Group. The best model managed a 41.2 percent resolve rate on 170 real large-scale refactoring commits across seven languages. Refactoring is the purest test of knowing-when-you're-done: zero tolerance for behavior change, complete reversibility, cross-file scope. It's exactly the task where "I ran the tests I thought of and they pass" is worthless — because the tests you thought of are the ones your change was shaped around. A 41 percent resolve rate on refactoring while the same models clear 95 percent on SWE-bench Verified tells you the remaining gap is almost entirely in the verification layer, not the code-writing layer.
The convergence of these three results — Factory, Prime Intellect, ProMax — across totally different benchmarks and organizations is the strongest signal we've had that the bottleneck in agentic software has moved. Code generation is largely solved. Completion judgment is not.
The Contrarian Take: Stop Buying Models, Start Building Instruments
Most teams shopping for agents right now are optimizing the wrong variable. They compare model leaderboards, argue about pricing per million tokens, and A/B test frontier models against each other on their own tasks. But Factory's experiment — like every controlled harness comparison this year — shows that the spread between architectures dwarfs the spread between models. Sol and Fable differ by single digits on most benchmarks. Sol alone versus Sol in a system with an independent completion standard differed by 17.6 points on the same 24 tasks.
If you're building with agents and you don't have anything in your loop that plays the validator role — an external, executable standard of completion, derived from the outcome, constructed before implementation narrows attention, enforced by something that isn't the implementer — then you don't have an agent system. You have a very fast worker with no definition of done, and you are the validator. Every "the agent said it finished but the feature doesn't work" incident in your issue tracker is the single-agent condition running in production.
This is also, for what it's worth, the central argument of Harness Engineering: The Complete Guide to Agent-First Software Development: the durable engineering skill in the agent era is designing the system around the model — the roles, the information walls, the standards, the stopping conditions — because the model inside the loop keeps changing every quarter and the architecture is what you own.
What to Do This Week
You don't need Factory's 96-hour campaign runs to apply this. Three concrete moves:
- Write the completion standard before the agent writes code. For your next real task, write down — as executable checks, not prose — what evidence would prove the work is done. Do it before prompting. If you can't write the checks, you don't understand the task well enough to delegate it. That was always true of human delegation too; agents just exposed it.
- Separate the grader from the worker. Even a cheap second pass helps: a separate context, ideally a separate model or at minimum a fresh session with no access to the implementer's reasoning, that scores the artifact against the standard and reports clustered findings — not raw logs. The information wall in Factory's design wasn't decoration; it's what stopped the validator from being anchored by the implementer's rationalizations.
- Move the stop decision out of the agent. Your harness — not the model — should decide when a run is over, based on the instrument's verdict. Any agent loop whose exit condition is "the model said done" will systematically undershoot on complex work and you'll never see it in the logs, because nothing in the logs looks like failure. It looks like confidence.
The Uncomfortable Endorsement
One detail in Factory's results deserves more attention than it will get. The system condition's GDAL recreation matched 90 percent of the reference's measured behavior in 115 thousand lines — for a program whose reachable surface is roughly 600 thousand. That means the multi-role system didn't just score higher; it produced an artifact of a fundamentally different magnitude. Same model. The capability to write 115 thousand coherent lines was always there. Nothing unlocked it except a definition of done that refused to accept 17 thousand.
The next time a vendor tells you their new model is 5 percent better on a leaderboard, ask the question Factory's experiment actually answered: in your harness, who decides when the work is finished? If the answer is "the model," you're leaving more than 5 percent on the table — and unlike model capability, this one is yours to fix this week.