99 out of 100. That's how many entries on the SWE-bench Verified leaderboard are self-reported. One lab — the lab that co-created the benchmark — quietly stopped reporting the score. And on July 28, 2026, OpenAI published the autopsy.
Their verdict was brutal. SWE-bench Verified, the 500-task human-validated subset that became the industry's yardstick for coding agents, is contaminated and broken. More than half the problems OpenAI audited have flawed tests that reject correct solutions. Every frontier model they probed — GPT-5.2, Claude Opus 4.5, Gemini 3 Flash — could reproduce the exact gold patch verbatim from a task ID alone. The benchmark isn't measuring coding ability anymore. It's measuring memorization and harness tuning.
This isn't a story about OpenAI throwing shade at competitors. It's a story about what happens when a benchmark saturates: the signal migrates from the thing being measured to the apparatus doing the measuring. And the apparatus, in this case, is the agent harness.
The Autopsy
OpenAI's Frontier Evals team audited 138 SWE-bench Verified problems that their models failed to consistently solve across 64 independent runs. Each problem was reviewed by at least six experienced software engineers. The findings, published July 28:
59.4% of the audited tasks contain material flaws. 35.5% have test cases so narrow they enforce specific implementation details — rejecting functionally correct code because it doesn't use the exact function name the original PR author happened to choose. 18.8% have tests that check for functionality never mentioned in the problem description. The remaining 5.1% have miscellaneous issues.
One example: a pylint task where the tests import a function called get_annotation that the problem description never mentions. Many valid solutions fail on import errors. The benchmark isn't testing whether the model can fix the bug. It's testing whether the model can guess the exact name a specific developer chose in a specific PR.
As Mia Glaese from OpenAI's Frontier Evals team put it: the assessment "no longer measures coding capability of our agents, but the agent's ability to correctly guess how to name a specific function."
The Contamination Problem
The flawed tests would be bad enough. The contamination is worse.
SWE-bench's problems are sourced from public GitHub repositories — Django, SymPy, scikit-learn, pytest, Flask, matplotlib. These are exactly the repositories that frontier models train on. OpenAI built an automated red-teaming setup: they gave GPT-5 a task ID, description, gold patch, and PR tests, then let it probe GPT-5.2-Chat, Claude Opus 4.5, and Gemini 3 Flash over 15 turns to elicit contamination.
All three models leaked. GPT-5.2 output the exact gold patch from a short snippet of the task description — including the specific early-return condition if username is None or password is None. Claude Opus 4.5 recalled the exact four-line functional change, the filename, the method name, and quoted verbatim the inline comment from the diff. Gemini 3 Flash, given nothing but the task ID, output the exact regex formula for username validation and the precise line numbers of the change.
Independent research corroborates this. One study found that 32.67% of successful SWE-bench Verified patches involved solution leakage. Models recall correct file paths from training data up to 76% of the time. When a model can "solve" a third of the tasks partly by remembering the answer, a 95% score is measuring memory as much as capability.
State-of-the-Art Progress Has Flatlined
Here's the number that should give everyone pause. Over the last six months, state-of-the-art on SWE-bench Verified crawled from 74.9% to 80.9%. Six points in half a year. Meanwhile, on every other capability axis — reasoning, multimodal, math — models have jumped double digits in the same period.
The ceiling isn't the model. The ceiling is the benchmark. When GPT-5.2 solved 31 tasks that OpenAI had classified as nearly impossible, it wasn't because GPT-5.2 had a breakthrough in coding reasoning. It was because GPT-5.2 had seen the release notes during training and knew that the edit_only parameter was introduced in Django 4.1 — a fact the test required but the problem description never mentioned.
That's not progress. That's data leakage dressed up as capability.
The Four Numbers Hiding Behind One Name
Here's where the story gets interesting for anyone building agent systems. "SWE-bench" is no longer one number. It's four, and they diverge wildly.
Take Gemini 3.1 Pro. On SWE-bench Verified, it scores 80.6%. On SWE-bench Pro — the harder, contamination-resistant successor with 1,865 tasks across 41 repositories in Python, Go, TypeScript, and JavaScript — it drops to 54.2% on a vendor scaffold. On Scale AI's standardized SEAL public harness, where every model runs the same agent, it falls to 46.1%. That's a 34.5-point cliff from the headline number to the apples-to-apples comparison.
Claude Opus 4.5 shows the same pattern: 80.9% on Verified, 45.9% on SEAL-standardized Pro. A 35-point drop.
And Claude Fable 5 — the model currently dominating launch coverage with a 95.0% Verified score and an 80.3% Pro score — has not yet appeared on Scale's standardized SEAL leaderboard at all. The 80.3% is vendor-reported using Anthropic's own scaffold. Epoch AI's independent evaluation was still pending. Treat it as a marketing claim awaiting confirmation.
The Harness Is the Dominant Variable
This is the part that matters for anyone shipping agent infrastructure. Scale AI's analysis puts the swing from harness choices at 10 to 20 points. The cleanest demonstration: three different agent systems ran the same Claude Opus 4.5 model against SWE-bench Pro and produced scores from 50.2% to 55.4%. The model was held constant. Only the scaffold changed. Five points of difference, purely from how each agent managed context, tool calls, and retry logic.
Now scale that up. The typical vendor scaffold differs from a standardized harness across six variables:
- Context management: Vendor scaffolds tune retention per model and are often generous. More retained context helps the model find the right files.
- Attempt budget: Vendor caps are higher, sometimes multi-attempt. More turns means more chances to converge on a passing patch.
- Tool-use integration: Vendor scaffolds use model-native, deeply optimized tooling. Cleaner tool calls reduce wasted turns and parse failures.
- Retry logic: Vendor scaffolds implement custom recovery on test failure. Recovering from a failed run rescues otherwise-lost tasks.
- File navigation: Vendor scaffolds use bespoke search and indexing. Faster file discovery is most of the work on real repos.
- Error feedback: Vendor scaffolds parse and summarize errors for the model. Readable errors help self-correction between attempts.
None of these variables are the model. All of them move the score. When Anthropic reports 80.3% on SWE-bench Pro and the best standardized Claude score is 51.9%, the 28-point gap isn't model capability. It's harness engineering.
The SWE-bench leaderboard now ranks scaffolds, not models. And most people reading it don't realize that's what they're looking at.
The Private Codebase Reality Check
If you want the number that actually predicts what a coding agent will do on your codebase, there's one dataset that matters. Scale maintains a private commercial SWE-bench Pro subset — 276 tasks drawn from proprietary codebases the models could not have trained on. Every model's score collapses when it moves there.
Claude Opus 4.6 (thinking) leads at 47.1%, down from 51.9% on the public set. GPT-5.4 xHigh drops from 59.1% to 43.4% — a 15.7-point fall. The best agent in the world, on code it has genuinely never seen, resolves fewer than half of realistic engineering tasks unaided.
That's the number to plan around. Not 95%. Not 80%. Forty-seven percent.
Why This Vindicates Harness Engineering
Most coverage of OpenAI's analysis framed it as a benchmarking story — a tale of contamination and flawed test design. That's the surface. The deeper lesson is structural.
For two years, the industry treated the model as the primary variable and the harness as plumbing. Pick the model with the highest SWE-bench score, wire it into your tools, and ship. The benchmark encoded that mental model: one number, one model, one ranking.
That framing is now empirically false. The benchmark proved it by dying. When the same model produces scores ranging from 43% to 80% depending on the harness wrapped around it, the harness isn't plumbing. It's the product. The model is the component.
This is the central thesis of harness engineering as a discipline. The leverage in agent systems doesn't live in the weights. It lives in the orchestration layer — the context management, the tool routing, the retry logic, the verification gates, the file navigation strategies. The teams that invest in their harness will outperform teams running a "better" model on a worse harness. The SWE-bench data proves it: a 28-point swing from scaffold alone is larger than the gap between the best and second-best model families.
What to Do This Week
If you're building or buying coding agents, three things changed this week:
1. Stop trusting the headline number. Any SWE-bench Verified score above ~80% is best read as a tier filter, not a ranking. It tells you a model is worth evaluating further. It does not tell you that a 95% model will outperform an 88% model on your codebase — because much of the gap is scaffold and memorization, not capability. When a vendor quotes SWE-bench, ask which variant, which scaffold, and whether it's self-reported.
2. Demand standardized evaluations. Scale's SEAL leaderboard is the only place every model runs the same harness. That's the comparison to trust for capability ranking. If your vendor can't produce a standardized eval, or quotes only vendor-scaffold numbers, treat their claims accordingly. The era of accepting self-reported benchmarks as evidence is ending — pushed there by the lab that helped create the benchmark in the first place.
3. Build your own eval harness. The private codebase numbers — 43% to 47% for the best models — are the honest expectation-setter for proprietary work. But those are aggregate numbers across 276 tasks. Your codebase is not an aggregate. Run 50 real issues from your backlog through your agent stack. Measure the pass rate. That number, measured on your harness against your code, is the only benchmark that decides anything. The harness engineering practices in Harness Engineering: The Complete Guide to Agent-First Software Development cover the eval loop patterns that make this practical to build and maintain.
The Benchmark Era Is Ending
OpenAI recommends SWE-bench Pro. They're right that it's better — less contaminated, harder, more realistic. But SWE-bench Pro has the same structural problem, just delayed. It will saturate. The vendor scaffolds will optimize for it. The same dynamic will repeat.
The deeper shift is that single-number model comparison is dying. When one benchmark name points to four different numbers, and the harness can swing the score by more than the model choice itself, the discipline that wins isn't picking the highest row on a leaderboard. It's building a controlled, scaffold-aware evaluation against the work you actually do.
The benchmark stopped measuring models. It started measuring harnesses. The teams that internalize this — that treat their harness as the competitive surface and their eval loop as the source of truth — will be the ones shipping agents that work. The rest will keep chasing a number that no longer means what they think it means.