GPT-5.5 and Claude Opus 4.7 differ by four-tenths of a percentage point on resolve rate across 523 agentic coding tasks, and by roughly one cent per run. On any leaderboard, they are the same model. According to the trajectory analysis JetBrains published on August 31, they are in fact opposites: one hallucinates in a third of its runs and always checks its work; the other barely hallucinates, writes the cleanest patches in the field — and skips validation nearly a quarter of the time. Same score. Opposite failure modes. Pick wrong, and you won't see it in the benchmark. You'll see it in production, in the failure mode you didn't know you were selecting for.
This is the most important evaluation result of the quarter, and almost nobody will treat it that way, because it doesn't produce a ranking. It produces something better: proof that the aggregate score — the number the entire model-selection industry is built on — is now the least informative digit on the board.
What JetBrains Actually Measured
The setup is simple. Take a coding agent — JetBrains' Junie — and instead of scoring only whether it resolved the task, record everything the agent did along the way: the files it read, the searches it repeated, the commands it ran, the tests it executed or didn't, the shape of the final diff. The trajectory doesn't reveal private reasoning, but it shows how the model worked the repository. Did it localize the bug before editing? Did it test its assumptions? Did the patch stay contained, or did it drag in unrelated changes?
From those logs, JetBrains built a four-perspective pipeline:
- Outcome: did the patch resolve the task, and which tests passed or failed.
- Efficiency: tokens, model calls, tool calls, seconds, cost.
- Patch quality: did the change touch the relevant files and symbols, stay contained, avoid unnecessary complexity.
- Process quality: how the agent moved through exploration, implementation, and validation — whether it reproduced the problem, repeated work, or stopped without testing.
Deterministic metrics from logs. Semantic evaluation from LLM judges. Result plus path. It's the difference between judging a surgeon by survival rate and watching the operation.
The Headline Numbers Hide the Story
The first comparison ran Claude Opus 4.7 against Gemini 3.5 Flash across four benchmark datasets totaling 523 tasks. Opus resolved 267 (51.1%); Gemini Flash resolved 254 (48.6%). Close. But the genuinely striking number is agreement: both models produced the same outcome on 430 of 523 tasks — both solved 214, both failed 216. Only 93 tasks separated them at all.
Two models with different architectures, different training data, and different owners agree on 82% of tasks, and the industry's default evaluation instrument — a resolve-rate leaderboard — compresses that entire distribution into two decimal numbers two and a half points apart. The information being thrown away is the actual signal.
Because on those 93 separating tasks, and on everything the score never sees, the models are nothing alike.
Profile One: The Diagnostician Who Doesn't Test
Opus 4.7's profile reads like a strong senior engineer with a blind spot. Its strength is diagnosis: it was substantially more likely to identify the underlying cause of ambiguous defects — finding the right mechanism or architectural layer rather than patching the visible symptom — and it solved 53 tasks that Gemini missed. If your problem is understanding an unfamiliar repository, that's the profile you want.
Then the weakness, and it's a weird one: what happens after localization. Some runs found the right mechanism but stopped at a reproduction test, missed a companion call site, or implemented a plausible custom solution instead of following an existing repository pattern. More concretely: in 123 runs, Opus performed no executable validation at all. Sixty-eight of those runs still resolved the task — which means 68 green checkmarks on the scoreboard were patches whose correctness was never confirmed by running them. Resolved, yes. Verified, no. A score can't tell the difference; a regression three weeks later can.
Profile Two: The Validator Who Invents APIs
Gemini 3.5 Flash's profile is the mirror image. Its risks are grounding and convergence: it kept searching after already reaching relevant code, repeated equivalent commands, burned steps on build infrastructure, and leaned on unverified APIs, dependencies, paths, and test fixtures. JetBrains assessed 195 of its runs — 37.3% — as containing moderate or severe hallucination, against 130 for Opus. Eighty runs (15.3%) showed significant or severe patch redundancy, more than twice Opus's 6.5%.
So: one model that finds the root cause and sometimes doesn't check its answer, and one that checks diligently while inventing a quarter of the APIs it touches. Both finish within three points of each other. If your review capacity is the bottleneck, the redundant-patch model is your problem. If you merge on green tests, the validation-skipper is your problem. The leaderboard prices them identically.
The Tie That Isn't a Tie
JetBrains then widened the field: GPT-5.5, Opus 4.7, Gemini 3.5 Flash, and Qwen 3.6 27B FP8 on the 522 tasks all four share. GPT-5.5 posted the highest resolve rate at 51.5% — and was the only model that ran an executable check in literally every run. Opus led every patch-quality metric and produced the shortest trajectories of the four. Qwen 3.6 27B FP8 resolved 38.9% at 3% of GPT-5.5's cost per run.
Look at GPT-5.5 versus Opus specifically, because it's the cleanest argument in the whole dataset. Four-tenths of a point apart on resolve rate. Within a cent of each other per run. By leaderboard logic, interchangeable. Their patches are not: moderate or severe hallucination in 33.7% of GPT-5.5's runs versus 24.7% for Opus; significant redundancy in 13.2% versus 6.3%; Opus with the shortest trajectories; GPT-5.5 with perfect validation discipline while Opus skipped validation in 23.6% of its runs. What GPT-5.5 sells you is process reliability. What Opus sells you is artifact quality and efficiency. The scoreboard says "tie" because the scoreboard averages away everything that differs.
And Qwen's row is the quiet one that changes procurement: 12.6 points behind on resolve rate, but cheap enough that a failed run costs almost nothing. As JetBrains puts it, which model is preferable depends on whether the expensive part of your work is diagnosis, patch review, or the run itself. That's not a ranking question. It's a routing question — and routing questions are harness questions.
Why This Is Happening Now
The shift from scores to trajectories isn't a methodological fashion. It's what happens when the scores run out of resolution. Three data points from the past few months, all pointing the same direction:
Aggregate benchmarks are saturating. Terminal-Bench 2.1's top agents are around 84%, which is why Terminal-Bench 3.0 exists — and on 3.0, the best model (Claude Opus 5, by Snorkel's reporting) achieves 43.5%. When everyone aces the test, the test stops discriminating, and the discriminating information migrates below the aggregate: not whether the agent passed, but how, and what it broke on the way.
The aggregates were partly measuring broken tests. OpenAI's own February 2026 audit of SWE-bench Verified examined 138 problems its models couldn't consistently solve and found most weren't hard — they were broken: roughly 59% contained flawed test design or unclear descriptions, and more than a third rejected functionally correct solutions. OpenAI stopped leading with Verified scores. When the reference answer is wrong, pass/fail is noise, and trajectory review is the only signal left.
Trajectories are becoming product infrastructure. JetBrains isn't alone. AgentLens — an ACL 2026 paper with an open-source benchmark — evaluates coding agents on instruction-following, tool use, verification, error recovery, and developer experience across full trajectories, and its authors run it as a nightly pipeline to catch regressions in their own agent. This is eval shifting from an event (launch day, leaderboard submission) to a continuous instrument. When evaluation runs nightly on trajectories, model selection stops being a ranking you consult once and becomes a profile you match against task mix — continuously.
The Evaluation Unit Is the Harness
Here's the structural point, and the reason this matters more than one interesting blog post from an IDE vendor. A trajectory is not a property of the model. It's a property of the model plus the harness — the tool schemas, the context management, the approval gates, the timeout policy, whether the loop strips reasoning state between turns. JetBrains is explicit about this caveat: their profiles describe Opus 4.7 and Gemini 3.5 Flash as run in Junie's scaffold, not the models in general. Run the same weights through a different harness and you'll get different behavioral profiles.
That caveat is the whole game. If behavior is produced by model-plus-harness — and it is — then behavioral evaluation is harness engineering. The team that logs trajectories, computes validation rates and redundancy rates on its own tasks, and tunes its loop against them is doing the same discipline as the team that just picks the top leaderboard model, except one of them is steering and the other is reading a compass that no longer points anywhere in particular. The trajectory is where model capability and harness design become inseparable, which is exactly why it's the right unit of analysis for agent-first development — an argument I make at book length in Harness Engineering.
What to Do This Week
Stop asking "which model is best" and start asking "which failure can I afford." The JetBrains data gives you the decision table. Review-constrained team? Avoid the high-redundancy profile. Merge-on-green-tests culture? The validation-skip rate is your actual defect rate, not the benchmark score. Cost-constrained batch work? A 27B model at 3% of cost with 12 fewer points may be strictly better economics.
Compute one new metric on your last 100 agent runs: validation rate. What fraction of merged patches were actually executed — tests run, command attempted — before the agent declared done? JetBrains found 23.6% of Opus runs skipped it, 68 of them still "passing." If you don't know your number, your resolve-rate dashboard is overcounting verified work.
Log trajectories, not outcomes. You don't need JetBrains' pipeline or an LLM judge to start. Repeated file reads, unchanged command retries, tool failure loops, diff size versus task scope — these are deterministic, cheap to derive from logs you already have, and they were the signals that separated four near-identical scores into four distinct products.
Rerun your model bake-off as a profile comparison. Take the two or three models that tied on your internal benchmark and diff them on hallucination rate, patch containment, trajectory length, and validation discipline over the same tasks. The tie will break. It always does — just never on the metric you were looking at.
Four-tenths of a point. One cent per run. And under that statistical tie, two entirely different engineers — one who diagnoses brilliantly and ships unverified, one who validates religiously and hallucinated in a third of its runs. The leaderboard can't see either of them. Your production system sees both, every day. The teams that start selecting on trajectories instead of scores are about to make the rest of the market look like it's betting on coin flips — because at the frontier, that's what a resolve-rate tie actually is.