92.8% and 27.3%. Same model, same launch, same company's own comparison table — a 65.5-point spread between two numbers that are both real. The first is SWE-2's score on Terminal-Bench 2.1, and it led nearly every headline about Cognition's launch on September 10: the highest score in the table, ahead of Claude Fable 5.1 at 91.4% and GPT-6 Astra at 89.9%. The second is SWE-2's score on Terminal-Bench 4, where Fable 5.1 sits at 55.8% and Astra at 57.9%. Here is the detail that reframes both: Terminal-Bench 4 is the current edition of the benchmark. 2.1 is the superseded one. The row where SWE-2 leads is retired. The row where it trails by 28 points is live.
I noted this model in passing earlier this week as a long-horizon caveat inside a bigger story. It deserves the full treatment, because what happened here is bigger than one model having a weak spot. SWE-2 is the cleanest demonstration yet of a pattern that is quietly restructuring how model launches work: benchmark version skew has become a launch strategy. When suites saturate faster than they can be replaced, every release inherits a menu of benchmarks ranging from comfortably solved to genuinely open — and which ones lead the press release is a marketing decision, not a measurement one.
Most coverage read SWE-2 as "frontier coding at 64% off." The more accurate summary, derived entirely from Cognition's own published table, is stranger: it's an open-weight base model plus roughly five points, distributed evenly across everything, sold only inside one harness. Let's run the arithmetic.
What Actually Shipped
The mechanics first, because they matter for everything that follows. SWE-2 is Cognition's new coding model for Devin, post-trained from Moonshot AI's Kimi K3 — the 2.8-trillion-parameter open-weight MoE with 104 billion active parameters per token. It is available in Devin Desktop and Devin CLI, with rollout underway on Devin Web and Fusion. There are no published weights and no per-token API. There is no documented way to point your own harness at this model. If you want SWE-2, you use it inside Devin, and the cost figures Cognition quotes — 64% cheaper than Fable 5.1, roughly a quarter of Astra's cost — are mean dollars per rollout on their benchmarks, not token prices. The invoice bundles model, harness, scaffolding, and serving into one number that cannot be compared against anyone else's rate card.
The genuinely impressive part of the release is not any percentage. It's the trajectory efficiency. Cognition's prior model, SWE-1.7, had a documented habit of reading a codebase deeply before touching anything: median first real code edit at step 48. SWE-2 at medium effort makes that edit at step 18. Mean steps per run on FrontierCode 1.1 Main fall from 127 to 53. On that same benchmark, SWE-2 medium beats SWE-1.7's score while taking 58% fewer turns and costing 81% less. That came from a reinforcement learning setup that penalized wasted spend — training medium, high, and max effort tiers in a single run, with cost penalties tuned to the local slope of Kimi K3's Pareto frontier. This is exactly the kind of optimization agent teams should want more of: the model was explicitly trained to stop burning tokens on exploration theater.
But the efficiency gains and the capability claims sit in the same table, and the table has a shape nobody seems to have looked at.
The Arithmetic Nobody Ran
Line SWE-2's scores up against its own base model and something almost mechanical falls out:
- FrontierCode 1.1 Main: SWE-2 at 50.0% versus Kimi K3 at 44.2% — +5.8
- DeepSWE 1.1: 73.0% versus 68.5% — +4.5
- Terminal-Bench 2.1: 92.8% versus 88.3% — +4.5
- Terminal-Bench 4: 27.3% versus 21.5% — +5.8
Four benchmarks. Four gaps. All between 4.5 and 5.8 points. The post-training moved the level, not the shape. Wherever Kimi K3 is strong, SWE-2 is strong plus about five. Wherever K3 is weak — Terminal-Bench 4 being the clear case — SWE-2 is weak plus about five. That is a remarkably consistent result for a fine-tune, and it hands you a practical evaluation shortcut: look up the base model's score on whatever workload you actually run, add five points, and you have a better forecast than the launch headlines will give you.
It also sharpens the Terminal-Bench 4 problem from "soft spot" into structural fact. Yes, everyone drops when a benchmark gets harder. The interesting part is by how much, and the fall is not proportional. Going from Terminal-Bench 2.1 to 4, Claude Fable 5.1 loses about 35.6 points and GPT-6 Astra about 32.0. SWE-2 loses 65.5. Its base model loses 66.8. On long-horizon agentic work, SWE-2 does not merely sit below the frontier models — it degrades roughly twice as fast as they do when the task runs long. The post-training that reliably added five points everywhere could not touch the degradation curve at all.
Degradation Rate Is the New Capability Metric
For agent workloads, the degradation rate matters more than any single score, and here is the contrarian claim of this post: most model selection today optimizes the level and ignores the slope. A model's benchmark score is a snapshot of performance at the horizon length the benchmark happened to be written for. But an agent in production does not live at one horizon. It lives on a distribution of horizons, and the long tail of that distribution is where your costs, your failures, and your 3 a.m. pages come from.
Think about what a 2x degradation rate means mechanically. If model A holds capability twice as long as model B before falling off, then for any task at the edge of B's horizon, A is the difference between completion and a half-finished diff. Worse, degradation compounds inside an agent loop: a model that starts failing at step N produces errors that a supervising pass then tries to fix, extending the run into the region where it fails harder. The failure isn't just a miss — it's a miss that generates more failing work. This is why "it scored 92% on the short suite" is nearly worthless information for deciding what to put at the center of a long-running harness, and why the 27.3% is the only number in SWE-2's table you should plan against.
The same week made the point from a completely different direction. On September 9, the Rails team published Stage 2 of Agents on Rails, a benchmark that asks models to ship real-world features on a real application — 20 feature tickets, 90 minutes and $60 per run, generous by any standard. Stage 1, with small atomic tasks, was nearly solved: two models hit 92%. Stage 2's best result: GPT-6 Astra at 35%, solving 21 of 60 runs. A small model that was Stage 1's most cost-efficient darling — GPT-5.6 Luna, which completed 46 of 63 atomic tasks for 90 cents — completed 0 of 60 feature tickets. The Rails team's diagnosis is worth quoting in substance: at atomic scale you can ship a model's work without checking; features require high-level planning, and a model that can't do that doesn't ship — it submits half-done jobs and skips the test suite entirely.
Two benchmarks, two teams, one message: the gap between "knows the framework" and "can ship the feature" is not closing this year, and the models that look best on yesterday's suites are not the ones that survive today's. Terminal-Bench 2.1 and Agents on Rails Stage 1 measure the same thing — short-horizon competence. Terminal-Bench 4 and Stage 2 measure what your agent actually does for a living.
Why the Retired Benchmark Won the News Cycle
It is tempting to call the coverage lazy, but the incentives run deeper than that. Terminal-Bench 2.1 is saturated — when the field spans 88 to 93%, the suite is no longer discriminating between frontier and non-frontier models; it is a backdrop. A saturated benchmark is nonetheless still useful to a launcher, for exactly the wrong reason: on a saturated suite, small differences read as decisive wins. "Top of the table, ahead of Anthropic and OpenAI" is a sentence you can only write on a benchmark everyone has nearly solved. On the live suite, the same table reads "trails by 28 points." Both sentences are true. One of them travels.
There is a second, subtler asymmetry. FrontierCode — where SWE-2's 50.0% sits within a point of Fable 5.1's 50.9% — is Cognition's own benchmark. It is, to be fair, a serious piece of evaluation design: 20-plus open-source maintainers, more than 40 hours per task, each maintainer defining what "mergeable" means in their own repo. It is also a house instrument, run by the company whose model tops it, with comparisons executed inside Cognition's own Devin CLI. None of that makes the numbers wrong. All of it belongs in the sentence where you repeat them. The strongest independent signal in the entire table is DeepSWE 1.1 — produced by Datacurve, not Cognition — where SWE-2's 73.0% genuinely beats Fable 5.1's 67.4% and sits just behind Astra's 74.1%. That is a real result. It is also the fourth-most-quoted number of the launch.
The pattern generalizes. As benchmark suites age, we will see more launches led by numbers from superseded or self-owned suites, with the live and independent numbers further down the page. The defense is not cynicism — it is reading tables instead of headlines. Three questions decode any launch table in under a minute: Is this benchmark the current edition? Who built and ran it? And what does the base model score — because post-training, as SWE-2 just demonstrated with almost mathematical neatness, tends to add a constant, not a correction.
The Lock-In You Can't Benchmark
One more layer, because it connects to the structural story of this moment. SWE-2 ships with no weights and no API — Devin only. That is not a temporary gap in the rollout; it is the product. Cognition trained a model whose headline advantage is trajectory efficiency inside a specific harness, and the only way to buy the efficiency is to buy the harness. The cost claims don't decompose, by design: "64% cheaper than Fable 5.1" describes dollars per rollout inside Devin, a figure that bundles model, harness, and serving into one opaque ratio.
This is the same consolidation pressure I wrote about when OpenAI made the Codex harness free, arriving from the opposite direction. OpenAI's play: commoditize the harness, capture the sessions. Cognition's play: fuse model and harness into a single purchasable object, so the model comparison and the harness comparison can no longer be separated. In both cases, the layer you thought you were choosing independently — the model — is being welded to a runtime someone else owns. SWE-2 may be the more honest version of the two, because at least it doesn't present the weld as a gift. But anyone assembling an agent stack this quarter should notice that "which model" is quietly becoming "whose runtime," and price the switching cost accordingly.
The evaluation discipline this demands is the one thing that hasn't commoditized. Your own harness, run against your own repository, measuring cost per solved task at your horizon distribution — that remains the only decision instrument that can't be gamed by a launch table. It is also, not coincidentally, the core practice of harness engineering as a discipline: treat the model as a swappable component and the evaluation loop as the asset.
What to Do This Week
- Compute your degradation slope. Take the model at the center of your harness and run the same task family at 2x, 5x, and 10x the horizon. Plot solve rate against horizon length. That curve — not any single benchmark score — tells you where your agent's wall is, and whether the model you just read about would move it.
- Check the edition date on every number in your decision docs. If your team's model-selection memo cites a benchmark, verify it's the current edition, note who runs it, and look up the base model's score if the candidate is a post-train. The plus-a-constant pattern means the base model's weaknesses are inherited, not fixed.
- Test the feature-ticket layer, not the atomic layer. The Agents on Rails Stage 2 corpus and protocol are public, with raw data in their ai-evals repo. Run your current default model on a handful of real feature tickets from your own backlog with a 90-minute budget. Most teams doing this for the first time discover their effective solve rate is nowhere near what their short-task evals implied.
- Refuse benchmarks that can't leave the vendor. A model you can't point your own harness at can't be evaluated on your terms — only on the vendor's. That's acceptable for a product you buy as a product. It's disqualifying for a component you were planning to compare.
SWE-2 is a genuinely interesting release: a serious cost-penalty training technique, real trajectory efficiency, and a fine-tune so consistent it accidentally published its own fingerprint. But the launch's most durable lesson isn't about Cognition at all. It's that in a market where suites saturate in months, the gap between the current benchmark and the retired one is where launches go to hide — and the engineers who read tables instead of headlines are the only ones who will notice. The 92.8% was real. So was the fact that it stopped counting.