2.85x. That is the end-to-end inference speedup on Gemma 4 E2B running under vLLM on an A100 — from roughly 37,900 tokens per second to over 108,000 — with 100% token parity and zero quality drop, delivered in a handful of upstream pull requests that took days instead of months. It is also, as of October 13, a piece of marketing. That speedup is one of the headline exhibits in the launch post for Gemini 4 Argon, Google DeepMind's new frontier model.
Read the Argon announcement carefully and you notice what's missing. There is exactly one benchmark claim in the entire post: #1 on the Vals Index, which scores models across finance, coding, legal, and tax work. No SWE-bench table. No thirty-row leaderboard screenshot. The centerpiece is something else entirely — a list of merged pull requests. A Chromium refactor that untangled a ~6 million line legacy folder inside a 40 million line codebase, finishing months of work in weeks. An overnight autonomous run against Google's Agent Development Kit that cut in-memory session prep time by 40% and cold imports by 21%, now live in google/adk-python. A 2.3x speedup to xTB, one of the most-cited quantum chemistry packages in existence, contributed to the official repository.
The most consequential detail of this launch isn't a capability. It's a genre choice. The biggest AI lab on earth just told you, by example, that a merged diff is more persuasive than a benchmark score. They're right. And the implications run deeper than the launch post admits.
What Actually Shipped
The mechanics first, because they matter. Gemini 4 Argon went globally available on October 13 through the paid tier of the Gemini API in AI Studio and Android Studio, in Google Antigravity for AI Ultra subscribers, and in the Gemini app and AI Mode in Search for Ultra and Pro users. It carries an industry-leading 1M unified input-output token limit, with a new continuation protocol documented for developers — infrastructure plainly built for runs that outlive a single context window.
The pricing is a studied piece of aggression: $2 per million input tokens and $10 per million output tokens as an introductory rate, with cached input at 95% off. When the introductory period expires on November 30, it doubles to $4/$20 — which happens to be the exact sticker price Meta put on Muse Wonder one day earlier. The intro pricing isn't a discount; it's a customer-acquisition instrument timed to a competitor's launch window, and anyone architecting against Argon should model the post-November price, not the headline.
But the receipts are the story. Beyond the upstream contributions, Google reports internal effects from the past few weeks: agentic code submissions up more than 35% among teams using Argon in Antigravity, and code rollbacks reduced by nearly a third. And the ADK run deserves a second look from anyone building harnesses, because of how it was driven: the team handed Argon their .agents/skills rulebook and a one-paragraph goal — eliminate framework overhead without breaking existing apps — and let it work autonomously overnight. One paragraph of prompt. The harness carried the rest. The output survived review and landed in a production framework that powers enterprise agents.
Why You Trade a Leaderboard for a Git Log
The timing of this pivot is not a coincidence, because the benchmark regime collapsed within the last five weeks.
On September 1, vals.ai archived its independent SWE-bench Verified leaderboard and stopped running the benchmark on new models. Its final board tells you why: Claude Opus 5 at 97.00%, DeepSeek V4 Pro at 96.40%, GPT-5.6 Sol at 96.20%, a stack of models within fractions of a point and inside each other's error bars. A paired statistical test on the shared tasks can't separate the top eight submissions — the first significant break appears at rank nine. On October 10, OpenAI published an essay with the remarkable title "Why SWE-bench Verified no longer measures frontier coding capabilities" — the lab that curated the benchmark writing its obituary.
The successors are wobbling too. SWE-bench Pro, built specifically to resist contamination with 1,865 tasks across 41 repositories and a private commercial set, carries grading failures serious enough that Epoch AI designated the benchmark "Flawed" — audits found roughly 8.5% of trials accepting incorrect code and roughly 24% rejecting correct code. Call it one in three trials mis-graded. The decimal points on these leaderboards imply precision the underlying instruments no longer carry.
Into that vacuum, the merged diff walks in with three properties no public benchmark can match. It is adversarially validated — a maintainer who did not run your marketing department looked at the change and accepted it into a codebase they defend for a living. It is load-bearing — the vLLM optimizations are now shipped to every downstream user of the most widely deployed inference server in the field. And it is legible to buyers — a CTO who cannot interpret a McNemar test can absolutely interpret "Google's agent made our dependency 2.85x faster over the weekend."
Most people read Argon's launch as another entrant in the benchmark war. The actual news is that a lab with every incentive to quote scores — they hold a #1 ranking right now — chose to spend the body of their announcement on work product instead. When the party with the best numbers stops showing the numbers, believe them.
The Fine Print on Receipt Evidence
Before you retire your eval harness, though, read the receipts the way a maintainer reads a pull request: looking for what's hidden.
Survivorship bias is now the dominant failure mode. Nobody publishes the overnight runs that produced nothing mergeable. For every ADK optimization that landed, there are presumably abandoned branches, and the ratio is the actual capability number — one nobody has disclosed. The receipts you see are selected from a distribution whose size you don't know.
The internal numbers are self-graded. "35% more agentic code submissions" and "a third fewer rollbacks" are Google measuring Google, with no defined numerators, denominators, or control group. This is exactly the class of vendor claim that independent harness testing has spent two years knocking seven points off. Treat it as directionally interesting, not as data.
A receipt is not reproducible. You can re-run a benchmark. You cannot re-run the Chromium refactor. The evidence genre that replaced the leaderboard is unfalsifiable by construction — you either believe the anecdote or you don't, and the only hedge is to reproduce the workflow on your own codebase.
And notice the shape of the tasks chosen: an inference server with token-parity checks, a framework with a test suite, a chemistry package with benchmarks, a refactor with correctness gates. The receipts concentrate precisely where verification already exists. The agent proof-of-work is still, at bottom, "the tests passed" — the agent just moved the test run from a benchmark datacenter into someone else's repository, where a human co-signs the result. That is a real upgrade in evidence quality. It is not the abolition of benchmarks. It's benchmarks becoming local: your repo, your tests, your maintainer as the grader.
The Twenty-Hour Gap
There's a second axis opening, and Argon's architecture is built for it. FrontierSWE v2 — the ultra-long-horizon benchmark, 34 tasks spanning systems optimization, scientific computing, and applied ML research, each with a 20-hour budget and five trials in a purpose-built harness called Proximus — published results that should reset how you read model comparisons. Claude Fable 5.1 leads at 56.29%. GPT-5.6 Sol, statistically tied with the best models alive on SWE-bench Verified at 96.2%, scores 32.2% — a twenty-four point gap between models the saturated benchmarks cannot separate. The benchmark's own framing: it reveals very large differences between models whose performance is close everywhere else.
The token economics inside that leaderboard are equally instructive. GLM-5.3 burned 333 million tokens to score 30.2%; Sol scored higher on 182 million. On long horizons, token spend no longer predicts outcome — planning quality does. This is the regime where Argon's 1M unified token budget and continuation protocol are aimed: the model's shipping feature is not raw intelligence but the ability to stay coherent across a working day. Notably, on FrontierSWE v2's public board Argon sits at 55% — a hair behind Fable 5.1, and behind GPT-6 Astra at 65.5% — so the long-horizon crown is contested, which is precisely why Google is competing on a different evidence axis.
Put the three data points side by side: the short-horizon benchmark saturates and gets archived; the long-horizon benchmark spreads the field by two dozen points; and the launch evidence migrates to merged diffs in production repositories. These are not three stories. They are one story: the discriminating signal moved out of the synthetic benchmark and into real work, first as evaluation, now as marketing.
What to Do This Week
- Start an agent receipt log. Every time an agent's change survives human review and merges, record it: repo, diff size, verification method, who accepted it. Six months of that log is a capability benchmark for your own harness that no vendor can contest — and it's the only benchmark that predicts your actual outcomes.
- Demand numerators and denominators. When a vendor quotes "35% more submissions" or "a third fewer rollbacks," ask: measured against what baseline, over what period, with what control? The refusal to define the denominator is the tell.
- Price the horizon, not the token. Argon's $2/$10 doubles on November 30. Model your long-horizon workloads against $4/$20 — and against Muse Wonder's identical list price — before you commit architecture. The 95% cache discount matters more the longer the run: twenty-hour agents re-read the same context constantly.
- Build one long-horizon eval. You don't need 34 tasks. Take three pieces of work your team genuinely estimates at four-plus hours, define a scorer, and run your candidate models with a five-trial average. The spread you find will embarrass the leaderboard you were quoting.
- Reproduce a receipt instead of trusting it. Give your own agent a one-paragraph goal against a dependency you actually use, overnight, and see what survives review. That's the only falsification available for this evidence genre — and it's cheap.
The Proof Moved Into Production
The uncomfortable read of this week: the industry didn't solve the benchmark crisis, it routed around it. When scores stopped separating models, the evidence relocated to the one place selection pressure is real — production repositories, where changes earn their keep by being merged and surviving users. Evaluation didn't disappear; it got outsourced to maintainers who never agreed to be graders.
This is the endpoint of an argument I've been making since long before it was fashionable — that the only eval that matters is work, verified, in the environment where work happens; it's the through-line of Harness Engineering: The Complete Guide to Agent-First Software Development. What I didn't predict was that launch marketing would get there before most engineering orgs. Google now considers a merged pull request a stronger sales artifact than a #1 ranking. Your organization should learn the same lesson for a different purpose: the receipt log is becoming the way capability is demonstrated, and the teams who start keeping one now will have the only track record that can't be copied.
The leaderboard didn't die this week. It moved into the git log, one maintainer at a time.