24%. That's the number now circulating through engineering leadership decks everywhere. Microsoft studied tens of thousands of its own engineers during their early-2026 rollout of Claude Code and GitHub Copilot CLI, and found that adopters merged roughly 24% more pull requests than they would have otherwise. The lift persisted across a four-month window.
The paper — published July 1 on arXiv by Emerson Murphy-Hill, Jenna Butler, and Alexandra Savelieva — is the first large-scale empirical study of command-line AI coding agents inside a major enterprise. It's rigorous, it's honest about its limitations, and it contains data points the vendor marketing will never surface.
Here's what nobody is telling you: 24% more merged PRs might not mean 24% more productivity. It might not mean any productivity at all. And the study's own data — if you read past the headline — reveals why measuring agent impact is the hardest unsolved problem in the field.
What the Study Actually Found
The paper, titled "Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI," covers Microsoft's rollout across tens of thousands of engineers. Three findings matter:
Adoption spread through social networks, not mandates. Engineers didn't start using Claude Code because their VP sent an email. They started because the engineer sitting next to them was using it. First use correlated with team proximity and peer influence — the same dynamics that spread any tool adoption inside large organizations. This is consistent with every developer tool adoption study going back decades, but it's the first confirmation that AI coding agents follow the same pattern.
Retention correlated with coding activity, not demographics. The engineers who kept using the tools weren't distinguished by age, role, team size, or seniority. They were distinguished by how much code they already wrote. High-activity developers adopted and retained at higher rates than low-activity developers. This flips a common assumption — that AI tools disproportionately help less experienced developers — on its head. The people who write the most code are the ones most motivated to offload it.
Adopters merged 24% more pull requests. This is the headline. The authors use merged PRs as a proxy for output and explicitly acknowledge the limitation: "a merged PR is not the same as the value it delivers." The 24% lift was measured over a four-month window and persisted throughout.
That qualifier — "a merged PR is not the same as the value it delivers" — is doing a lot of heavy lifting. It's the difference between a study that says "agents make developers more productive" and a study that says "agents make developers merge more pull requests." Those aren't the same claim.
Why PR Count Is a Broken Metric
The software engineering community has known for decades that counting lines of code is a terrible productivity metric. Frederick Brooks warned about it in The Mythical Man-Month in 1975. The industry collectively agreed: measuring output volume incentivizes output volume, not output quality.
Merged PR count is lines-of-code counting wearing a trench coat.
Here's the mechanism by which AI coding agents inflate PR count without inflating productivity. When a developer uses Claude Code or Copilot CLI, they tend to break work into smaller increments. A task that previously resulted in one 500-line PR now results in three 150-line PRs. The agent writes each one faster, the developer reviews and merges each one faster, and the PR count triples. The actual feature shipped is identical. The calendar time is similar. But the metric screams "200% productivity gain."
Multiple developers in the Hacker News discussion of the study confirmed this pattern from their own experience. One put it bluntly: "I've also seen an increase in merged PRs, but it coincided with developers opening smaller PRs. There's no actual increase in coding productivity, and it means the context-switching burden from review work went up."
That last point is the hidden cost. More PRs means more review cycles. More review cycles means more context switches for every reviewer on the team. If a team of ten engineers each merge 24% more PRs, that's 24% more review work distributed across the same ten people — unless the reviewers are also 24% faster at reviewing, which is a different claim entirely. The study measures the PR author's output. It doesn't measure the reviewer's burden.
The Token Economics Nobody Talks About
Buried in the paper's data is a figure that should make every CFO's eyes water. The study references token consumption at organizational scale — and one data point from the discussion reveals the magnitude. A single high-volume user consumed 281 billion tokens. At Claude Opus 4.6's listed pricing of $5 per million tokens, that one user's API spend could exceed $1.4 million.
Now, enterprise pricing is negotiated below list, and the study covers a rollout where Microsoft likely has favorable terms. But even at a fraction of list price, the token economics of organizational-scale coding agents are staggering. The paper explicitly notes: "At organizational scale, token spend can run into millions of dollars annually, so misreading adoption, retention, or impact can make a rollout expensive without changing engineering velocity."
This is the real finding of the study, and it's not in the headline. The question isn't "do coding agents make developers 24% more productive?" The question is "does a 24% increase in merged PR count justify millions of dollars in annual token spend?" And the study can't answer that — because merged PR count isn't a measure of economic value.
An engineer on the Hacker News thread made the point precisely: "more features doesn't mean anything if they don't translate into economic value. Refactoring PRs and dependency updates are easily automated now and it saves a lot of productivity, but if you compare the costs and economic value gained it might not actually justify enterprise token spending."
The METR Contradiction
If the Microsoft study weren't complicated enough, there's a prior study that paints a different picture entirely. In 2025, METR (Model Evaluation and Threat Research) ran a controlled experiment where developers estimated how long a task would take before knowing whether AI would be used. Tasks were then randomly assigned to "with AI" or "without AI" conditions, and actual completion times were compared against estimates.
The result: participants self-rated themselves as 20% more productive when using AI. Objective measurement showed they were 19% less productive.
A 39-percentage-point gap between perceived and measured productivity. Developers felt faster. They weren't. The tasks took longer.
The METR study and the Microsoft study aren't directly comparable — different methodologies, different populations, different timeframes. METR used controlled task assignments with experienced developers working on unfamiliar codebases. Microsoft measured organic adoption across tens of thousands of engineers on their own codebases. But together, they frame the range of uncertainty: somewhere between 19% slower and 24% more PRs, depending on how you measure and who you measure.
That range is enormous. And it exists because we don't have a reliable productivity metric for AI-assisted development. Not merged PRs. Not lines of code. Not self-reported satisfaction. Not task completion time. The field is measuring a phenomenon it can't define.
What the Adoption Pattern Actually Reveals
Here's the finding I find most strategically important, and it's not the 24% number. It's the adoption mechanism.
First use spread through social networks. Engineers adopted because peers adopted. This means that coding agent rollout is not a procurement problem — it's a culture problem. You can buy enterprise Claude Code licenses for every engineer on your team. That won't make them use it. What makes them use it is seeing a respected colleague ship faster with it.
This has a concrete operational implication. If you're rolling out coding agents across an organization, your highest-leverage investment isn't licensing or training — it's identifying your highest-activity developers (the study shows they adopt and retain at the highest rates) and making their success visible. Internal demos, shared workflows, MCP servers they've built — these are the vectors that spread adoption. Mandates and top-down training programs are the wrong tool.
The retention finding reinforces this. The engineers who stick with coding agents are the ones who already write a lot of code. They're not the ones who need the most help — they're the ones who have the most to offload. If your rollout strategy targets junior developers or low-activity contributors first, expecting the biggest gains, the data says you're betting against the pattern.
The Measurement Problem Is the Infrastructure Problem
The deeper issue — and the reason this matters for agent infrastructure specifically — is that our measurement tooling hasn't caught up with our agent tooling.
When an engineer uses Claude Code to implement a feature, the activity generates data: tokens consumed, files modified, tests run, build cycles, review iterations. That data exists. It's logged in the agent's telemetry, in the CI pipeline, in the git history. But nobody is correlating it. The agent harness knows how many tokens were spent. The CI system knows whether the build passed. The project management system knows when the feature shipped to production. The incident management system knows whether it caused an outage.
These data silos exist because the harness, the CI pipeline, the project tracker, and the incident system were built by different vendors at different times for different purposes. But the measurement gap in agent productivity exists precisely because nobody has built the integration layer that connects them.
This is a harness-engineering problem. The discipline of building the infrastructure that makes agent activity observable, attributable, and measurable — that's where the real work needs to happen. The model will get better. The agent will get smarter. But if we can't measure whether the output is actually valuable, we're flying blind at million-dollar altitudes.
Microsoft itself recognizes this. One of the study's authors presented Microsoft's internal framework for measuring developer productivity at BUILD 2026, and the ACM Queue published a detailed writeup of their approach. Their model goes beyond PR count — it attempts to tie engineering activity to customer-facing outcomes. It's a start, but it's Microsoft-internal. The industry needs an open standard.
What to Do This Week
If you're rolling out coding agents at your organization — or already have — here's how to act on this data without falling into the measurement trap:
- Stop using merged PR count as your north star metric. Track it if you want, but don't make decisions based on it. It's too easy to game and too disconnected from value. The number will go up. That doesn't mean productivity went up.
- Define what "productive" means for your team before you measure it. Is it features shipped to customers? Is it cycle time from issue to production? Is it defect rate? Is it customer satisfaction? Pick the metric that actually correlates with business outcomes — not the one that's easiest to collect from git.
- Track token spend per engineer and correlate it with output quality, not output volume. If one engineer is spending $50,000 in tokens and another is spending $5,000, and their feature output is comparable, you have an efficiency problem — not a productivity problem. Token attribution is a harness feature. If your agent doesn't expose it, you're operating without cost controls.
- Instrument your review pipeline. If PR count goes up 24% but review cycle time also goes up, you've shifted work from authors to reviewers — not eliminated it. Measure review turnaround time, review iteration count, and reviewer load distribution. The productivity gain has to be net-positive across the team, not just for the PR author.
- Identify your highest-activity developers and make their agent workflows visible to the rest of the team. The data says adoption spreads through peer influence. Your best adopters are your best distribution channel. Give them a platform to share what works.
The Real Headline
The Microsoft study is valuable. It's the first real dataset on organizational-scale coding agent adoption, and it confirms things practitioners already suspected: adoption is social, retention is driven by coding volume, and the output proxy goes up. These are genuine insights.
But the 24% number — the one that will appear in every vendor pitch deck and every engineering leadership presentation for the next year — is not the finding. The finding is that we still can't measure whether coding agents make organizations more productive. We can measure that they make organizations merge more pull requests. We can measure that they consume millions of dollars in tokens. We can measure that developers feel faster.
What we can't measure is whether any of that translates into shipping better software faster. And until we can, every productivity claim about AI coding agents — including the ones in this post — is a hypothesis dressed up as a data point.
The hardest part of agent-first development isn't building the infrastructure. It's knowing whether the infrastructure is working. That's the discipline I explored at length in Harness Engineering, and it's the gap the industry most urgently needs to close. Not with better models. With better measurement.
The 24% will be quoted. The measurement gap will be ignored. And that's exactly backwards.