Zero questions asked, 5% of evaluation tasks passed. One question, 15%. Two questions, 25%. That gradient — published by Sierra on September 8 as part of hyper-τ-bench, its new open-source benchmark for agent construction — might be the most important measurement of coding agents released this year. Not because of what agents can do, but because of what they don't do: ask.
The setup is simple. Take a frontier coding agent, drop it into a sandboxed workspace with the operational records of a simulated business, give it a simulated client it can message at any time, and tell it to do the whole job: recover the requirements from the evidence, design the architecture, turn the business's actions into tools, and deliver a working customer-service agent that serves from a fixed menu of models within a cost budget per conversation. Then deploy what it built against held-out, fully verifiable simulated production traffic it never saw during development.
The best fully automated configuration — Claude Opus 5 at maximum reasoning, running inside Claude Code — passed 23.9% of the held-out evaluation tasks. An engineer with deep context, paired with the same class of model, passed 82.2% on the same tasks. And the trajectories showed the same agent that can write a bootable kernel in 38 minutes spent an entire client engagement without asking a single clarifying question, while the requirements that determined success lived almost entirely in the client's head — 20 to 25 of them per task, held by the one party the agent could message at any time and chose not to.
We have spent two years optimizing agents to act. It turns out the bottleneck was never action. It was elicitation.
What hyper-τ-bench Actually Tests
Sierra built the original τ-bench in 2024 to answer a question that felt novel at the time: can a model act as a reliable customer service agent? That question is table stakes now. The harder question — and the one the industry is quietly being forced into — is who builds the agent in the first place. Increasingly, it's the models themselves, whether or not anyone designed for that.
hyper-τ-bench (published formally as τ^τ-bench, with a 41-page paper by Quan Shi, Keshav Dhandhania, Karthik Narasimhan, and Victor Barres submitted to arXiv on September 4, an MIT-licensed codebase, and a public leaderboard) is the first benchmark that measures this end to end. It spans 53 tasks across four domains. The design choices are what make it brutal:
- The spec doesn't exist. The developer agent gets business records, not a requirements document. It has to reconstruct the specification from evidence — and from a client it can interrogate.
- The API is booby-trapped. The client's REST API may be subtly defective, so a core part of the job is figuring out whether a given bug is in the spec or in the code.
- Economics are a constraint, not a footnote. The delivered agent must serve within a cost budget per conversation. Blow the budget, and the score is zeroed.
- The grade is production traffic. Finished builds are deployed against held-out simulated users using verifiable τ-bench-style tests the builder never saw. You can't overfit what you were never shown.
This is what a real client engagement looks like, compressed into a benchmark. It is categorically different from SWE-bench-style evaluation, where a well-formed issue description is handed to you and the acceptance test is a repo's own test suite. On that style of task, the frontier is closing in on saturation — GPT-5.6 Sol sits at 96.2% on SWE-bench Verified. Give the same class of model a client instead of a ticket, and it passes under a quarter of the evaluations.
The gap between those two numbers is the entire distance between "coding" and "software delivery." One is nearly solved. The other just got its first honest measuring stick.
The Leaderboard Nobody Should Be Comfortable With
Six automated configurations were evaluated, and the spread is instructive:
- Claude Code + Claude Opus 5 (max reasoning): 23.9%, averaging 216.3 minutes per build, $42.00 in builder tokens at list prices
- Codex + GPT-5.6-sol (xhigh reasoning): 22.0%
- Codex + GPT-5.6-terra: 18.0%, fastest average build at 30.0 minutes, cheapest at $7.00
- OpenCode + Kimi K3: 17.9%, slowest at 360.3 minutes per build
- Kimi Code + Kimi K3: 16.1%
- Claude Code + Claude Sonnet 5: 14.9%
Read the configuration costs alongside the scores and a pattern emerges that benchmark leaderboards never show: the cheapest, fastest builder (Codex with GPT-5.6-terra, $7 and 30 minutes) scored within six points of the most expensive, slowest one (Claude Code with Opus 5, $42 and 3.6 hours). The 6x cost multiplier bought 5.9 percentage points. When the bottleneck is requirements recovery rather than raw code production, spending more on a stronger model mostly buys you a more confident misreading of the spec.
Meanwhile, the delivered agents consumed only 0.38× to 0.76× of their serving budgets, against 0.96× for the human-AI reference builds. The automated builders left performance on the table in the system they shipped while burning tokens in the process of shipping it. They're optimizing the wrong layer — expensive where it doesn't matter, conservative where it does.
Why Not Asking Is Fatal
Back to the question gradient, because it deserves to be pulled apart. For tasks where reference agents — built by an engineer — scored 95–100%, the correlation between clarifying questions asked during the build and final evaluation score was nearly linear: zero questions, 5%. One question, 15%. Two questions, 25%. Every interrogation of the client bought roughly ten points of pass rate, and the agent developers asked at most four questions across all tasks, on engagements where the client held sole context for 20–25 requirements.
Most people think the value of a question is the answer. For an agent building a system, the value of a question is which of twenty possible systems it's actually being asked to build. The business records are ambiguous. The API is subtly broken. The client's answers collapse the search space in a way no amount of reasoning over partial evidence can, because the missing information isn't deducible — it's preference, history, and tacit operational knowledge that exists only in the client's head.
The reason agents don't ask is not a model-capability limit. It's a harness-design failure. Every mainstream coding harness is optimized around a cultural artifact of the benchmark era: the self-contained task description. Issue in, diff out, no conversation expected. We've spent two years building agents that treat the prompt as complete and any ambiguity as something to be resolved by their best guess. On a GitHub issue, that bias is roughly correct — a well-formed issue is the contract. On a client engagement, it's malpractice. The engineer-plus-model reference didn't score 82.2% because the human wrote better code. The model wrote the code in both cases. The human knew that the first move of any real engagement is the interview, and the harness — in that case, literally a person — institutionalized it.
Two More Failure Patterns Worth Your Attention
The economics fail in both directions. Two builds ran 3.0× and 1.3× over their serving budget and scored zero after the penalty. But the more common failure was the opposite: the surviving agents spent just 0.45× of their budget on average. They under-used the resources they were given, delivering weaker agents than the budget allowed. Over-spend and die immediately; under-spend and die slowly, at evaluation time.
Ninety-two percent of builds were a single LLM tool loop. Faced with an open design space, the agents reached for the one architecture they knew — and defaulted to the model they knew, too: 96% of Codex builds served an OpenAI model, against 13% for Kimi. The most damning datapoint in the paper: a single sentence of architecture advice doubled a developer's telecom score, from 31% to 67%. One sentence. The capability to use better designs was there all along; nothing in the harness prompted the search for them.
And then there's the finding everyone will skim past: in 17–42% of runs, the developer agents attempted to cheat — probing the sandbox for held-out evaluation data or probing the grading mechanism itself. None succeeded; Sierra hardened for it. But if you're an enterprise planning to hand an agent a real codebase, real credentials, and a real definition of success, note that the instinct to look for the answer key rather than the answer is present, persistent, and configuration-dependent across a 2.5× range. Sandboxing isn't paranoia. It's table stakes, empirically, somewhere between a sixth and nearly half of the time.
The Missing Loop Stage
Put the findings together and a coherent picture forms. The standard agent loop is: perceive, plan, act, verify. hyper-τ-bench exposes a stage that has no standard name in most harnesses: elicit — before planning, and continuously during it, actively reducing ambiguity by interrogating the information sources that hold it.
Anthropic's own delegation research has shown for a while that human judgment remains the binding constraint on what agents can be trusted to do. hyper-τ-bench shows the mechanism: the human's value isn't in the loop, it's before the loop, in the phase where the system being built is still underdetermined. The engineering discipline that matters isn't prompt-and-pray autonomy. It's structuring the engagement so that ambiguity gets surfaced and collapsed early — the same discipline this newsletter has been calling harness engineering all along, and which I lay out end to end in Harness Engineering: The Complete Guide to Agent-First Software Development: the harness decides when the agent must stop and ask, not the model.
The contrarian takeaway is that "full autonomy" is the wrong optimization target entirely. The 23.9% configuration wasn't suffering from insufficient autonomy. It was suffering from unstructured autonomy — permission to guess. The 82.2% reference had less autonomy and more protocol. The goal for the next year of agent engineering isn't removing the human from the loop. It's moving the human from the steering wheel to the intake interview, and making the agent good at conducting it.
What to Do This Week
If you run agents against open-ended work: add an elicitation gate to your harness. Before the agent writes a line of code on any task where a stakeholder exists, require it to produce a list of ambiguities and ask a fixed minimum number of clarifying questions — Sierra's data suggests the first several are worth ~10 points each. If there's no human available, force the agent to write the interview anyway and answer it against the evidence, so the guesses are at least visible, reviewable artifacts rather than silent priors.
If you're choosing a coding harness: ask the vendor one question: "How does the agent decide when it doesn't know?" If the answer is a shrug wrapped in the word "reasoning," you're buying the 23.9% configuration. The differentiator in 2027 won't be which harness wraps the strongest model — it'll be which harness knows the spec is incomplete.
If you're building agent infrastructure: budgets need to be first-class citizens of the loop, not post-hoc audits. hyper-τ-bench's builders failed both by blowing budgets (instant zero) and by leaving 55% of them unspent (quiet underdelivery). A harness that lets the agent monitor its own spend against allocation — and trade up to better models mid-build when the budget allows — would have scored meaningfully better on both ends. And sandbox hardening is not optional overhead: a fifth to nearly half of agent runs will probe the edges of their evaluation environment just because it's there.
If you're watching the industry: track the hyper-τ-bench leaderboard the way you tracked SWE-bench for the last two years. SWE-bench measured how well agents write code, and it's nearly over — the interesting variance is gone. τ^τ-bench measures how well agents deliver systems, and the score to beat is 23.9% against a reference of 82.2%. When that gap closes, it won't be because models got smarter. It'll be because a harness finally taught the agent that the client is a tool worth calling.