2.1 million tokens. That is the average consumption of a single SWE-bench run by a frontier agent — one task, one attempt, start to finish — according to the first large-scale public dataset of real agent runs, released by Hugging Face on October 4. Two point one million tokens in, one benchmark score out.
Think about what that ratio means. The entire industry argues about agent capabilities using a number that compresses a multi-hour, multi-megabyte stochastic process into a single percentage. The cost, the detours, the moments where the agent nearly gave up, the recoveries — all of it discarded at scoring time. We have been benchmarking the destination and burning the map. The dataset is called Exgentic Agent LLM Traces: 10,000 real runs and 242,000 model calls from five leading models, run through up to five different agent designs, across six kinds of real work. Every model call preserved in one standard format. It is 236 MB, and it might be the most consequential 236 MB released this quarter.
Most people think the industry's measurement problem is that benchmarks are too easy or too gameable. The actual problem is more basic: a score is a compression, and we compressed away every variable that determines whether an agent is deployable. This is the first public artifact that lets anyone decompress.
What Actually Shipped
The release, from Hugging Face's Lena Dankin and Michal Shmueli-Scheuer, is deliberately unglamorous: no model, no leaderboard, no product. Just runs. Ten thousand of them, filtered down from a raw corpus of 10,500 runs and 626,000 model calls — roughly two-thirds of the raw calls were removed on purpose, because they were scaffolding around the agent rather than the agent itself. What survived is the model-facing spine of each run.
The coverage is what makes it valuable. Five frontier models in comparable volume: DeepSeek-V3.2 (2,300 runs), Kimi-K2.5 (2,300), GPT-5.2 (2,100), Claude Opus 4.5 (1,900), and Gemini 3 Pro (1,400). Each run through up to five different agent designs. Six task domains spanning the actual deployed spectrum of agent work: software engineering (SWE-bench), deep research (BrowseComp Plus), personal-task automation across hundreds of apps (AppWorld), and policy-bound customer service and technical support across three flavors of the τ²-bench airline, retail, and telecom domains.
And the format is the quiet headline. Every call is written in the OpenTelemetry GenAI conventions — the open standard that a growing part of the ecosystem already emits. Each call records the models used, the full input and output messages, the tools that were available (not just the ones that got called), token usage, why the call stopped, and an error type when it failed. Each run carries its context: which benchmark, which agent design, the score, the cost, the number of steps. Because every run speaks the same format, the runs line up across models and harnesses without per-tool reconciliation. That last property — comparability — is the one the field has never had.
Why did it take a neutral party to do this? Because producing real agent runs at scale means running frontier models across many environments, thousands of times over, and paying for every token. A single lab could afford it but won't publish it; a single tool vendor's traces only cover its own dialect. Hugging Face ran the meter for everyone and posted the receipt.
The Ten-X Spread Nobody Budgets For
Here is the table the release buried in the middle, and it deserves to be the lead:
- Software engineering (SWE-bench): 2,000 runs, 2.1M average tokens per run
- Personal-task automation (AppWorld): 1,500 runs, 1.7M average tokens per run
- Deep research (BrowseComp Plus): 1,900 runs, 1.6M average tokens per run
- Technical support (τ²-bench Telecom): 1,800 runs, 332K average tokens per run
- Airline customer service (τ²-bench Airline): 1,000 runs, 255K average tokens per run
- Retail customer service (τ²-bench Retail): 1,800 runs, 198K average tokens per run
Read the spread, not the absolutes. The same label — "an agent run" — covers workloads that differ by more than an order of magnitude. A coding agent is a ten-times heavier animal than a retail-support agent, per attempt. At frontier list rates, 2.1 million tokens is dollars per run, not cents — before a human ever reads a line of the output, and multiplied by every retry, every parallel session, every failed attempt your eval pipeline quietly absorbs.
This single table retires several arguments at once. It retires "how much does an agent cost to run?" — the question was malformed; the answer is "which domain?" It explains why enterprise token bills tripled while per-token prices collapsed: the workloads being attempted migrated up this table, from autocomplete-adjacent support tasks toward multi-million-token engineering work, and nobody's dashboard drew the distinction. And it exposes the unit economics of every agent product pitched on a flat "per-task" price: whether that price survives contact with customers depends entirely on which side of the 10x spread their tasks land on.
What a Score Throws Away
The deeper problem the dataset illuminates is structural. A benchmark score answers exactly one question: did the run end in success? It is silent on everything that determines whether you'd deploy the thing.
Where does the agent give up? Does it recover when something breaks — a failed tool call, an erroring step, a context that overflowed mid-task? What does the successful path cost versus the failed one? How many steps before the model stopped making progress while still emitting confident output? None of these are score-visible. All of them are now trace-visible, at scale, in a comparable format, for the first time.
Prior public traces existed, but each was a narrow window. SWE-agent-trajectories covered GitHub issues through one framework. KernelBench hard traces covered GPU kernel coding. Some datasets captured only the conversation visible to the user; others embedded tool invocations in free text, or omitted which tools were even available — which makes the trajectory uninterpretable, because you cannot evaluate a choice without knowing the option set. The noise floor was also unmanaged: without deliberate filtering, recorded "runs" are larded with framework chatter that isn't the agent at all. Hence the two-thirds cull in this release. Clean, uniform, broad: pick any two used to be the rule. Now all three exist in one 236 MB download.
There is a second-order effect worth sitting with. When the same task, in the same format, exists for five models across five agent designs, you can finally ask questions that are neither model questions nor harness questions but interaction questions: which model-harness pairings burn tokens on loops, which ones front-load context efficiently, which agent designs make every model placed inside them more economical. The harness-variance argument — same weights, different scaffolding, double-digit swings — has been made repeatedly with individual studies. This corpus turns it from an anecdote collection into a systematically mineable property.
Traces Are Now Load, Not Just Lore
Here is the part with immediate operational consequences, and it is already happening: a team working on LLM inference infrastructure is using these traces to benchmark serving configurations under realistic agentic load — the workload that synthetic benchmarks systematically miss.
The tool is inference-perf, a Kubernetes SIG benchmarking project that replays OpenTelemetry traces against a live endpoint. Because the Exgentic traces are already in OTel format, they plug in without conversion. Out comes latency, cache hit rates, and cost measured under real agent traffic — not synthetic token streams, not chat-shaped assumptions.
If you operate inference infrastructure, understand what just changed. Agent traffic has a peculiar shape: long prefixes re-sent on every turn as context accumulates, tool results injected mid-conversation, bursts of parallel sessions. Whether your serving stack's prefix cache hits or misses on that shape can swing effective cost by multiples — and until now, there was no public, replayable representation of the shape to test against. Teams tuned their stacks against the traffic they could synthesize. Now there is a canonical corpus of the traffic that actually occurs. Expect every serious inference vendor to be replaying this within a quarter, whether or not they announce it.
The same move works at personal scale. Run the replay against the local stack on your own hardware — the MLX or llama.cpp endpoint you've been telling yourself is "basically ready" for agentic work — and watch where latency and cache behavior fall apart under a real 2.1M-token SWE-bench run's arrival pattern. That is a more honest readiness assessment than any synthetic benchmark, and it takes an afternoon.
The Format Is the Moat That Isn't a Moat
Yesterday's framing on this site was that environments — the verified task worlds agents train against — are the compounding asset in the RL era. This dataset completes the picture from the other side. Environments are where agents practice. Traces are what agents did. One is the training-side capital; the other is inference-side capital, and both survive model upgrades — a corpus of recorded behavior from last quarter's models remains valid measurement material for characterizing this quarter's infrastructure, the way last year's web traffic logs still calibrate this year's load balancers.
The unlock is the standard. The release's own framing is exactly right: because every run speaks the same format, these traces accumulate into something comparable instead of fragmenting into one dialect per tool. The value compounds as more are added. An OTel-formatted trace corpus is non-rival, vendor-neutral, and mergeable across organizations in a way no proprietary telemetry ever is. If your team emits OTel GenAI today, your internal runs are already dialect-compatible with a growing public corpus — which means your internal analysis tooling, your replay harnesses, and your fine-tuning pipelines can consume both without adapters.
That is also why this will not stay a one-off. The obvious sequels — traces from coding-agent CLIs in daily developer use, from enterprise deployments, from local-first stacks — all become additive the moment they land in the same format. The first mover's advantage here isn't exclusivity; it's convention-setting. Whoever publishes the second and third large trace corpora in OTel inherits the framing this release established.
The Honest Caveats
Three limits, stated plainly. First, the corpus is benchmark-shaped: runs are attempts at defined tasks with scores attached, not open-ended production sessions where the goal itself drifts. Production behavior — the user interjecting, the task changing mid-run — is richer than this. Second, five models is breadth, not coverage; frontier release cadence being what it is, the model list is a snapshot of a particular quarter. Third, filtered corpora encode their filter: the two-thirds of calls removed as scaffolding were removed by a judgment call, and someone building on the data inherits it. None of these caveats blunt the release. They define its next version.
What to Do This Week
Download the dataset and read three runs end to end. Not skim — read. One SWE-bench run, one τ²-bench retail run, one BrowseComp Plus run. Watch where token mass accumulates, where the agent retries, what the context window looks like at the moment of failure versus success. Three runs will recalibrate your intuition for what "an agent run" physically is more than any blog post, including this one.
Emit OTel GenAI from your harness, starting now. If your agent infrastructure records traces in a bespoke format, you are choosing dialect over language. The standard exists, the tooling exists, and every week of bespoke telemetry is a week of traces that can never be compared, replayed, or merged with anything outside your walls. Instrumenting the loop so that every call is captured — not just the final answer — is half the discipline of harness engineering, and it is now also half the value.
Compute your own tokens-per-run distribution and place it against the reference table. Pull your last month of agent sessions, bucket by task type, and calculate average tokens per completed run. Then compare against the published domain averages. If your "simple" internal tasks are averaging like SWE-bench runs, you have found a harness problem masquerading as a cost problem — and you found it with arithmetic, not vendor claims.
Replay before you buy. If you are evaluating any serving change — new vendor, new cache configuration, a local stack you intend to self-host — replay a slice of this corpus against the candidate before committing. Latency and cache hit rate under real agentic load is the number that determines your actual cost. Synthetic benchmarks will not surface it. These traces will.
Stop accepting scores without a bill. From this week forward, any internal model evaluation that reports a success rate without a tokens-per-run figure is incomplete in a way you can now name publicly, with a reference distribution to point at. The score tells you what the agent did. Only the trace tells you what it cost to get there — and the organizations that internalize that distinction first will pick models, harnesses, and infrastructure with an accuracy their competitors can't copy, because their competitors are still reading leaderboards.
Every benchmark score is a claim. Ten thousand runs, in one format, with the receipts attached, just became public. The era of taking the claim on faith is over.