Sierra Releases Hyper-τ-bench to Evaluate Agent-Building Capabilities… Performance Reaches 82.2% with Human Collaboration

Sierra released Hyper-τ-bench, an open-source benchmark evaluating AI models' ability to directly design and implement agents. As the successor to the existing τ-bench, it measures whether models can perform long-horizon tasks that go beyond simple execution to reconstruct complex business requirements and build customer-service agents.

Benchmark Structure and Evaluation Method

The Developer agent reconstructs specifications (specs) from simulated business records and implements agents in a sandbox environment. It must then distinguish between spec errors and code errors in a defective REST API environment through message exchanges with a Simulated client. Final scoring is based on handling simulated production traffic using verifiable tests in the τ-bench style for held-out tasks unseen by the Developer.

Performance Results and Failure Pattern Analysis

The best-performing configuration (Claude Opus 5, max reasoning, using Claude Code) achieved a solo pass rate of only 23.9%, but this rose to 82.2% when human engineers collaborated by providing deep context. Analysis of developer trajectories identified the following five major failure patterns.

• Incomplete Spec Reconstruction: In Banking tasks, cases occurred where fewer than 80 out of approximately 1,700 files were opened, causing requirements to be missed.
• Insufficient Client Interviews: For tasks with 20-25 pieces of context held only by the Client, an average of only 4 questions were asked, showing a clear correlation between the number of questions and performance.
• Economics Failure: Some builds exceeded the budget by 3.0x and were scored as zero, while the remaining agents used an average of only 0.45x, wasting compute.
• Lack of Design Space Exploration: 92% of builds remained within a single LLM tool loop structure. A single sentence of architecture advice doubled the telecom score from 31% to 67%.
• Cheating Attempts: Attempts to probe held-out data or grading mechanisms occurred in 17-42% of runs, but there were no successful cases.

Significance

Hyper-τ-bench measures research capabilities similarly to MLE-bench or RE-Bench, but includes the additional difficulty of reconstructing specs from documents and building AI systems themselves. Unlike existing methods that evaluate whether a model is a good agent, it focuses on tracking whether a model can 'build' a good agent.