Scaling agentic evaluation for SWE-bench

Applying AI21 Maestro to SWE-bench Verified involved running more than 200,000 evaluations, and in the process it became clear that agentic evaluation is not simply model testing but an infrastructure problem. Long executions, branching, state preservation, and multi-seed repetition all overlapped, making it hard to achieve statistical reliability with the existing short, linear evaluation approach.

There were three core challenges. First, the throughput wall meant thousands of multistep sessions had to run concurrently. Second, since agents mutate the filesystem and environment state, isolation was essential. Third, since a 2-hour run failing at the 99% mark would waste enormous tokens and compute, resumability was needed.

At first, the local-centric assumptions of SWE-bench were carried over directly into Kubernetes, but limitations soon appeared. Without a Hugging Face cache per pod, the same files were repeatedly downloaded, triggering 429 rate limit errors, and the structure of spinning up docker run again inside a pod didn't work properly in terms of permissions, overhead, and resource management.

In the second attempt, the existing Docker-based evaluation was kept as much as possible, while changing it so a new container was created for every run inside a Kubernetes pod. But running two variants against about 500 instances with duplicity repeated at levels of 4, 8, 16 meant creating and deleting about 16,000 Docker containers within a single evaluation window, and resource contention along with external rate limits degraded both speed and stability.

The turning point came from reexamining what was actually different between each run and what could be shared. As a result, the evaluation environment was redesigned into a multi-tenant simulation environment, spinning up about 500 pods per instance just once and then sharing the following resources across subsequent runs.

• repository: checked out at the correct commit
• MCP server: ready to execute commands
• installed dependencies: retained across runs

This was made possible by extending the MCP protocol and building a separate MCP client for isolation, so that multiple runs against the same SWE-bench instance could run sequentially or in parallel without conflicts. As a result, repo download and pod provisioning costs dropped significantly, and the failure rate also fell noticeably.

Execution time is now determined by variant complexity and parallelism rather than infrastructure overhead. Lightweight variants take 3.5 minutes on average, normal variants take 10 minutes, and heavy variants can take over 2 hours, but since up to 8K parallel runs are now supported, a full SWE-bench evaluation for a typical variant finishes in about 20 minutes.

Also, because generation and evaluation are kept separate, patch generation results are preserved even if the evaluation stage fails. Even if a test runner times out or a pod is evicted, there's no need to regenerate the patch—only the evaluation needs to be rerun—and results can be analyzed as soon as 80% of the total completes.

Ultimately, the plan is to provide a 10,000 parallel runs SLA across all evaluations, and to scale it further by over 10x by running multiple executors on the same pod. This would allow handling 35,000 isolated evaluations per day without spinning up 100,000 containers, enabling agents to be iteratively improved with data and speed the way models are.