Improving Best-of-N Performance via Budget-Aware Execution for SWE Agents

Existing SWE agent experiments have applied a consistent compute budget under the assumption that all tasks have the same difficulty. However, actual task difficulty follows a heavy-tailed distribution, and a significant number of tasks can be solved with little effort.

For example, about 50% of tasks in the SWE-rebench benchmark were solved with just a single GPT-5.2 rollout, yet the existing policy consumed 5 rollouts for every task, incurring unnecessary costs.

In response, AI21 presents budget-aware execution strategies that adaptively allocate compute resources according to task difficulty. The key strategies are as follows:

• Cascading with early stopping: If a task is solved early, subsequent rollouts are stopped, reducing the cost of the final rollout.
• Parallel execution with early stopping: Optimizes speed by reducing the wait time for the final rollout through parallel execution.

Through these strategies, it is possible to find the optimal balance between cost and speed while maintaining quality.