Test-Time Compute for SWE-bench

AI21 Maestro is a general-purpose agentic framework that dynamically allocates test-time compute (TTC) during execution, substantially boosting performance on SWE-bench-verified. The core idea is to structure the orchestration itself rather than relying solely on LLM reasoning, so that computational resources are used efficiently on longer tasks.

On SWE-bench, Maestro was given only bash access, and compared under the same model and tool conditions as SWE-bench's ReAct-based mini-swe-agent. As a result, performance improved for both GPT-5 and GPT-5 mini, and GPT-5 paired with Maestro rose to the level of Claude 4.5 Opus and Gemini 3 Pro based on public results.

Maestro's effectiveness stems largely from three factors.

• Horizontal scaling: Even running a cheaper model multiple times in parallel can yield higher accuracy at lower cost, as long as the correct output is properly selected. For example, running gpt-5-mini across 8 trajectories can achieve a better score at lower cost than a single run of gpt-5.

• Maestro models the expected cost and value of each branch using pre-simulation learning priors or developer input, and automatically parallelizes only when it's worthwhile. When a branch terminates, a validator and reducer select the final candidate, and if one branch succeeds, the others can be stopped or a stuck agent can be cut off. It's designed so that even state-changing operations can be parallelized without conflicts.

• Structured plans: In conventional ReAct agents, planning, judgment, and execution are all mixed into token generation, so while it looks like a program, it's actually a stochastic process with large variance in cost, latency, and accuracy. Maestro expresses plans as code using a Plan Definition Language (PDL), achieving parallel execution, step-level TTC, explicit data/control flow, plan enforcement, Python-friendly generation, and security based on minimal syntax.

• Exploring the action space: Rather than a single 'optimal action,' Maestro explores the problem space by building a portfolio spanning different models, cost profiles, and planning approaches. The goal is to approach the cheapest execution path that succeeds for each task, compensating for the limitations of a single agent through orchestration.

Structured planning was evaluated on SWE-bench in two ways: one using a user-created plan resembling the mini-swe-agent prompt, and the other using a planner model generated at runtime for each item. Both approaches outperformed the ReAct agent using the same model, and the improvement margin grew larger as parallel execution was scaled up.

The execution traces show a clear difference. The ReAct agent often fell into a false "aha" moment, jumping into a patch early without sufficient understanding, then repeatedly failing after cycles of fixes and tests. In contrast, structured execution spent more compute building context and writing tests first, ultimately arriving at a solution.

Overall, this approach shifts the Pareto frontier not by making models larger or using more thinking tokens, but by better orchestrating existing capabilities. In other words, it redefines TTC not simply as "thinking more," but as a problem of structurally allocating resources and paths during execution.