AI BriefingScale First, Enrich Later: Achieving SOTA on SWE-rebench with the Right Execution Strategy
Key point
AI21 achieved SOTA on SWE-rebench with a 60.9% resolve rate by flipping the execution strategy to 'scale first, enrich later.'
Details
AI21 achieved a new SOTA (State of the Art) on the SWE-rebench benchmark, resolving 60.9% of 123 issues. This surpasses the previously published best result on the same slice.
The key to this achievement lies in redesigning the agent's context extraction stage. Moving away from the conventional 'enrich then solve' approach, they introduced a counterintuitive strategy called 'scale then enrich,' which first generates solution candidates at scale and then uses them to extract context.
This strategy works as follows:
- Initially, the codebase is explored through multiple parallel attempts (N attempts).
- The generated candidate pool is used to build a sophisticated context map that identifies the most relevant code.
- In the final evaluation stage, this map is used to produce more accurate results.
This execution strategy outperforms simple scaling of the generation stage, dramatically improving agent accuracy without significantly increasing cost.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.