Better and cheaper together: Open models explore, frontier models patch

AI21 analyzed seven weeks of internal developer traffic and found that only 11% of turns involve pure code generation, with most interactions being clarifying questions or context sharing. In contrast, public benchmarks like SWE-bench focus almost exclusively on code generation and bug fixing. To address this, AI21 built a 613-task benchmark reflecting real traffic patterns, including short messages and mid-session context. When testing their intelligent Gateway, which routes requests between cheap and premium models based on difficulty, the team found that public benchmarks caused the Gateway to over-route to premium models (nearly 50% vs a 25% target). Their new benchmark aligned closely with live traffic, landing within 0.5 points of the target, demonstrating that public benchmarks overestimate task difficulty and mislead model selection.