Nathan Lambert argues Artificial Analysis v5 benchmarks favor closed models over open-weight alternatives

Details

Artificial Analysis released Intelligence Index v5, which weights Knowledge Work, Coding, Science, and General categories equally at 25% each to reflect real-world usage.
The new ranking places 8 closed models ahead of the best open-weight model, Kimi K3.1 Pro, suggesting the benchmark measures frontier capabilities where open models currently lag.
Nathan Lambert argues that while open models are competitive in coding, they fall behind in science evaluations like Terminal-Bench-Science and CritPt v2, which are heavily weighted in the new index.
The shift to private benchmarks in v5 is defensible against overfitting but reduces transparency, creating a trade-off between accuracy and community verification.
Lambert notes that major closed labs like Anthropic and OpenAI have largely stopped optimizing for these public indices, focusing instead on internal R&D and enterprise customers, while smaller labs remain dependent on them for branding.

Key point