How Far Behind Are Open Models?

The performance gap between open models (weight-available models) and closed models (API-only models) was analyzed using data from 17 benchmarks.

The analysis found that on private benchmarks, where the data is not publicly available, open models lag behind closed models by about 8 to 10 months, while on public benchmarks, the gap is about 4 to 6 months. Looking at the trend of the gap, it was narrowest around January 2025, when DeepSeek R1 was released, but has been widening again since then.

Key points to consider when interpreting the gap are as follows:

• Benchmark bias: Public benchmarks tend to underestimate the actual gap by about 2x.
• Potential performance degradation: When using open models through third-party providers, implementation issues can cause slight performance degradation, which may make the measured gap appear larger than it actually is.
• Real-world task gap: Since model developers are likely to train toward benchmark data, the gap that occurs in real-world tasks is estimated to be much larger than the measured figures.