DeepSWE points out errors in coding model leaderboards

The DeepSWE benchmark released by Datacurve revealed that major existing coding model leaderboards have been misgrading to a significant extent.

According to this audit, leaderboards currently trusted in the industry fail to accurately reflect models' actual performance, suggesting a risk that companies may make wrong decisions when selecting and adopting models.

Key points:

• DeepSWE precisely verifies the performance of frontier coding models.
• It points out the misgrading problem in existing leaderboards, raising questions about their reliability.
• It provides guidance for corporate technology decision-makers to be cautious when evaluating and adopting models.