LiveCodeBench Leaderboard Released

LiveCodeBench, developed by researchers at UC Berkeley, MIT, and Cornell, has been released. This benchmark measures LLMs' code generation ability using problems from coding platforms such as LeetCode, AtCoder, and CodeForces.

To prevent data contamination, a major problem with existing benchmarks, it records the release date of each problem and introduces a 'scrolling over time' approach that verifies performance using problems released after the model's training data cutoff.

Evaluation is conducted from multiple angles through the following four scenarios:

• Code Generation: Generating correct code based on problem descriptions
• Self Repair: Evaluating the ability to fix code through error feedback
• Code Execution: Predicting execution results for given code and inputs
• Test Output Prediction: Generating expected output using only the problem description and input values

According to the performance test results, GPT-4-Turbo showed the best performance in most scenarios, while Claude-3-Opus excelled at test output prediction and Mistral-Large excelled at tasks based on natural language reasoning.