AI BriefingCode Generation Benchmark BigCodeBench Released
Key point
BigCodeBench, a new LLM code generation benchmark for evaluating practical programming ability, has been released.
Details
The existing HumanEval consists of simple, algorithm-centric tasks that fail to reflect real-world software development environments, and due to data contamination and overfitting issues, it has limitations in evaluating the generalization ability of LLMs.
To address this, the newly released BigCodeBench focuses on precisely evaluating the practical programming capabilities of LLMs.
Key Features:
- Massive task scale: Includes 1,140 function-level tasks utilizing 139 libraries.
- Practice-oriented design: Provides complex instructions from a user's perspective without step-by-step guidance, requiring the ability to solve problems by combining various functions.
- Rigorous verification: Thoroughly verifies program behavior through test cases with an average branch coverage of 99%.
- Reliable evaluation: Introduces the calibrated Pass@1 method to measure performance, compensating for LLMs' tendency to omit long code.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Step 5 Preview Released: 600B Parameter MoE Model Highlights Agentic Tasks and Financial Analysis PerformanceQwen3.5-based Small Decision Model 'Kev' Released with TypeSafe API Compatibility and Performance Metrics