AI BriefingNeurIPS 2025 E2LM Competition Held
Key point
A NeurIPS 2025 competition is being held to build a new benchmark that evaluates scientific knowledge and reasoning ability in the early training stages of LLMs.
Details
In the early stages of LLM development (under approximately 200B tokens), existing benchmarks fail to provide meaningful signals, making it difficult to interpret experimental results. The NeurIPS 2025 E2LM Competition aims to build a new benchmark that can effectively capture early training signals of LLMs in the scientific knowledge domain.
Participants submit their solutions using the lm-evaluation-harness library, and evaluation is based on the following three metrics:
- Signal Quality score (SQ): measures discriminative power in early training stages
- Rank Consistency score (RC): stability of rankings across models
- Compliance with Scientific knowledge score (CS): alignment with scientific facts
The competition began on July 14, 2025, and final results will be announced in November. Total prizes are available, including $6,000 for 1st place, along with a student track prize.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Taste-Bench: Frontier LLMs Achieve 59.7% Accuracy on Decision Forking Tasks
Toss Builds 'Toss Benchmark' to Bridge Gap Between Public Leaderboards and Real-World Performance
DeepSeek V4.1 Flash Successfully Exploits All 11 Vulnerabilities in AI Hacking Benchmark
GPT-5.6 Luna finds 75% of verified bugs at 3.6% of GPT-6 Astra's cost, but with lower precision