Epoch AI Defines InnovationEval Benchmark to Test AI's Ability to Discover Novel Post-Training Algorithms

Epoch AI has introduced InnovationEval, a benchmark designed to assess whether AI can automate end-to-end machine learning research by discovering true algorithmic innovations. The task requires AI agents to develop a new post-training method that surpasses a strong GRPO baseline, with the result needing to be strong enough for an excellent paper and novel relative to the field at the start of 2026.

Benchmark Structure and Constraints

The evaluation constrains the scope to ensure that success requires genuine novelty rather than simple optimization or data manipulation. Agents are restricted to making algorithmic changes that affect the loss function, its updates, and/or rollouts within the verl/ directory. Infrastructure, training data, and hyperparameter tuning (beyond specific constraints) are fixed or prohibited to prevent cheating.

• Scope: Modifications are limited to algorithmic changes in loss updates and rollout logic. Infrastructure components like the rollout engine, sharding, and reward manager are fixed.
• Resources: Agents operate in an internet-blocked sandbox using Modal GPU execution. The task involves two areas (question-answering and code), each contributing 50% to the grade.
• Scoring: The goal is to match or exceed the performance of the original SDPO paper (reference) while beating the shipped GRPO baseline. Scores are based on held-out test results at specific training-time marks (1h and 5h in reference seconds).

Uncertainty and Future Outlook

The source evidence indicates that it is currently uncertain when future models will be able to independently discover a meaningful AI algorithmic innovation. It notes that leading models from a year ago would have fared significantly worse, suggesting rapid capability advancement. Epoch AI plans to periodically rerun similar evaluations for newer models, refreshing the task to account for memorization, to track progress toward automated AI R&D.