Senior SWE-Bench: An Open-Source Benchmark That Evaluates Agents as Senior Engineers

While existing AI coding evaluations have remained at a junior level with clear requirements, Senior SWE-Bench measures AI agents' real problem-solving abilities by providing high-difficulty tasks similar to actual work environments.

The key features are as follows:

• Realistic Instructions: Instead of overly detailed requirements, it tests agents' autonomous judgment through ambiguous instructions in the form of natural language messages.
• Runtime Investigation: Beyond simple code fixes, it evaluates the ability to track down complex bugs that occur during execution, such as log analysis, checking profiling data, and performing reproduction steps.
• Tasteful Solves: Beyond code that simply works, it verifies whether the solution follows existing codebase conventions and whether the code quality is appropriate, using a Validation Agent and quality metrics.

This benchmark focuses on reproducing the workflow of a senior engineer, where the agent defines the problem itself and finds the optimal solution within a complex system.