DABstep Benchmark for Data Agents Released

DABstep, developed jointly by Adyen and Hugging Face, is a new benchmark for evaluating the performance of AI agents in the field of data analysis.

DABstep goes beyond simple math or code problems, reflecting the complex workflows that real data analysts face. Its key features are as follows:

• Practice-focused tasks: It includes over 450 tasks drawn from Adyen's actual operations, measuring practical data analysis capability.
• Combining structured and unstructured data: It requires the ability to handle unstructured data such as complex documents alongside structured data such as databases.
• Objective evaluation: It uses a binary outcome method that leaves no room for interpretation, increasing the objectivity of evaluation.
• Ease of use: Without complex setup, models can be tested and results submitted to the leaderboard as long as a code execution environment is available.

Even the best reasoning-based agents currently achieve only about 16% accuracy on DABstep, showing that data analysis agent technology is still in its early stages.