Auto-BenchMax Released, Doubling MCP Score

To address the shortage of training data for tool-calling agents, a data synthesis pipeline named Auto-BenchMax has been released, which precisely mimics the structural distribution of benchmarks. It highlights the issue where simply simplifying and mass-generating tasks, as in previous approaches, leads to performance degradation due to structural mismatch with the original benchmark, and proposes a methodology to resolve this.

Core Principle: Structural Distribution Matching

Auto-BenchMax operates on the principle of minimizing the difference between the original benchmark and synthetic data by comparing structural histograms before and after generation. Specifically, it distinguishes between two synthesis paths based on the benchmark's scoring method.

• Rule-based Scoring (Method B): Constructs the correct answer in advance, verifies it through an evaluator, and then adopts it as the task.
• Judge Model Scoring (Method A): Extracts facts from the answers of strong models executed in real environments to use as the correct answer.

Performance Improvement

When performing a single iteration of training based on the Qwen3-Coder-30B-A3B model, the MCP-Atlas public set score rose significantly from 19.1 to 40.4, and the communication domain score in Tau2-bench rose from 35.96 to 85.09. The authors anticipate that the MCP-Atlas score can be raised above 50 through additional iterations.

Usage and Limitations

This project provides procedures and code for analyzing benchmark scoring code to select an appropriate synthesis path, rather than targeting a specific dataset. However, reproduction requires high computational resources, such as a single node with 8 GPUs and DeepSpeed ZeRO-3 configuration, and caution is required for commercial use due to the absence of a LICENSE file in the repository.