Agent-Based Development at Copilot Applied Science

Analyzing massive amounts of agent trajectories demands enormous intellectual labor from AI researchers. This is because manually reviewing hundreds of thousands of lines of JSON data generated when analyzing benchmarks like TerminalBench2 or SWEBench-Pro is virtually impossible.

To solve this, eval-agents was created to automate repetitive analysis tasks. Developers built an agent-based coding environment combining Copilot CLI, Claude Opus 4.6, and VSCode to maximize development speed. In particular, by leveraging the Copilot SDK, they accelerated the agent creation process by immediately using existing tools and MCP servers.

The three core strategies for efficient agent utilization are as follows:

• Prompting strategy: Make detailed requests conversationally, and use planning mode before running agent mode.
• Architecture strategy: Perform frequent refactoring, documentation updates, and code cleanup in parallel.
• Iteration strategy: Under the principle of "trust but verify," focus on improving the process rather than the agent.

Through this approach, in just 3 days, a team of 5 members added 11 new agents and 4 skills, achieving remarkable productivity by writing a total of 28,858 lines of code.