Evaluating Large-Scale Multi-Agent Systems

Failures in agent systems don't occur simply because of one wrong response, but arise from compound causes such as incorrect handoffs, missed signals by specialist agents, and improper review triggers. To improve this, we need to identify recurring behaviors across the entire set of traces, beyond individual responses.

This workflow proposes a Macro-eval approach for Multi-Agent systems. Using an EV (electric vehicle) order workflow as an example, it simulates an environment where various specialist agents—pricing, compliance, supply, and factory routing—collaborate.

Evaluation is divided into two levels:

• Lower-level evals: Evaluate the quality of individual agents, tool use, and handoffs. In the example, Promptfoo is used to measure policy compliance and the appropriateness of specialist routing.
• Macro evals: Synthesize the results of numerous lower-level evaluations to analyze which problems recur and where they concentrate.

Through this process, thousands of agent events can be compressed into a small number of behavior patterns that both technical and business stakeholders can understand, allowing the core problem areas of the system to be identified.