Apple Releases 'Agent Seer' to Automatically Generate AI Agent Evaluation Scenarios Using Only MCP Specifications

Apple researchers have released Agent Seer, a pipeline that automatically synthesizes realistic test scenarios for evaluating AI agents that use external tools, relying solely on Model Context Protocol (MCP) specifications.

Previous approaches relied on manual authoring by domain experts, resulting in low scalability and an inability to reflect evolving APIs. Agent Seer leverages semantic information inherent in specifications—such as function names, natural language descriptions, and typed parameter schemas—to generate scenarios without examples or real-time tool access.

Key Features and Results

• Automated Scenario Generation: It enriches the original schema, generates step-by-step scenarios including synthetic tool outputs, and then expands these into multi-turn conversations based on simulated data.
• High Accuracy: Tests across seven diverse domain MCP specifications showed excellent tool call accuracy and conversational consistency, achieving complete tool coverage for small and medium-sized specifications.
• Analysis of Quality Impact Factors: Parameter schema complexity is the most significant correlate of quality variation, while tool set size has a relatively smaller impact. Additionally, 'argument value accuracy', which is not visible through rough name-matching metrics, was identified as a primary failure factor in incomplete scenarios.