Microsoft establishes first standardized evaluation baseline for Copilot using synthetic enterprise data

Details

Microsoft developed the Industry Evaluation Data Generation Platform to create the first standardized evaluation baseline for Microsoft Copilot, addressing the need for complex test data without using sensitive customer information.
Synthetic enterprise environments grounded in public sources (O*NET, job postings, SEC filings) achieved a 39% improvement in composite quality scores, with some industry-specific environments scoring above 0.9 out of 1.0.
An automated six-stage pipeline generates industry-specific queries, reducing dataset creation from multi-day manual efforts to automated runs and saving an estimated $300,000 in total project costs.
The system simulates dynamic, time-evolving organizations across pharmaceutical, capital markets, and IT/software verticals to test Copilot against realistic, domain-specific workflows.
Privacy and safety are built-in by design, ensuring all data is fully synthetic and screened for harmful or personally identifiable material, allowing teams to reuse datasets without privacy review overhead.
The platform is published to an internal shared catalog for cross-team reuse, enabling reproducible regression testing and benchmarking for Microsoft Copilot.

Key point