Evals Frameworks Driving the Next Stage of Enterprise AI

More than 1 million companies worldwide are adopting AI, but many organizations are struggling to achieve the outcomes they expected. To close this gap, OpenAI uses evals (evaluation frameworks) to measure and improve whether AI systems meet expectations.

OpenAI researchers use frontier evals to measure a model's overall performance, but to ensure performance in a specific business environment or workflow, contextual evals tailored to an organization's needs are essential.

The core process for effective evaluation is as follows.

• Specify: Technical and domain experts collaborate to define the AI's purpose and workflow. Success criteria are established for each stage, and a golden set—authoritative reference data reflecting expert judgment—is built.
• Measure: Performance is measured against the golden set in a test environment that simulates real conditions. Errors that occur during this process are analyzed to classify an error taxonomy.
• Improve: The system is iteratively improved through error analysis, which reviews the outputs of initial prototypes.

Evaluation should not be treated as a purely technical task; it must be a cross-functional process involving various departments such as product, sales, and HR to define business goals.