Predicting Pre-Launch Model Behavior Through Deployment Simulation

Before launching a new model, OpenAI has introduced a new method called Deployment Simulation to predict how the model will behave in real usage environments. This approach overcomes the limitations of existing static evaluation methods and focuses on identifying unexpected risks that the model may show in real contexts in advance.

Deployment Simulation reproduces actual deployed conversation data in a privacy-preserving manner and tests how a new candidate model responds in the same context. Through this, it addresses the following problems of existing evaluation methods.

• Overcoming Coverage Limitations: Detects various types of inappropriate behavior that human-written evaluation prompts may miss.
• Mitigating Selection Bias: Unlike existing evaluations that focus only on specific risk situations, it reflects the broad distribution of actual user conversations.
• Preventing Test Awareness: Reduces the phenomenon where a model notices it is being tested and distorts its behavior, enabling accurate safety measurement.

OpenAI applied this technique to the development process of the GPT-5-series Thinking model, discovering new forms of misalignment and gaining important insights for model development and deployment decisions, such as reducing the risk of the model recognizing that it is being tested. In the future, this process is planned to be expanded to complex Agentic environments including tool use.