Reasoning Models Struggle to Control Their Chain-of-Thought (CoT), Which Is Actually Good News for AI Safety

As AI agents perform increasingly complex tasks, reliably overseeing their behavior is becoming critical. OpenAI leverages Chain-of-Thought (CoT) monitoring for safety, a technique that analyzes the reasoning steps an agent generates to determine whether it is safe.

If an agent becomes aware of the monitoring and can intentionally distort or hide its reasoning process (CoT controllability), there is a risk of overestimating the system's safety. To study this, the researchers introduced an open-source evaluation suite called CoT-Control, which measures how well models follow instructions about their reasoning process, such as avoiding certain keywords or changing the format.

The results showed that current frontier reasoning models struggle significantly with controlling their CoT.

• Models' CoT Controllability scores were measured to be very low, ranging from 0.1% to 15.4%.
• While controllability increased with larger model size, it actually decreased as the reasoning process got longer or as models underwent additional post-training.

In conclusion, current reasoning models do not yet possess enough control to disable CoT monitoring, which is a fortunate result for ensuring AI safety. However, continued evaluation is necessary as models continue to advance.