Introducing On-policy distillation (OPD) technology

On-policy distillation (OPD) is a core technique used in the post-training of the latest high-performance models such as Qwen 3.6/3.7, GLM-5.1, and DeepSeek-V4.

The existing approach had a limitation: when a model made an error during the reasoning process (rollout), it had to rely solely on noisy reward signals spread across the entire trajectory. OPD addresses this in the following way.

• Identifying the error point: A separate model reads the reasoning trajectory and pinpoints the exact location where the error occurred.
• Inserting hint tokens: Hint tokens are inserted right above the point where the error occurred.
• Efficient training: Only a forward pass is performed, without a new decode process. The inserted hints guide the model to assign a low probability to the error token, and the model learns this probability distribution, effectively reducing specific mistakes.