GLM 5.2 Reduces Tool-call Failure Rate by 21.2% with Real User Data-based RFT and Hint-guided Self-distillation

GLM 5.2 applied a new training methodology combining Rejection Sampling Fine-tuning (RFT) and Hint-guided Self-distillation to bridge the gap between synthetic environments and real-world usage. To overcome the limitations of existing RFT, which only learned from successful sessions—thereby reinforcing intermediate errors and discarding learning signals from failed sessions—a new pipeline was introduced that utilizes both successful and failed data from real user sessions.

Training Methodology: Hint-guided Self-distillation

The proposed method processes non-error turns in successful sessions via Imitation (CE Loss), and turns with errors that have valid Hints via Correction (KL Loss). A Hint is defined as a short corrective instruction based on information the model already possessed at the time of the mistake, enabling On-Policy Self-Distillation (OPSD). The Teacher model receives the Hint to generate the Target distribution, while the Student model is trained to mimic the Teacher's distribution without receiving the Hint.

Evaluation Results and Performance Improvements

Performance was verified through offline benchmarks and live user sessions (A/B testing).

• Hint Effect Verification: When Hints were provided, the explicit evidence-based correction rate improved significantly from 40.0% to 75.0%, and the Tool-error avoidance rate improved from 75.1% to 93.7%.
• Offline Tool Error Rate: Error rates decreased in the RFT-only (1.35%) and RFT-plus-OPSD (0.87%) checkpoints compared to Stock GLM 5.2 (2.79%).
• Live Usage (A/B Test): The subsequent checkpoint (Checkpoint 2) applying RFT-plus-OPSD reduced the Tool-call failure rate by 21.2% compared to the previous checkpoint (Checkpoint 1). However, the reduction in user complaints (MID+ rate) was not statistically significant.

Limitations and Implications

This method trains at the 'decision' level rather than the entire session, using successful sessions as anchors for behavior to preserve, and complaints and Tool errors to identify avoidable failures. Currently, the strongest evidence is limited to improvements in Tool Reliability, with the limitation that single-Turn correction does not guarantee overall Task success. The ultimate goal is to create an Agent that requires less correction, rather than an Agent that recovers after correction.