PROOF-Gen: Better Distillation from Optimized Data

When distilling tool-calling capabilities into deployable models, existing methods used a 'generate-and-filter' mechanism that retained only successful trajectories generated by the teacher and discarded failures. This caused failure signals to disappear from the same difficult scenarios every time.

To address this, Apple introduced PROOF-Gen (Per-scenario Reflective Optimization to Overcome Failed Generation). This method has a reflector analyze execution traces and evaluation feedback for each failed scenario to write correction instructions that guide the teacher to generate successful trajectories. During training, these instructions are removed, allowing the student model to learn from clean demonstrations without task-specific scaffolding.

On the τ²-bench benchmark, PROOF-Gen recovered 93% of failed scenarios. This improved the Pass^1 score of Qwen3-4B-Instruct-2507 from 0.132 to 0.529, and Gemma 4 E4B-it saw a 7.2%p increase on BFCL v4 multi-turn.

In actual deployment pipelines, the goal completion rate improved by 6.3%p, and positive transfer effects were observed for on-device models. An average performance improvement of 1.48%p was also confirmed for non-English languages.