Clean traces made the difference

To build a small, fast multi-turn tool-calling model, the team tested an approach of training directly on production traces as-is.

The experiment was conducted on Google Research's Schema Guided Dialogue (SGD) dataset, specifically in the restaurant reservation domain. There were 3 tools—FindRestaurants, ReserveRestaurant, and respond_to_user—and the student model was Qwen3-1.7B + LoRA rank 64.

When trained on refined, human-annotated data, the results were excellent.

• Student model Qwen3-1.7B: 0.866 LLM-as-a-judge score
• Comparison model GLM-5 (744B): 0.835
• Qwen3-235B: 0.768
• GPT-OSS-120B: 0.765
• DeepSeek-3.2: 0.744

In other words, the 1.7B model scored 3 points higher than its 744B teacher, confirming that with good data, even small models can be trained to be powerful at task-specific tool-calling.

The problem arose when real production-style noise was mixed in. The authors report that as soon as realistic traces were introduced, direct training performance dropped by 14–28 percentage points, and they lay out which failure modes broke the model and what actually fixed it.