When Should AI Tutors Intervene?

Allen Institute for AI released TutorMoments, a replay-based framework that evaluates an AI tutor's ability to decide whether to immediately assist a student or wait to let them think for themselves, based on real math tutoring conversations.

• Utilized 462 one-on-one math tutoring conversations with US students in grades 2–7.
• 27 teachers marked over 1,500 critical judgment moments and wrote free-text annotations.
• Conversations up to the decision point are provided to an LLM tutor, while another LLM simulates the student's role.

Evaluation results showed that when models receive only the instruction to "teach well," they tend to provide excessive assistance with the solution process and answers rather than promoting student thinking. While explicitly specifying the balance between when to help and when to step back in the prompt improves performance, it still falls short of the level of human teachers who consistently respond appropriately to student situations.

The research team released the de-identified TutorMoments-Preview dataset, the replay pipeline code, and reproduction results of the evaluated model tutors.