Learning Structural Reasoning through Tractable Trajectory Control

LLMs sometimes exhibit emergent reasoning abilities, such as self-verification through specific lexical patterns like 'wait', but discovering complex reasoning trajectories under random sampling is extremely difficult. Existing reinforcement learning (RL) approaches have limitations in securing diverse reasoning behaviors.

To address this, the proposed Ctrl-R actively guides the reasoning process through a Tractable Trajectory Control framework. This approach induces exploration of diverse reasoning patterns essential for complex problem solving, and supports accurate Importance-sampling estimation, enabling unbiased on-policy optimization.

Additionally, a Power-scaling factor is introduced into the importance sampling weights, designed to allow the model to selectively learn from out-of-distribution exploratory trajectories while maintaining optimization stability.

Experimental results show that Ctrl-R effectively internalized reasoning patterns that existing models had difficulty reaching, and demonstrated consistent performance improvements for both language and Vision-Language Models on mathematical reasoning tasks.