Guide to Reproducing DeepSeek R1's 'Aha Moment'

This covers the technical method for reproducing the "aha moment" shown by DeepSeek R1 (a phenomenon in which a model spontaneously increases its thinking time and re-evaluates its strategy, growing its reasoning ability) in a small model.

By using a number puzzle called the Countdown Game, the model is guided to learn self-verification and exploration abilities on its own through reinforcement learning (RL) alone, without any separate human feedback.

The core algorithm, GRPO (Group Relative Policy Optimization), unlike the existing PPO, estimates a baseline through scores within a group without a Value Function model, thereby reducing memory usage and computational cost.

• Sampling: Generate multiple outputs per prompt.
• Reward Scoring: Assign rewards based on rules or outcomes.
• Advantage Calculation: Calculate and normalize the Advantage based on the average reward within the group.
• Policy Optimization: Optimize the policy using the calculated advantage and KL divergence.

This guide includes setting up a distributed training environment using DeepSpeed and vLLM, and provides a case of execution in a 4x NVIDIA H100 GPU environment.