Dynamic Data Snoozing

In GRPO, examples where all rollouts receive the same reward end up with advantage equal to 0, providing no training signal. If such examples are processed as-is, even with dynamic sampling or replacement strategies, generation costs simply increase, and many batches end up not actually being used for training, greatly slowing down training.

To reduce this, offline difficulty stratification is first applied. Based on the completion success rates of several baseline models, the data is divided by difficulty, and a curriculum is created by deferring examples that are too hard while removing or reweighting examples that are too easy at the appropriate time.

Next, dynamic difficulty filtering is used. Assuming rewards are in the 0~1 range, if all rollouts within a group receive a reward of 1, the example is judged to be too easy and is filtered out; even if it reappears later, it continues to be filtered, saving on rollout generation costs. This approach showed roughly 3X compute efficiency improvement on GSM8K, while final test scores remained similarly around 84%.

However, problems arise in multi-task settings.

• Some datasets get filtered out faster than others, which can lead to task starvation.
• For data with conflicting objectives, such as safety, grounded question-answering, and open-ended question-answering, mode collapse toward one behavior can occur.
• In actual experiments, as the answerable / non-answerable ratio became skewed toward one side, a phenomenon was observed where performance on the answerable subset declined.

To address this problem, dynamic data snoozing is proposed. Instead of permanently removing overly easy examples, they are excluded only temporarily by a set snoozing factor, then reconsidered later. This maintains efficiency while, over time, reincorporating previously easy examples into training, preventing the data distribution from collapsing toward one side.

In experiments, as the snoozing factor increased, the temporary performance drop became larger, but it was recoverable afterward. On GSM8K, aggressive dynamic filtering showed a 3X efficiency improvement, while dynamic data snoozing still provided a 2X compute gain.

In practice, this can be applied as follows.

• If datasets are relatively independent, use a larger snoozing factor to save more computation.
• If there is significant interaction between datasets, set it low, such as 1~2 epochs, to prioritize stability.
• A stochastic approach, snoozing with a probability of 1/N each epoch, is also possible and can serve as an alternative to deterministic snoozing.
• Furthermore, this can be extended to dynamic example weighting, which dynamically adjusts the sampling weights not just of easy examples but of all examples.

The key point is that as the most expensive part of online RL increasingly becomes generation cost, data efficiency comes to determine training efficiency. Dynamic data snoozing is a simple solution that captures both training stability and compute efficiency by appropriately resting easy examples rather than discarding them outright.