Jane Street Research Discovers Non-Monotonic Patterns in Sequence Weight Scaling

Jane Street researchers published findings showing that the scaling laws of sequence weighting in large language model (LLM) training exhibit unpredictable (non-monotonic) patterns. This raises questions about existing scaling law extrapolation methods and suggests caution when establishing data mixing strategies.

Non-Monotonic Scaling Phenomenon

The researchers conducted experiments on their own dense/MoE models and the open-weight Qwen 2.5. As a result, they confirmed that the effective sequence weight exponent ($p^*$) changes with model size as follows:

• Small models: Low exponent (learning general patterns regardless of data weighting)
• Medium models: High exponent (learning specific patterns proportional to data weighting)
• Large models: Low exponent again (learning all patterns regardless of weighting)

This rise-then-fall pattern is analyzed as being due to the model learning general and idiosyncratic patterns in the data at different rates. Additionally, it was observed that as the number of training epochs increases, this peak shifts toward smaller model sizes.

Practical Implications

Sequence weighting behavior in small models does not clearly predict behavior in large models. Therefore, the following countermeasures are recommended when developing large models:

• Evaluate and extrapolate data mix results only on sufficiently large models
• Adjust training weights to correct for observed aberrant scaling

This study points out the limitations of existing extrapolation models such as the Chinchilla scaling law, emphasizing the importance of analyzing the causes of prediction bias in addition to curve fitting in scaling research.