Scaling Laws for Mixture Pretraining Under Data Constraints

As language models scale, the required data volume increases, yet some data sources, such as low-resource languages or specialized domains, are inherently limited in size. Strategies that mix such sparse data with abundant general data entail a fundamental trade-off between insufficient exposure to the target domain and overfitting due to excessive repetition.

Apple researchers analyzed this trade-off through over 2,000 language model training experiments. The results revealed that repetition is a key driver of target domain performance, and that mixture training allows for significantly higher repetition rates than single-source training. Sparse target corpora can be reused up to 15–20 times, with the optimal number of repetitions varying depending on target data size, compute budget, and model scale.

The researchers introduced a repetition-aware mixture scaling law. This law accounts for the diminishing value of repeated target tokens and the regularization role of general data, providing a principled method to compute practical mixture configurations for pretraining under data constraints.