AI BriefingScaling Domain Data Repetition in LLM Pre-training
Key point
We confirmed that the optimal number of domain data repetitions increases with model size during LLM scaling.
Details
As LLMs scale, the training token budget must increase to maintain an appropriate tokens-per-parameter (TPP) ratio. However, high-quality domain data is less scalable than general web data, leading to a dilution effect where its proportion in the training mix decreases as model size grows.
Repeating existing high-quality data is effective in offsetting this dilution, but excessive repetition can cause overfitting. The research team analyzed this trade-off in real-world LLM scaling environments.
Key findings are as follows:
- Model Size and Repetition Count: Under a fixed TPP condition, the optimal number of repetitions slightly increases as model size grows.
- Domain-specific Correlation: The optimal repetition count shows a strong negative correlation with the final validation loss of the respective domain. Domains with lower loss benefit from more repetitions.
- Relationship with Data Volume: The amount of unique domain data shows only a weak relationship with the optimal repetition count.
These results suggest that repetition counts tuned on smaller proxy models with the same TPP can serve as practical estimates for larger models.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
LLM Mathematical Excellence Driven by Pre-training Data Quality, Not Verifiability