How to Handle Scaling Laws Carefully

Scaling laws are one of the core empirical findings in deep learning, describing the principle that as model size (N), dataset size (D), and compute (C) increase, training loss (L) decreases in a predictable power-law curve.

This law serves as a framework for deciding how to optimally allocate limited compute resources between model size and data volume. In practice, it is used by deriving scaling laws from small-scale experiments and then using them to estimate the number of tokens and the amount of compute needed to build large-scale models.

The predictability of generalization error, which underlies scaling laws, has been studied since the past. Amari et al. (1992) used a Bayesian approach to derive various forms of learning curves depending on whether the data contains noise and on the characteristics of the learning algorithm.

According to the early research of Hestness et al. (2017), the following common patterns were observed across various deep learning fields such as neural machine translation and image classification.

• Generalization error follows a power law with respect to several factors, including data size
• Even as model performance improves, the error curve merely shifts, while the exponent of the power law itself does not change
• Changes in architecture affect the offset of the error (E), but do not affect the exponent