Revisiting Scaling Properties of Downstream Metrics in LLM Training

LLM scaling has so far mostly focused on proxy metrics like pretraining loss, and predicting downstream task performance has been considered unstable. This study proposes a direct framework that models benchmark performance directly from the training budget, and shows that at a fixed token-to-parameter ratio, the log accuracy of various downstream tasks is well explained by a simple power law.

The direct approach showed better extrapolation performance than the existing two-stage procedure. The existing method tends to accumulate intermediate prediction errors, but this new approach reduces such issues by estimating performance directly from the training scale.

The authors also presented a functional form that predicts accuracy across token-to-parameter ratios, as well as an equation that accounts for inference compute through repeated sampling. Experiments were validated on models up to 17B parameters, up to 350B tokens, and across two dataset mixtures.

For reproducibility, the full pretraining loss and downstream evaluation results were also released. This paper was accepted to the Workshop on Memory for LLM-Based Agentic Systems at ICLR.