Reassessing the Data Moat

A discussion between Ryan Greenblatt and Shuchao Bi re-evaluates the causes of AI model performance gains. While data volume and quality were previously considered the core competitive advantage, both experts argue that algorithmic progress is a greater driver of recent AI advancements.

Greenblatt analyzed that retraining past models with the same compute resources can yield performance close to current top-tier models. This suggests that improvements in model architecture and training methods have had a larger impact than the increase in data itself.

Bi also pointed out that raw data does not have an optimal distribution, and changing the data distribution to equalize intelligence per token contributes to improvements in scaling laws. Greenblatt emphasized that a scientific approach to data curation and filtering processes, rather than human expert data, is key to improving the quality of pre-training data.