The Science Theory of Deep Learning

It is argued that a scientific theory explaining deep learning is actually taking shape, and that this can be called learning mechanics.

The core lies in dealing with macroscopic statistics and quantitative predictions about the dynamics of the training process, hidden representations, final weights, and performance, rather than the details of individual models.

The authors lay out five pillars supporting this direction.

• Idealized interpretable settings: simple models that give intuition for real-world training
• Tractable limits: computable limits that reveal fundamental phenomena
• Simple mathematical laws: laws that explain important macroscopic observations
• Hyperparameter theory: reducing the complexity of training into a simpler framework by separating it out
• Universal behavior: common patterns that cut across systems and settings

What these studies have in common is that they deal with training dynamics, explain coarse aggregate statistics, and place importance on falsifiable numerical predictions.

The authors compare this perspective to mechanics in physics, and also discuss its relationship with statistical and information-theoretic approaches. In particular, they view it as potentially having a complementary relationship with mechanistic interpretability.

They also examine counterarguments such as "a fundamental theory is impossible" or "it doesn't matter," and conclude by laying out open problems in this field and directions for newcomers.