A Scientific Theory of Deep Learning Will Emerge

This summarizes the trend toward treating the internal workings of deep learning as a single scientific theory. The core is learning mechanics, which views learning as a dynamical system created by parameters, data, tasks, and learning rules.

This perspective sees the hard problem of deep learning as lying not in simple opacity but in complexity. Neural networks have non-convex, over-parameterized structures, and since learning is a process of forming internal representations, existing classical theory alone is not sufficient.

The piece presents the following as conditions such a theory should satisfy.

• Start from first principles
• Speak quantitatively
• Be predictable through experiments
• Explain training, representation, and final weights together
• Be practical, and be clear about the extent of what it explains

It also cites several axes as evidence that such a theory is already taking shape.

• Interpretable learning dynamics in deep linear networks
• Kernel-like explanations via NTK and linearized networks
• The mean-field and lazy-rich distinction
• Scaling laws, hyperparameter theory, and universal phenomena

It also notes that limits such as infinite width and infinite depth simplify complex systems and provide insights that remain valid for actual finite-scale models. It emphasizes that these results are also important for model design, optimization, data design, AI safety, and mechanistic interpretability.