Deep Learning Theory

This explains why deep learning generalizes without overfitting through output space dynamics. It defines the empirical NTK (eNTK) using the training output vector and the Jacobian, and tracks how gradient flow moves the outputs and loss.

This framework is claimed to apply to any differentiable architecture and convex loss, without infinite-width/infinite-depth assumptions. During training, directions where the loss actually vanishes are split into the signal channel, and directions that are not reached are treated as the reservoir; information remaining in the reservoir is considered invisible at test time.

From this perspective, the following phenomena are unified into a single framework.

• benign overfitting: the phenomenon where noise remains in the reservoir
• double descent: a phenomenon arising as noise moves back and forth between the signal channel and the reservoir
• implicit bias: the path in which gradient descent follows large kernel eigenvalues first, favoring low-norm solutions
• grokking: the process by which signals initially memorized later enter the signal channel due to kernel changes, resulting in delayed generalization

In the appendix, the authors also propose an Adam variant that updates parameters under the condition mu_k^2 > sigma_k^2/(b-1). Through this, the authors claim training directly aligned to population risk, 5x faster grokking, suppression of memorization in PINNs, and improved DPO fine-tuning.