Early Prediction of Grokking via Graph Analysis

This methodology monitors neural network topology during training by combining the Fiedler value (the second-smallest eigenvalue of the Laplacian) of the neural network weight graph with the Scheffer CSD (Critical Slowing Down) metric.

The key experimental results are as follows:

• Detection: The lambda-2 metric detects the approach of grokking 21,000 steps before test accuracy changes.
• Classification: It was confirmed that grokking and catastrophic forgetting have distinct structural signatures (gradient differences).
• Steering: Intervention via structural guidance achieved a knowledge retention rate of 91.7% (compared to 2.6% without intervention).
• Compounding: When performing sequential tasks, it accelerates grokking by 48x while showing high knowledge retention.
• Preemptive curriculum: Accurately predicts the risk of conflict between tasks, preventing knowledge loss.

This research was conducted on a 2-layer MLP and a 1-layer Transformer, applying early-warning indicators from complex systems science to weight graphs. However, since this is currently toy task-based research, validation on large-scale models is needed.