Sakana AI Releases 'Augmented Lagrangian Predictive Coding' Capable of Training 1000-Layer Neural Networks

Sakana AI proposed Augmented Lagrangian Predictive Coding (PC-ALM) to address the signal decay issue in deep networks within existing Predictive Coding (PC). PC-ALM introduces a dual variable (λ) in each layer, performing primal-dual inference on the Augmented Lagrangian instead of free energy (FPC). This allows each layer to function like a PI feedback controller combining current error (P) and accumulated error (I), enabling the computation of gradient signals for the global loss function through local computation alone. It has been theoretically proven that dual neurons converge to backprop credit signals in the linear network limit.

Experimental results show that PC-ALM successfully trained MNIST on a 1000-layer Residual MLP (width 32, ReLU), maintaining performance within approximately 2%p of Backprop. This marks the first instance of achieving 1000-layer depth with a layer-local method, contrasting with existing PC where accuracy degrades as depth increases. It is also effective on CIFAR-10 and Tiny ImageNet, showing improved performance over existing PC when training ResNet-10.