AI BriefingMatrix Orthogonalization Improves Memory in Recurrent Models
Key point
Applying Orthogonalization to mLSTM's memory matrix improved the recurrent model's associative recall ability under noisy conditions.
Details
Research has been conducted to improve the performance of recurrent neural networks (RNNs) such as mLSTM in order to catch up with the Transformer's powerful Associative Recall (AR) capability. Existing mLSTM showed excellent performance on certain benchmarks, but had limitations in measuring performance under noisy conditions (NAR).
This research borrows ideas from the optimization tool Muon, proposing a method that applies Orthogonalization to mLSTM's memory matrix during the Read step. This prevents updates in a specific direction from dominating and helps ensure that weak memory information is not lost.
Experimental results showed that the orthogonalized mLSTM achieved an overwhelming performance improvement over the existing mLSTM on the Noisy AR task of MAD:
- Vocab 80, Len 512: Accuracy 87.5% (+18.4%p compared to Baseline)
- Vocab 96, Len 1024: Accuracy 68.5% (+45.4%p compared to Baseline)
The research team found that performance was highest when the orthogonalized memory was used only at the Readout step without being written back.
