[CVPR 2022] Continual Learning Research Trends

Artificial neural networks show excellent performance in single-task learning, but face the Catastrophic Forgetting problem, where they forget previous information when learning new tasks. The latest research directions to overcome this are broadly divided into three categories.

First, continual learning using Transformer models. Instead of the complex expansion methods of existing CNN-based models, the DyTox algorithm leverages the characteristics of Transformers to propose efficient and effective model expansion by utilizing task-specific tokens and embeddings.

Second, continual learning in Unsupervised (Self-supervised) settings. Recent studies have experimentally demonstrated that using Contrastive Learning can suppress Catastrophic Forgetting better than existing supervised learning using Cross Entropy.

Third, research on the proper use of Batch Normalization (BN) in continual learning settings. Since existing BN can be biased toward current task data and distort information from previous tasks, improved methodologies such as Task-balanced Batch Normalization are being proposed to address this issue.