[ICML 2022] Part 2: Self-supervised learning - LG AI Research BLOG

Self-supervised learning is a method for learning meaningful data representation from unlabeled datasets, and is broadly classified into two directions: Self-prediction and Contrastive learning.

This article focuses on the ADIOS(Adversarial Masking for Self-Supervised Learning) framework, one of the papers presented at ICML 2022, which effectively learns the semantic information of images.

Existing Masked image modeling had the problem of randomly masking pixels and relying on the local correlation of surrounding areas for reconstruction. This prevents the model from properly learning the high-dimensional semantic information of images.

ADIOS proposes a structure in which the Occlusion model and Inference model learn by opposing each other.

• Occlusion model (M): Generates a mask the size of the original image, and is trained to maximize the distance between the representation of the masked version and the original.
• Inference model (I): Takes the masked image as input, reconstructs it using surrounding context, and minimizes the distance so that it has the same representation as the original image.

This approach can be applied to existing Self-supervised models such as SimCLR, SimSiam, and BYOL, and induces the model to infer meaningful parts of the image beyond simple pixel reconstruction.