[CVPR 2022] Large Vision-Language Models: What's Next?

The large-scale model training boom triggered by GPT-3 in the NLP field is now expanding into the Vision-Language field. The number of parameters in a model determines the diversity of information it can capture, but as scale increases, massive amounts of data and GPU resources are required.

In Phase 1, OpenAI's DALL-E emerged, opening the era of zero-shot text-to-image generation. DALL-E is a Transformer model with 120 billion parameters, trained on 250 million image-text pairs, and is capable of generating concepts not present in the training data. Here, the CLIP model is used to analyze the similarity between images and text.

Phase 2 covers the combination of VQ-VAE (Vector Quantized-Variational AutoEncoder) and Transformer. To apply the Transformer, which was originally text-only, to images as well, DALL-E uses VQ-VAE.

VQ-VAE consists of an encoder that compresses images and a decoder that reconstructs them. In particular, through the Vector Quantization (VQ) process, the compressed information is separated into tokens of a certain size, and each is replaced with the most similar token in a Codebook, converting the image into a format similar to text and thereby improving training efficiency.