Chinese CLIP: Contrastive Vision-Language Pretraining Based on Chinese

When translating the existing CLIP for application to Chinese, there are limitations such as translation quality issues and failure to adapt to the image domain of Chinese culture. To address this, the Chinese CLIP project was launched for Chinese multimodal representation learning.

Instead of training from scratch, the training method adopts an efficient two-stage pretraining approach.

• Stage 1: Initialize a ViT-B or ResNet-based vision encoder and a Chinese RoBERTa language encoder. With the vision encoder frozen, the language encoder is adapted to fit the output space of the CLIP vision encoder.
• Stage 2: Unlock the vision encoder and jointly fine-tune both towers to learn the image distribution of Chinese data.

A total of 200 million image-text pairs were used, including Chinese data from LAION-5B and the Wukong dataset. The model is provided in a total of 5 versions: ResNet-50, ViT-B/16, ViT-L/14, and ViT-H/14.

Experimental results showed overwhelming performance over existing models on major datasets such as MUGE, Flickr30K-CN, and COCO-CN. In particular, it showed the largest performance gap on MUGE, a native Chinese dataset, demonstrating the powerful utility of language-specialized models.