AI Briefing[NeurIPS 2022] Transferring Pretrained Multimodal Representations via Cross-Modal Similarity Matching
Key point
Introduces BeamCLIP, a technique that efficiently transfers the representational power of the large-scale multimodal model CLIP to a smaller model.
Details
CLIP, trained on large-scale image-text pair data, shows excellent zero-shot performance, but it has the limitation of requiring massive computing resources and time.
To address this, LG AI Research proposes BeamCLIP, which transfers the representational power of a large-scale model to a smaller target model such as ResNet-18. BeamCLIP is an unsupervised representation transfer method in which a student model learns to regress the teacher model's vector representations for unlabeled images.
Two key methodologies are used as core techniques:
- Cross-modal Similarity Matching (CSM): mitigates the topological ambiguity that can arise in similarity-based vector regression.
- Context-based Prompt Augmentation (CPA): properly encodes text prompts without lexical ambiguity.
Experimental results show that BeamCLIP achieves 66.2% on ImageNet top-1 linear probe accuracy for ResNet-18, demonstrating superior performance compared to existing vision-based self-supervised learning (SSL) methods such as SimCLR (51.8%) and SwAV (63.7%).
