AI Briefing[NeurIPS 2022] Transferring Pretrained Multimodal Representations via Cross-Modal Similarity Matching
Key point
Introduces BeamCLIP, a technique that efficiently transfers the representational power of the large-scale multimodal model CLIP to a smaller model.
Details
CLIP, trained on large-scale image-text pair data, shows excellent zero-shot performance, but it has the limitation of requiring massive computing resources and time.
To address this, LG AI Research proposes BeamCLIP, which transfers the representational power of a large-scale model to a smaller target model such as ResNet-18. BeamCLIP is an unsupervised representation transfer method in which a student model learns to regress the teacher model's vector representations for unlabeled images.
Two key methodologies are used as core techniques:
- Cross-modal Similarity Matching (CSM): mitigates the topological ambiguity that can arise in similarity-based vector regression.
- Context-based Prompt Augmentation (CPA): properly encodes text prompts without lexical ambiguity.
Experimental results show that BeamCLIP achieves 66.2% on ImageNet top-1 linear probe accuracy for ResNet-18, demonstrating superior performance compared to existing vision-based self-supervised learning (SSL) methods such as SimCLR (51.8%) and SwAV (63.7%).
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
LAION Big Video Dataset
Toward the Physics of Multimodal Pre-training: Knowledge Flow, Modality Synergy, Early Integration, and Training Recipes