AI BriefingTransferring Pre-trained Multimodal Representations via Cross-modal Similarity Matching
Key point
This work proposes BeamCLIP, a methodology for efficiently transferring the representational power of large-scale multimodal models to small target models.
Details
Existing large-scale multimodal models such as CLIP show excellent zero-shot performance, but have the limitation of requiring massive data and computational power for training. To address this, BeamCLIP presents a method for transferring the representation of a large-scale model to a small target model such as ResNet-18.
BeamCLIP is an unsupervised representation transfer approach, in which the student model is trained to regress the vector representation of the teacher model. In this process, two key techniques are used.
- Cross-modal Similarity Matching (CSM): used to alleviate the problem of topological ambiguity.
- Context-based Prompt Augmentation (CPA): used to properly encode text prompts without lexical ambiguity.
Experimental results show that BeamCLIP achieves 66.2% on ImageNet-1K top-1 linear probe performance, demonstrating superior performance compared to existing vision-based self-supervised learning (SSL) methods such as SimCLR(51.8%) and SwAV(63.7%).
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
LAION Big Video Dataset
Toward the Physics of Multimodal Pre-training: Knowledge Flow, Modality Synergy, Early Integration, and Training Recipes