Text-Conditioned JEPA for Learning Semantically Rich Visual Representations

The existing I-JEPA predicts latent features at masked locations, but semantic learning was difficult due to the uncertainty of occluded regions.

TC-JEPA conditions on image captions and adjusts the predicted patch features via sparse cross-attention over the input text tokens. As a result, patch representations become better predicted by the text, leading to more semantic visual representations being learned.

• downstream performance, training stability, and scalability all improved or showed promise.
• It presents a new vision-language pretraining paradigm that trains solely through feature prediction.
• It outperformed contrastive methods, especially on tasks requiring fine-grained visual understanding and reasoning.