The Annotated JEPA

This article implements Joint Embedding Predictive Architectures (JEPA) step by step from scratch, explaining in detail how it works. Just as 'The Annotated Transformer' covered the Transformer, the goal here is to build every component of JEPA and complete the training loop.

JEPA is a Self-Supervised Learning methodology proposed by Yann LeCun, focused on training models to understand the structure of the world without labels. The key idea is to perform prediction within a Representation Space (Latent Space) rather than getting bogged down in pixel-level details, thereby ignoring unnecessary noise and learning only meaningful features.

The main contents are as follows:

• I-JEPA: A method that learns semantic image representations by predicting the representations of masked regions of an image from the visible context
• V-JEPA & V-JEPA 2: Models that extend I-JEPA to video data
• LeJEPA: A recent attempt to replace engineering heuristics with a Distributional Regularizer

On the implementation side, for educational purposes, large-scale training engineering techniques such as FlashAttention and Mixed-precision are excluded, with the focus instead on clearly conveying the mathematical principles and core architecture.