DeepMind's TIPSv2 Vision-Language Encoder

DeepMind's TIPSv2 is the next generation of image-text encoders, digging into the surprising finding that distillation produces better patch-text alignment than standard pretraining. In particular, they observed a phenomenon where a smaller student surpasses a larger teacher, achieving stronger performance on zero-shot segmentation, and narrowed the core cause down to visible token supervision.

Based on this, they revised pretraining in three ways.

• iBOT++: Applies the patch-level self-distillation loss to all tokens, not just masked patches, strengthening dense alignment
• Head-only EMA: Applies EMA only to the projector head rather than the whole model, reducing trainable parameters by 42% while maintaining performance
• Multi-Granularity Captions: Mixes in Gemini Flash captions alongside alt-text and PaliGemma captions to provide richer text supervision

The biggest change comes from iBOT++. Extending supervision to visible patches improved ADE150 zero-shot segmentation by +14.1 mIoU, and patch-level representations became far more tightly aligned.

The difference is also clear in visualizations. In PCA feature maps, TIPSv2 shows sharper object boundaries than existing TIPS and SigLIP2, and while maintaining smooth representations like DINOv3, it achieves finer semantic segmentation.

The performance evaluation is broader. TIPSv2 showed strong results across 9 tasks and 20 datasets, achieving SOTA on all 4 zero-shot segmentation benchmarks in the dense image-text domain. In global image-text, it ranked best or second-best in 5 out of 7 evaluations, and in image-only tasks, it was best or second-best in 7 out of 9.

The comparative experiments are also impressive. When compared with DINOv3 at the largest shared size, ViT-L, TIPSv2 beat the DINOv3 teacher on 4 out of 6 shared evaluations while using 6x fewer parameters and 15x fewer images. Ultimately, TIPSv2 shows that redesigning pretraining around patch-text alignment can directly raise dense representations to distillation-level quality during pretraining itself.