Meta FAIR Finds Limited Impact of Vision Encoder Pretraining
·2026.11.18 17:00
Key point
Dedicated vision parameters contribute to performance, and placing them in the layers preceding the backbone achieves performance equivalent to that of an encoder.
Details
- Concluded that among the three design elements of multimodal vision encoders, dedicated vision parameters (Dedicated Vision Capacity) have a significant impact on performance, while the effect of pretraining is limited.
- Models without an encoder require 4.4x more compute for a 1.7B backbone and 3.0x more compute for a 4B backbone to achieve performance equivalent to encoder-based models.
- The in-fusion structure, which places dedicated vision modules in the 8 layers preceding the backbone, achieved performance equivalent to the encoder (Avg. VQA 49.2 vs 48.9) under the longest training budget.
- The in-fusion structure has a Time To First Token (TTFT) that is 13–18% shorter compared to the encoder.
- Experiments were limited to dense-architecture backbones of 6.4B or fewer parameters and training data at 224x224 resolution; no code or weights were released.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.