Meta Research Finds Pretrained Vision Encoders Are Optional for Multimodal Performance

·2026.11.16 09:00

Key point

A study by Brown et al. shows that adding vision-specific capacity in the first 8 layers of the backbone (in-fusion) achieves 50.1 Avg. VQA in midtraining, performing comparably to pretrained encoders while reducing time-to-first-token latency.

Details

  • Pretrained vision encoders are optional: scratch-trained encoders achieve comparable Avg. VQA scores to pretrained encoders, even when pretraining is free. More encoder pretraining raises ImageNet accuracy but not VQA.
  • Vision-specific capacity is essential: encoder-free models require approximately 3x more compute to match encoder performance. Adding the encoder's 0.4B parameters to the backbone without dedicated vision modules yields negligible improvement in Avg. VQA.
  • In-fusion design, placing vision-specific modules in the first 8 layers, achieves 50.1 Avg. VQA on a 1.7B backbone in midtraining, outperforming Mixture-of-Transformers (MoT) at 48.5 and encoder-free at 39.8, while remaining comparable to the scratch encoder (48.6).
  • In-fusion reduces time-to-first-token latency compared to standard encoders. In-fusion adds 37ms to the encoder-free baseline, while standard encoders add 64ms and full-layer MoT adds 100ms.

Experimental Setup and Controls

  • All models trained from scratch with identical data, backbone architecture, and learning recipes to isolate design variables.
  • Vision encoders are 27-layer, 0.4B parameter ViTs; pretraining objectives include MetaCLIP, WebSSL, Cap, and CapPa.
  • Benchmarks include Avg. VQA (16 Cambrian benchmarks) and Avg. Text (12 language benchmarks), with lifetime compute ranging from 6.5x10^20 to 2.9x10^22 FLOPs.

Comparison with Existing Methods

  • In-fusion outperforms MoT (Mixture-of-Transformers) which uses separate modules per modality, achieving higher Avg. VQA with fewer vision-only parameters in midtraining.
  • Previous studies (Tong et al., Shukor et al.) failed to isolate variables or used reduced learning rates for encoders, masking the true advantage of dedicated vision capacity.
  • Held-out caption loss correlates better with Avg. VQA trends than average validation loss, which shows minimal difference (max 0.05 nats) between encoder and encoder-free models.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.