Meta FAIR and NYU study of 200+ MLLMs finds vision encoder pretraining optional but vision-only parameters required

Details

Vision-only parameters are required for strong VQA performance; encoder-free models trail at every size from 0.6B to 6.4B and require about 3x the compute to catch up.
Separate encoder pretraining is optional; encoders pretrained with four objectives at best tie an encoder trained from scratch with the LLM at matched compute.
Placement before the LLM is optional; an "in-fusion" approach adding zero-initialized vision-only weights to the LLM's first 8 layers is on par with a standard encoder from 1.7B to 6.4B.
The in-fusion approach achieves about 13% lower time to first token compared to a standard encoder.
At 1.7B scale, vision-only branches in layers 1–8 reach 45.0 VQA score, compared to 44.4 for the standard encoder and 42.3 when text shares those parameters.
Experiments used Qwen3-1.7B and 4B configurations with 0.4B parameters of vision capacity, evaluated on 16 VQA benchmarks from Cambrian at 448 px and 12 language benchmarks.

Key point