Toward the Physics of Multimodal Pre-training: Knowledge Flow, Modality Synergy, Early Integration, and Training Recipes

A study has emerged that systematically analyzes how language and visual information interact in multimodal pre-training. The researchers investigated knowledge transfer, inter-modality synergy and competition, and the impact of training timing through controlled experiments on synthetic and large-scale real-world data.

Experimental results revealed that knowledge flow between language understanding, visual understanding, and visual generation exhibited different influences and asymmetries. Data complexity emerged as a key factor determining whether modalities synergized or competed with each other.

In terms of architecture, the following configuration promoted inter-modality cooperation:

• Shared Attention and normalization layers
• Feed-forward layers separated by modality
• Observed trends maintained even with different visual tokenizer designs

Integrating and jointly training from the early stages proved more effective than connecting modalities in the latter half of training or training them sequentially. Delayed integration also led to vision laziness, a phenomenon where the model relied on linguistic prior knowledge and failed to fully learn visual information.

The researchers proposed a pre-training recipe that secured strong generative performance using only 5% of the compute budget. Additionally, they validated key findings in a large-scale environment by training multiple 13.5B MoE models on 2T tokens.