Apple Research Introduces First Tri-Modal Masked Diffusion Models Pretrained from Scratch

Apple Research introduces the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image-text, and audio-text data. Diverging from previous approaches that initialized or finetuned unimodal bases, this work systematically analyzes multimodal scaling laws, modality mixing ratios, and noise schedules to provide optimized inference sampling defaults.

Novel Batch-Size Reparameterization

The study’s batch-size analysis yields a novel stochastic differential equation (SDE) based reparameterization. This method eliminates the need for tuning the optimal batch size, decoupling the physical batch size (chosen for compute efficiency) from the logical batch size (chosen to balance gradient variance during stochastic optimization).

Model Performance and Scope

A preliminary model showcases the capabilities of this unified design, achieving strong results at 3B model scale trained on 6.4T tokens. The model performs well in text generation, text-to-image (T2I), and text-to-speech (T2S) tasks. This work represents the largest scale systematic open study of multimodal discrete diffusion models conducted to date.