Microsoft AI Reveals Pre-training Recipe for 2.8T MoE Model 'MAI-Base-2'

Microsoft AI has released the Frontier Pre-training report, revealing the completion of pre-training for MAI-Base-2 and its technical details. The model adopts a Mixture-of-Experts (MoE) architecture with 2.8T total parameters and 80B active parameters, and is currently in the post-training stage.

Key Claims and Efficiency

Microsoft AI claims that MAI-Base-2 requires 4.2x to 12.3x less compute to achieve the same Bits Per Byte (BPB) performance as the competing model DeepSeek-V4-Pro-Base. Based on an internal held-out evaluation set, the model recorded efficiency gains of 12.3x in code, 8.3x in STEM, 6.6x in math, and 4.2x in QA. However, since the actual training compute figures for the comparison model are undisclosed, these numbers are estimates derived from scaling ladders and cannot be externally reproduced.

Training Infrastructure and Scaling

Training was conducted on 16,000 GB200 GPUs for 38 days, processing 50T tokens. The average Model FLOPs Utilization (MFU) was 21%, and Goodput was 89%. The team applied a 'scaling ladder' technique, using smaller models (L18–L42) to predict the performance of larger models. When extrapolating to compute budgets approximately 130x larger than L42, the error between predicted and actual values was around 0.3%.

Key Technical Changes

The following architectural and optimization changes were made compared to MAI-Base-1:

• Multi-round Dropless MoE: MoE is applied to all layers, introducing a method that repeats dispatch-compute-collect rounds with fixed capacity per expert group without dropping tokens. This improved MFU in MoE layers from 24% to 35%.
• Optimizer Change: Switched from AdamW to the Muon optimizer (for 2D parameters) to enhance training stability and efficiency. Momentum resharding was performed during the optimizer step to ensure ZeRO compatibility.
• Native VLM: Transitioned from a text-only model to a native Vision-Language Model (VLM). Vision data was trained from scratch using an Encoder-free structure without a separate image encoder, accounting for approximately 18% of total training tokens.
• Data Strategy: Used only human-created Organic Data without distilled data. The Knowledge Cutoff is March 2026. The proportions of code and PDF data were adjusted, and embedding-based diversity sampling was applied.

Limitations and Caveats

Weights are not public and will be licensed only to Microsoft Foundry partners through data provenance management. BPB is merely a next-token prediction metric and does not directly indicate downstream benchmark performance. Additionally, independent verification is difficult because the comparison evaluation sets are undisclosed.