AI BriefingScaling MoE During Training
Key point
Growing a 32-expert MoE to 64 experts mid-training cut GPU hours by about 32%.
Details
Proposed expert upcycling, which grows a MoE trained with 32 experts into 64 experts midway through training.
Existing experts are duplicated, then small bias noise is added to the router to break symmetry, and pre-training continues so each replica specializes. Top-K routing is kept fixed, so per-token FLOPs and inference cost remain unchanged, and loss-free load balancing ensures all replicas receive gradients, preventing routing collapse.
The results are as follows.
- On a 7B→13B interleaved MoE (32→64 experts, Top-2, similar to Llama 4), validation loss of 1.263 vs 1.267.
- Average accuracy across 11 downstream benchmarks of 56.4 vs 56.7.
- About 32% savings in GPU hours compared to training a 64-expert model from scratch.
- About 67% savings in the sunk-cost scenario where a 32-expert checkpoint already exists.
It was also validated on a 256 expert Top-8 full MoE, showing generalization to configurations like DeepSeek-V3, Kimi K2, and GLM-4.5 families.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Toward the Physics of Multimodal Pre-training: Knowledge Flow, Modality Synergy, Early Integration, and Training Recipes
Step 5 Preview Released: 600B Parameter MoE Model Highlights Agentic Tasks and Financial Analysis Performance
Xiaomi Releases Open-Weight Multimodal Model MiMo-V2.6-Pro… Higher Intelligence and Competitive Pricing Compared to Similar-Sized ModelsTechnique for Simultaneous Control of Four Memory Peaks in Long-Context MoE Training Released