AI BriefingSkewAdam Cuts MoE Memory by 97%
Key point
A new optimizer called SkewAdam reduces optimizer state memory for MoE training from 50.6GB to 1.29GB, enabling 6.78B MoE training on a single 40GB GPU.
Details
SkewAdam, proposed in a preprint paper published on arXiv, is a hierarchical optimizer designed to resolve the VRAM bottleneck in Mixture-of-Experts (MoE) model training.
While conventional AdamW requires 50.6GB of optimizer state memory for a 12.6GB model, SkewAdam applies differentiated precision based on parameter roles:
- Backbone (5% of total): Momentum + factored 2nd moment
- Experts (95% of total): Factored 2nd moment only
- Router (<0.01%): Exact 2nd moment
With this approach, optimizer state memory drops from 50.6GB → 1.29GB (97.4% reduction), and peak training memory drops from 81.4GB → 31.3GB, allowing a 6.78B MoE model to be trained on a single 40GB GPU without degradation in convergence quality or router stability, according to the report.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.