AI BriefingDomyn scales 1T MoE training to 1,300 tokens/s per GPU on JUPITER using TorchTitan
·2026.11.14 01:28
Key point
The team optimized FSDP and Expert Parallelism on 384 GH200 GPUs to reach 1,300 tokens/s per GPU, overcoming inter-node bandwidth bottlenecks.
Details
- Achieved 1,300 tokens/s per GPU throughput for a ~1T-parameter MoE on 384 GH200 GPUs, a 14x improvement over the initial baseline.
- Utilized 384-way FSDP and 4-way Expert Parallelism (EP) within nodes to minimize cross-node communication, leveraging the 900 GB/s Grace-Hopper C2C link for activation offloading.
- Reduced optimizer step time from 42 seconds to ~2 seconds by running Newton-Schulz iterations on the GPU instead of the CPU, contributing to a 35% throughput increase.
- Implemented FP8 expert all-gather and node-local resharding to cut communication volume, resulting in an 18% gain without affecting training loss.
- Fixed memory allocator issues by moving allocations to the compute stream, eliminating retries and boosting throughput by 21% while reducing peak reserved memory by 13.5 GiB.
- Model and training recipes are planned for open release, with all fixes contributed upstream to PyTorch and TorchTitan.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Core Automation demonstrates frontier-scale training across distributed 'Compute Islands' with 1% performance penaltyPPinterest Introduces ModuleLens for Isolated PyTorch Module Debugging and Agentic Optimization