TorchTPU and TorchTitan Achieve 50%+ MFU on Llama 3 405B Training on TPU v7x

Details

TorchTPU and TorchTitan achieve 50%+ Model FLOPs Utilization (MFU) and 500 tokens/sec per chip when training Llama 3 405B on 1,024 TPU v7x chips.
The system validates out-of-box execution on the 5,000 most downloaded Hugging Face models, supporting minimal code changes via .to("tpu").
Qwen3 235B MoE training on 8,192 TPU v7x chips maintains 28% MFU, while TorchRec matches JAX performance (18.3ms step latency) on DLRM v2 using SparseCore.
Key optimizations include SimpleFSDP for communication fusion, Scan higher-order operations for constant compile time, and Pallas SplashAttention for 17.5% speedup in diffusion attention blocks.
TorchTPU and TorchTitan are currently available as open-source projects on GitHub.
Future roadmap includes FP8/FP4 quantization, end-to-end RL support, and hardware-assisted memory offloading via zero-copy DMA over the next 6-12 months.

Key point