Analysis of BLOOM Model Training Infrastructure

This covers in detail the hardware and software engineering technology invested to train BLOOM, a large-scale multilingual language model with 176B parameters.

Hardware Configuration

• GPU: 384 NVIDIA A100 80GB GPUs (48 nodes)
• Compute Resources: Utilized France's Jean Zay supercomputer
• Network: NVLink 4 between GPUs and Omni-Path Architecture (OPA) between nodes

Software and Architecture

• Model Structure: An improved structure based on the GPT-3 architecture
• Training Stack: Integrated use of the PyTorch framework and the Megatron-DeepSpeed library

Data and Training Scale

• Dataset: 350B tokens across 59 languages (1.5TB of refined text)
• Training Duration: Approximately 3.5 months