Optimization Methods for Large-Batch SGD Training

This covers the Linear Scaling theory for solving the problems that arise when scaling single-GPU experiment results up to thousands of GPUs in a distributed training environment.

The core methodology is as follows:

• Learning Rate Scaling: The new learning rate is calculated using the formula learning_rate * (nk / baseline_batch_size) to maintain performance even with large batches.
• Linear Warmup: Gradually increases the learning rate during the initial training phase to ensure stability.
• Pipelining: Minimizes GPU idle time (bubbles) by starting gradient computation as soon as the computation of the previous layer finishes.

Applying this approach allows ImageNet to be trained very quickly while maintaining accuracy similar to single-node training even at a scale of up to 8k mini-batches.