Maximizing Deep Learning Performance Through First Principles

Optimizing the performance of deep learning models should start not from mere tricks, but from First Principles that identify the system's bottleneck points.

Performance efficiency is broadly divided into three components:

• Compute: The time the GPU spends actually performing floating point operations (FLOPS).
• Memory: The time it takes to transfer tensors within the GPU.
• Overhead: All other control and management time.

The key to optimization is identifying which bottleneck region the current system is in.

• Memory-bandwidth bound: When a lot of time is spent on memory transfer, increasing the GPU's compute capability (FLOPS) yields little performance improvement.
• Compute-bound: When time is concentrated on large-scale matrix multiplication (Matmul), code optimizations to reduce overhead have little effect.

On modern hardware, the growth rate of compute capability (FLOPS) is far faster than the growth rate of memory bandwidth. Therefore, to fully utilize the compute capability of expensive GPUs, it is important to resolve memory bottlenecks and maintain a Compute-bound state. In particular, specialized hardware such as NVIDIA's Tensor Cores is optimized for matrix multiplication, so failing to properly utilize it means the hardware's potential performance cannot be fully drawn out.