AWS Interconnect Technology for Distributed Training: An Introduction to High-Speed GPU-to-GPU Communication

In large-scale distributed training, GPU-to-GPU communication performance is a key factor determining overall efficiency. In an environment where hundreds of GPUs exchange gradients, performance is determined by how efficiently the path from GPU memory to a remote node's GPU memory is designed.

GPUDirect RDMA bypasses the CPU in the data transfer path. Previously, data in GPU memory had to be copied to CPU memory before being sent to the network card, but this technology allows the NIC to directly access GPU memory via PCIe to read data. This eliminates unnecessary copy steps within a node and reduces latency.

AWS supports this technology starting from the p4d.24xlarge instance, and combines it with EFA (Elastic Fabric Adapter) to maximize optimization. While GPUDirect RDMA removes CPU intervention from the data path, EFA's kernel bypass structure bypasses the OS network stack during inter-node transfers.

GPUDirect Async (IBGDA) removes even CPU intervention from the control path. The GPU kernel writes directly to the network card's command queue, initiating DMA transfer immediately without CPU permission. According to NVIDIA benchmarks, IBGDA achieved up to a 9.5x throughput improvement over the existing approach for small messages under 1KB.

Recently, for communication optimization in MoE (Mixture-of-Experts) models, DeepEP from the InfiniBand camp and PPLX-kernels from the AWS EFA camp are presenting different approaches that support Expert Parallelism.