AWS Computing Selection Guide for Distributed Learning (Part 3: Cluster Construction and Operations)

In distributed learning, dozens to hundreds of nodes must run the same software stack, including CUDA, cuDNN, NCCL, aws-ofi-nccl, and PyTorch. Even minor version differences between nodes can lead to communication errors or training failures, so pinning the execution environment in containers has become the standard.

It is difficult to apply Docker directly to traditional Slurm HPC environments. Docker requires root privileges and a persistently running daemon, and it does not share resource allocation, limits, or accounting with Slurm, which can result in misaligned GPU, CPU, and memory management or leftover containers after job completion.

A representative solution to this problem is the Enroot + Pyxis combination.

• Enroot: An NVIDIA container runtime that runs containers without root privileges or a persistent daemon
• Pyxis: An NVIDIA Slurm plugin that connects Enroot to Slurm to run containers on a per-job basis
• .sqsh images: Pinning verified execution environments, such as CUDA, NCCL, and PyTorch, into a single file

Docker images can be converted to .sqsh files via Enroot. In practice, instead of copying these to each node, they are stored on a shared filesystem like Amazon FSx for Lustre so that all nodes reference the same image. Using Pyxis allows submitting container-based jobs by simply adding the --container-image option while maintaining the existing Slurm partition, QOS, and sbatch workflows.

In AWS environments, the version of aws-ofi-nccl inside the container is also important. Since this plugin connects NCCL communication to the high-speed path of EFA, verifying and pinning the CUDA, NCCL, and aws-ofi-nccl combination within the image is necessary to ensure GPU-to-GPU AllReduce performance and distributed learning stability.

In AWS ParallelCluster-based architectures, the Slurm controller on the Head node receives jobs and places them on compute nodes. The Slurm client on each compute node calls Pyxis to run Enroot containers. Since all containers use the same .sqsh image, core environments such as CUDA, NCCL, and PyTorch remain consistent across multiple nodes and GPUs.