AWS Compute Selection Guide for Distributed Training (Part 2: Ultra-Scale Scaling and Instance Availability)

Amazon EC2 UltraClusters place thousands to tens of thousands of GPU instances on a petabit-scale non-blocking EFA network fabric to support large-scale distributed training. It maintains uniform inter-node bandwidth at a scale larger than typical Placement Groups, reducing bottlenecks in specific communication paths and GPU idle time.

UltraCluster is structured as a folded Clos network that connects multiple switches hierarchically rather than a single switch. UltraCluster 2.0 reduced the maximum host-to-host path from 7 hops in a 3-tier Clos to 5 hops in a 2-tier Clos, focusing on guaranteeing full bisection bandwidth, which matters more than latency differences.

Inter-node communication can be handled directly between GPU memory without going through the CPU, via EFA's SRD protocol and GPUDirect RDMA. If NCCL is not configured to use EFA through aws-ofi-nccl, it silently falls back to TCP, causing performance to plummet without any errors, so checking EFA and NCCL configuration is essential.

Securing high-performance instances relies on ODCR (On-Demand Capacity Reservation) and Capacity Block. Data is stored in S3 and served in parallel via FSx for Lustre; when thousands of nodes read simultaneously, filesystem throughput must be sufficiently provisioned, and data sharding and preload must be completed in advance.

Job scheduling is handled by SageMaker HyperPod, AWS PCS, Amazon EKS, and others, while checkpoints are stored in FSx and S3 to prepare for node failures or spot reclamation. UltraCluster alone is not sufficient—data pipelines, orchestration, and fault recovery systems must all be in place together.