AWS Organizes Foundation Model Infrastructure

AWS organized the foundation model training and inference stack into layers of compute, network, storage, orchestration, observability.

The core infrastructure is the P5/P5e/P5en/P6 GPU instance family and the EFA network. In particular, P6 introduces B200/B300 and emphasizes a configuration that uses NVLink/NVSwitch within a node together with EFA between nodes, comparing Tensor throughput, HBM capacity, and HBM bandwidth across H100, H200, B200, and B300 to show that the bottleneck in large-scale distributed training and inference lies in communication and memory movement rather than computation.

• EFA provides OS-bypass RDMA.
• EFAv3 reduces packet latency by about 35% compared to EFAv2.
• EFAv4 further improves collective communication performance by 18% compared to EFAv3.
• Storage is tiered into NVMe instance store, FSx for Lustre, and S3 to share the load of checkpointing and data loading.
• Resource management is explained around Slurm and Kubernetes, development around PyTorch and JAX, and observability around Prometheus and Grafana.

For large-scale deployments, AWS presents Amazon EC2 UltraClusters, a structure that densely places thousands of accelerated instances within a single Availability Zone and connects them with a petabit-scale nonblocking network.