Introducing AWS Interconnect Technology from a Distributed Training Perspective - AWS's Interconnect Foundation Technology, ENI

The performance of EFA ultimately depends on how you attach ENI (Elastic Network Interface). In EC2, instead of a physical NIC, a software-layer ENI takes on the networking role, and connection rules differ by instance type, including Primary ENI and Secondary ENI.

ENIs are divided into three types: ENA, EFA with ENA, and EFA-only. Since management traffic must be maintained on an IP basis, the Primary ENI must be either EFA with ENA or regular ENA, and EFA-only can only be used as a Secondary ENI.

On AWS high-performance instances, networking is managed across two layers: NetworkCardIndex and DeviceIndex. A network card is a logical unit of physical Nitro resources, and DeviceIndex refers to the order in which ENIs are placed within each card.

Key examples are as follows.

• p5.48xlarge / p5e.48xlarge: 32 network cards, 2 ENIs per card, 64 ENIs total
• p6-b200.48xlarge: 8 network cards, 4 ENIs per card, 32 ENIs total, 400 Gbps per card, 3,200 Gbps total

The core rules are also clear.

• The Primary ENI must always be assigned to NetworkCardIndex=0.
• Only up to 1 EFA ENI is allowed per network card.
• When using EFA on the Primary ENI, it must be EFA with ENA, not EFA-only.

For a practical configuration on p5.48xlarge, the approach introduced places both a management ENA and an HPC EFA-only together on the first card, while the remaining 31 cards are each filled with EFA-only. It's important to keep security groups and subnets consistent across the entire cluster, and All Traffic must be allowed within the same security group for EFA communication.

From an operations standpoint, rather than manually configuring 32 ENIs, it's more appropriate to automate this with CloudFormation, Terraform, or AWS ParallelCluster. After configuration, the recommended flow is to check the number of RDMA devices with ibv_devices, verify performance with fi_info, fi_pingpong, and nccl-tests, and continuously monitor with EFA node exporter and CloudWatch / Grafana.

Lastly, p6-b300.48xlarge, the latest generation released in 2025, is introduced as offering an even more advanced network configuration. The network design of AWS's high-performance GPU instances is evolving not toward simply increasing the number of cards, but toward boosting per-card bandwidth and distributed efficiency.