Building Large-Scale AI Training Networks with MRC (Multipath Reliable Connection)

OpenAI has released MRC (Multipath Reliable Connection) to the Open Compute Project (OCP). This protocol, co-developed over two years with AMD, Broadcom, Intel, Microsoft, and NVIDIA, aims to simultaneously improve the performance and resilience of GPU networks used for frontier model training. It extends RoCE, combining techniques from UEC with SRv6-based source routing.

In large-scale training, millions of data transfers converge in a single step, and network congestion or link flaps delay the entire job. To reduce this, OpenAI redesigned the network to be simpler, structured so that training doesn't stop even when failures occur.

The core is a multi-plane network.

• A single 800Gb/s NIC is split into 8 x 100Gb/s links, operating as 8 parallel planes.
• 64-port switches are expanded to 512-port, allowing about 131,000 GPUs to be connected with 2-tier switching.
• This approach reduces power and component count compared to conventional 3-4 tier designs, while increasing path diversity.

MRC adds packet spraying and adaptive path management on top of this. Rather than binding a single transfer to one path, it spreads it across hundreds of paths, and the receiving side writes directly to memory based on each packet's final memory address. MRC connections maintain a small amount of per-path state, switching away from congested paths and immediately excluding a path when loss occurs, then confirming recovery with probe packets. Packet trimming also reduces false positives that would otherwise mistake congestion loss for failure.

Routing has been changed to SRv6-based source routing. Since the sender directly specifies the packet's path and switches simply follow a pre-configured static routing table, there's no need to run dynamic routing like BGP. As a result, failure recovery happens on the microsecond scale, and training can continue without second-scale reconvergence.

This approach is already deployed on OpenAI's largest NVIDIA GB200 supercomputers, Oracle Cloud Infrastructure (OCI)'s Abilene site, and Microsoft Fairwater. In actual operation, even with multiple link flaps occurring per minute, there was no measurable impact on training, and it has also been used for training multiple OpenAI models leveraging NVIDIA and Broadcom hardware. Even when 4 tier-1 switches were rebooted, work continued without coordination with the operations team, and when 1 port drops out on an 8-port NIC, maximum speed decreases by only 1/8, with MRC recalculating paths to typically restore the plane within about 1 minute.