RCCLX: Innovating GPU Communication on AMD Platforms

Meta has open-sourced an initial version of RCCLX. RCCLX is an extended version of RCCL, fully integrated with Torchcomms, enabling rapid experimentation with communication innovations on AMD platforms as well.

It also integrates CTran into AMD platforms, supporting GPU-resident collectives such as AllToAllvDynamic. However, not all CTran features are included in the current open-source version, and Meta plans to add them over the coming months.

The core features introduced are Direct Data Access (DDA) and Low Precision Collectives. Both focus on reducing communication bottlenecks on AMD platforms and improving efficiency for large-scale AI workloads.

DDA is designed around the fact that the two stages of LLM inference—prefill, which processes the prompt to build the KV cache, and decode, which generates tokens one at a time—have different characteristics.

• Prefill is strongly compute-bound, as attention computation grows sharply with sequence length.
• Decode is a memory-bound stage dominated by reading the KV cache and model weights.
• In Tensor parallelism settings, AllReduce can account for up to 30% of E2E latency.

To reduce this, Meta created two DDA algorithms.

• DDA flat: At small message sizes, each rank directly reads other ranks' memory to perform local reduction. It reduces latency from O(N) to O(1), while increasing the amount of data exchanged from O(n) to O(n²).
• DDA tree: Splits AllReduce into two stages—reduce-scatter and all-gather—using direct data access in each stage. It moves the same amount of data as the ring algorithm while lowering latency to a constant level at somewhat larger message sizes.

On AMD MI300X, DDA showed performance improvements of 10-50% in the decode phase and 10-30% in the prefill phase compared to the RCCL baseline. This resulted in roughly a 10% reduction in TTIT (time-to-incremental-token), improving the decode phase, which has a large impact on actual user experience.

Low-precision collectives are distributed communication algorithms optimizing AllReduce, AllGather, AlltoAll, ReduceScatter for AMD Instinct MI300/MI350 GPUs. They support FP32 and BF16, and apply FP8 quantization for up to 4:1 compression, reducing communication overhead especially for large messages of 16MB or more.

These algorithms use P2P mesh communication to actively leverage the bandwidth and latency characteristics of AMD Infinity Fabric. The computation stage is performed in high precision (FP32) for stability, and precision loss is mainly determined by the number of quantizations per collective (about 1-2 times) and whether values fall within the representable FP8 range.

Internal experiments observed the following results:

• About 0.3% change on the GSM8K evaluation
• 9-10% decrease in E2E latency
• About 7% increase in throughput

Measurements were performed with param-bench rccl-tests, testing MI300 on ROCm 6.4 and MI350 on ROCm 7.0-based RCCLX. Each test was run with 10 warmup and 100 measurement iterations, and the figures in the graphs represent the average throughput over the measurement period.

RCCLX is integrated as a custom backend of the Torchcomms API, allowing users to migrate applications while keeping the same API even when the platform changes. Meta aims to functionally align this backend with NCCLX for NVIDIA, and plans to expand new features provided by CTran under the same API.

Finally, after installing Torchcomms, users can initialize a communicator like torchcomms.new_comm("rcclx", torch.device("hip"), ...), and enable low-precision collectives via the environment variable RCCL_LOW_PRECISION_ENABLE=1.