Core Automation demonstrates frontier-scale training across distributed 'Compute Islands' with 1% performance penalty

Details

Core Automation validates that frontier-scale training can be executed across distributed 'Compute Islands' (geographically separated clusters) to bypass power and grid bottlenecks, achieving a 32B parameter Sparse Mixture-of-Experts (MoE) model with only a ~1% performance drop compared to standard colocated training.
The 2-site case study utilized 160 B300 GPUs in the US and 160 GB300 GPUs in Asia, connected by a 25 Gbps link over 9,000 miles; the distributed setup recorded 0.2525 bits/byte on general coding evals and 0.1995 bits/byte on systems & kernels evals, versus 0.2549 and 0.2017 bits/byte for standard training.
Technical implementation relies on fragment-level staggered synchronization over 50-step windows, a Distributed Shampoo optimizer variant, and stochastic rounding with error feedback to compress communication, while routing statistics are synchronized at every step.
Network engineering focuses on tail latency optimization through traffic pacing, warm connection reuse, independent chunk scheduling, and selective recovery of missing pieces, ensuring slow links do not block overall updates.
A 'Butterfly Mixing' experiment on a single site (64 GB300 GPUs split into 8 cells of 8 GPUs) achieved training quality identical to All-reduce baselines by exchanging model fragment pairs over 3 rounds, eliminating the need for full synchronization.
The approach enables the use of preemptible compute, mixed accelerator generations (e.g., GB300 and B300), and resilience against regional outages, though it introduces risks regarding model integrity and safety when training spans multiple providers.

Core Automation demonstrates frontier-scale training across distributed 'Compute Islands' with 1% performance penalty

Key point

Details

Related