Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster

• Meta's Prometheus supercluster in New Albany, Ohio, has completed Phase 1 at 1GW scale, currently training Muse-based models including Muse Wonder.
• Achieved over 85% pre-training goodput and over 90% reinforcement learning goodput, with mean time between interruptions (MTBI) improved from 47k to over 54k GPU hours.
• Deployed NVIDIA GB200 NVL72 (Catalina) and GB300 NVL72 (Clemente) platforms, with the latter offering 288 GB HBM per GPU and over 1 exaFLOP FP4 peak performance.
• Future expansion includes the San Miguel platform based on NVIDIA Vera Rubin NVL72, featuring 2.52 peak EFLOPS FP4 training performance per rack.

Hardware & Network Architecture

• Utilizes NVL72 rack architecture where 72 GPUs act as a single accelerator, connected via a dedicated Ethernet back-end network for low-latency communication.
• Network fabric supports up to 20,736 GPUs per building using NVIDIA Spectrum-4 based switches and Meta's FBOSS networking stack.
• Implements PAFT (Parallelism-Aware Fault Tolerance) to isolate faults at the data-parallel replica level, using spare GPUs within the 72-GPU domain to maintain training continuity.

Operational & Power Management

• Dynamo Dimmer framework manages power transients by gradually adjusting GPU power limits to avoid hard stops during 30-40% cluster-wide power fluctuations.
• Fleet-scale firmware deployment automation reduced single-pot upgrade P90 time from over 7 hours to 2 hours for 100k+ GPUs.
• Cooling strategy combines Air-Assisted Liquid Cooling (AALC) for rapid deployment and Facility Liquid Cooling (FLC) for maximum power density in purpose-built AI facilities.