NVIDIA Exemplar Cloud: How to Get the Most Out of AI Infrastructure Performance

AI clusters built with the same NVIDIA H100, GB200 NVL72, and GB300 NVL72 systems can still see significant differences in training throughput. Real partner deployments, even when using the same workload, model, and global batch size, can be 8 to 12% slower than the NVIDIA reference architecture, and may fail to meet the 95% performance bar required for NVIDIA Exemplar Cloud validation.

Performance gaps are often the cumulative result of small losses across multiple factors, including kernel, hypervisor, BIOS, memory management, NUMA placement, and NCCL settings. To identify these, the specific signals seen in perf, NVIDIA Nsight Systems, and nccl-tests, along with how to tune them, are examined together.

The case studies cover the following four layers.

• Grace CPU and virtualization: How SMMU, IOMMU, page table, and page size settings affect kernel overhead
• x86 CPU performance and NUMA: Power management, turbo frequency, and process/helper thread placement issues
• NCCL and network fabric: Checking queue pair concurrency and collective communication settings in a 1.6Tbps environment
• Hardware installation status: Identifying silent installation defects that cannot be explained by software configuration alone

In the first case study, DeepSeek-V3 MoE FP8 pretraining based on GB200 NVL72 showed 12 to 14% longer iteration times on a VM compared to the bare-metal reference architecture. In contrast, dense models such as Llama 3 70B performed within 3%, and Nsight Systems traces captured signals of significantly increased CPU overhead in the MoE workload, which involves many small kernels.

Prerequisites for the diagnostics include an NVIDIA HGX H100, H200, or B200, a GB200 NVL72 or GB300 NVL72 cluster, an InfiniBand or RoCE interconnect, a distributed training workload with stable iteration times, root privileges, and nccl-tests built with the same NCCL version as the training stack. However, the checklist presented does not replace validation with actual applications, and which layer should be checked first varies depending on the platform and workload.