PyTorch 2.14 Release: NVGEMM Backend, c10d Fault Tolerance, Python 3.15 Wheel Support (feat. torchvision 0.29)

The PyTorch Foundation officially released PyTorch 2.14 on September 2, 2026. This release is the fourth version in a two-month cycle, incorporating 2,995 commits from 487 contributors.

Key Performance and Backend Updates

• Introduction of NVGEMM Backend: NVGEMM, which generates CUTLASS kernels using CuTeDSL, has officially joined Inductor. It supports epilogue fusion, scaled GEMM, NVFP4 GEMM, and competes as an auto-tuning candidate alongside Triton templates.
• Apple Silicon Linear Algebra Support: Linear algebra operations such as SVD, eigh, QR, and Cholesky decomposition are supported via native Metal kernels through MPS (Metal Performance Shaders).
• Platform Expansion: Includes ROCm 7.14 wheel support, native graph capture for Intel XPU, and targeting for the NVIDIA Rubin architecture (sm_107).

Enhanced Distributed Training and Fault Tolerance

• First-Class c10d Fault Tolerance: An interface was added to reconfigure process groups in place upon rank failure without restarting them. This preserves the warmed-up state of the cluster, reducing recovery time.
• nccl2 Backend Integration: The existing torchcomms has been integrated into the PyTorch core as the nccl2 backend, supporting non-blocking communicators and one-sided Remote Memory Access (RMA) windows.
• Flight Recorder Expansion: Hooks were added to make the Flight Recorder debugging tool, previously exclusive to NCCL, available across all backends.

Ecosystem Changes

• torchvision ABI Stabilization: Starting with torchvision 0.29.0, ABI (Application Binary Interface) compatibility with torch is guaranteed, eliminating the need to upgrade torchvision whenever the torch version is upgraded.
• Dynamic Shape and Branching: Multi-way branching is generalized with torch.switch, and dynamic shapes can be declaratively specified using the @dynamic_spec decorator.