Key Changes in PyTorch 2.12

PyTorch 2.12 broadly strengthens CUDA performance, export, graph capture, distributed training, ROCm, and MPS. This release was completed with 2,926 commits and 457 contributors.

The key changes are as follows.

• CUDA batched linalg.eigh is now up to 100x faster than before by adopting cuSolver syevj_batched, closing the long-standing performance gap with CuPy as well.
• torch.accelerator.Graph was added as a device-agnostic graph capture/replay API spanning CUDA, XPU, and out-of-tree backends, with backend-specific implementations registered via GraphImplInterface to maintain autonomy. is_capturing() was added to c10::Stream and torch.Stream.
• torch.export.save/load now supports the Microscaling (MX) quantization format and the float8_e8m0fnu dtype, making it possible to export compressed models such as MXFP4, MXFP6, and MXFP8.
• fused=True was added to Adagrad, providing a single-kernel optimized path like Adam, AdamW, and SGD.
• torch.cond leverages CUDA 12.4's conditional IF nodes to be captured and replayed within a CUDA Graph; it currently works with the eager and cudagraphs backends, with Inductor support left as a next step.
• addcdiv now uses FMA-based lowering to improve numerical consistency on CUDA and XPU, taking care of both compilation performance and reproducibility in optimizer-heavy training loops.

In distributed training, custom ops can now directly accept ProcessGroup objects, and c10d functional collectives such as all_reduce and reduce_scatter also accept both objects and string names. The Profiler Events API exposes flow ID, flow type, activity type, unfinished events, and Python function events, and NCCL collective traces gained a new seq_num that makes cross-rank correlation analysis easier. FlightRecorder has expanded its supported scope to include ncclx, gloo, and torchcomms in addition to the existing nccl and xccl, and a singleton race condition was also fixed.

On the platform side, enable_annotations was added to CUDA Graph, allowing metadata such as collective op name, process group, and message size to be included in traces, and a workqueue limit was added to Green Context. On ROCm 7.02 and above, expandable segments were added, along with rocSHMEM symmetric memory collectives and FlexAttention 2-stage pipelining. On ROCm 7.12 and above, hipSPARSELt is enabled by default, and FP8 semi-structured sparsity is also supported on MI350X (gfx950). Apple Silicon binary wheels now include macOS 26-based Metal-4 offline shader compilation, along with a dedicated API for directly loading precompiled .metallib files.