pytorch

Key New Features and Performance Improvements

• FlexAttention: Apple Silicon (MPS) support has been added, providing up to about 12x speedup over SDPA when applying sparse patterns. It also supports a deterministic backward pass for reproducible gradient computation in CUDA environments.
• CuTeDSL "Native DSL" backend: Provides Inductor with a second high-performance code path that operates alongside Triton, speeding up compilation of key GPU operations.
• nn.LinearCrossEntropyLoss: Combines prediction and loss computation to reduce peak GPU memory usage by up to 4x when training language models with a large vocabulary.
• torchcomms: A new communication backend for PyTorch Distributed that improves fault tolerance, scalability, and debugging capability when training on large-scale clusters.
• FSDP2: Overlaps reduce-scatter and all-gather communication through dedicated process groups, increasing distributed training throughput.
• Expanded platform support:
  • ROCm: AOTriton 0.12b and native HIP CMake support.
  • Arm: torch.compile support for Armv9-A targets.
  • Intel XPU: Exposes a new device telemetry API.
• Python 3.15 support: Python 3.15 wheel support on Linux, including free-threaded 3.15t.

Known Issues (Regressions)

• A problem has been reported where torch.compile does not work in CPU environments when using ROCm wheels.