PyTorch 2.15 Adds FP8 Support and Boosts LLM Decoding Speed on Apple Silicon

PyTorch 2.15 introduces torch.float8_e4m3fn tensors and F.scaled_mm for direct GPU multiplication, halving storage space compared to BF16.
LLM decoding performance improves significantly on M3 Ultra with torch.compile, achieving 5-6.5x speedups for MoE models like Qwen3.6-35B-A3B (9.4 to 61.3 tokens/s) and LFM2.5-8B-A1B (23.6 to 123.6 tokens/s).
Video decoding memory usage drops from 270.135 GiB to 8.108 GiB peak driver memory when using tiled VAE decode for Wan-AI/Wan2.2-TI2V-5B-Diffusers.
Image editing accuracy is fixed for Qwen-Image-2.1 VAE encoder, improving encode-decode round trip PSNR from 10.06 dB to 29.95 dB to match CPU results.

Performance Metrics

Softmax operations using Metal kernels reduce latency from 57 µs to 7 µs for [1, 128256] FP32 logits, a roughly 8x improvement.
Linear layers using GEMV kernels show a 1.23x geometric mean speedup across 96 BF16 layer shapes.
MoE decoding token throughput increases by approximately 1.3x on M3 Ultra due to native grouped matmul implementation.

Attention operations will fully migrate to Metal, removing MPSGraph dependencies and enabling dedicated vector kernels for decoding.
Further LLM decoding speedups are expected, with LFM2.5-8B-A1B projected to reach 126.0 tokens/s (+16%) and Qwen3.6-35B-A3B to 62.8 tokens/s (+19%) in BF16 compiled decoding.
Installation size is reduced by shrinking libtorch_cpu.dylib by approximately 25% (384 MiB to 287 MiB) and optimizing shader compilation.