New Kernel Fusion Techniques Hide Up to 90% of Normalization Latency in GEMM and Attention Kernels

·2026.11.14 21:52

Key point

New kernel fusion techniques for LayerNorm and RMSNorm hide up to 90% of normalization latency in GEMM kernels and achieve up to 35% speedup in attention kernels on NVIDIA B200 GPUs.

1 / 15

Details

The article presents novel kernel fusion strategies, including Lazy Pre-Norm and Multi-CTA Norm Fusion, to reduce the memory-IO overhead of memory-bound normalization kernels.
These techniques can hide as much as 90% of normalization kernel latency by fusing with GEMMs, addressing a bottleneck that accounts for roughly 20% of training latency in Meta's Kunlun architecture and about 10% in typical LLMs.
The FlashNormAttention algorithm fuses pre- and post-normalization into a single GDPA kernel, achieving up to 35% kernel speedup by leveraging CTA clusters, distributed shared memory, and Tensor Memory (TMEM) reuse.
Benchmarks were performed on NVIDIA B200 GPUs in Meta's data centers using bfloat16 data, showing significant gains in ad modeling workloads.

pytorch performance-optimization attention triton kernel-fusion nvidia-b200

Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.