AI BriefingFlashMask-3 Achieves 45-141% TFLOPs Gain Over FlashMask on H100 GPUs
·2026.10.20 23:02
Key point
The new Hopper kernel and distributed extension improve throughput by 1.80x to 3.71x over Megatron-Core baselines at context-parallel degrees 4-32.
Details
- FlashMask-3 introduces a mask-aware Hopper kernel with three-role warp specialization and a distributed extension featuring zero-communication mask localization and a sparsity-aware load balancer (HCC-IPO).
- On a single H100 across 12 masks at 4K-128K context, the system improves achieved TFLOPs by 45-141% over FlashMask, 34-71% over FlexAttention, and 2-44% over MagiAttention.
- At context-parallel degrees 4-32 with 8K sequence per device, it attains 31-75% compute-communication overlap efficiency and improves throughput by 1.80x-3.71x over Megatron-Core CP ring and 1.71x-2.92x over Megatron-Core CP all-gather.
- In end-to-end training of a 118B-A14B MoE at 128K on 128 H100 GPUs, throughput increases by 7.3-11.0% over a Dual Chunk all-gather baseline, with attention forward+backward 24.5-44.9% faster.
Technical Implementation
- Uses a column-wise sparse mask indexed by key positions but storing query-axis interval boundaries to reduce per-rank mask localization to a provably zero-communication clip-shift-max operation.
- Implements mask-aware compute-communication overlap with mask-driven communication pruning, where the forward all-gather uses topology-aware hierarchical routing.
- Built atop FlashAttention-3's TMA/WGMMA pipeline to exploit Hopper GPU capabilities, addressing limitations in previous Ampere-only implementations.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.