AI BriefingKDA 2.22x
Key point
Moonshot released FlashKDA, which is up to 2.22x faster than Triton on H20.
Details
MoonshotAI has open-sourced FlashKDA. It's a CUTLASS C++ forward kernel for Kimi Delta Attention (KDA), connected as a backend for flash-linear-attention.
Benchmarks were measured on H20, and performance gains over the existing Triton baseline were confirmed.
- T=8192, H=96, D=128 fixed length: 1.72x
- variable length / mixed seq_lens: 1.95x
- variable length / uniform 1024x8: 2.22x
The implication is clear. Linear attention variants are theoretically advantageous as length increases, but actual performance is determined by the kernel implementation. FlashKDA requires SM90 or higher, CUDA 12.9 or higher, and PyTorch 2.4 or higher, and is distributed under the MIT License.
However, the currently released figures are based on forward pass only, and were measured only on H20. Absolute performance on H100 or Blackwell may differ, and the backward kernel has not yet been released.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Exploring DeltaNet-family Linear Attention VariantsOpenArch Released: A Collection of PyTorch Implementations for Modern LLM Architectures
ZeroModels Released, Compatible with JAX, PyTorch, and TensorFlow