Qwen Unveils FlashQLA

Qwen has released FlashQLA, a high-performance linear attention kernel built on TileLang, claiming 2-3x forward and 2x backward speedups.

It is designed for agentic AI running on personal devices, and the core ideas are three-fold.

• gate-driven automatic intra-card CP
• hardware-friendly algebraic reformulation
• TileLang fused warp-specialized kernels

Automatic intra-device CP boosted SM utilization, with particularly strong effects in TP configurations, small models, and long-context workloads.

Also, rather than fusing the entire GDN flow into a single fully-fused kernel, they split it into two kernels for CP efficiency and backward efficiency. They explained that while this incurs additional memory I/O cost at large batch sizes, it delivers better real-world performance in edge device and long-context environments.

The most challenging part was the backward pass, and they stated that they implemented a 16-stage warp-specialized pipeline under extreme on-chip memory constraints, achieving a 2x+ kernel-level speedup.