#linear-attention
The latest AI and developer news about #linear-attention, with the original source and a short summary.
The latest AI and developer news about #linear-attention, with the original source and a short summary.
The model resolves slow-channel glare and write sensitivity bottlenecks to train 2.7x faster than Splash Attention at 32K context.
PACE uses periodic rounding and channel-aligned scaling to maintain accuracy at four bits, matching 8-bit performance on models up to 35B parameters.
The Raon-1 mini preview, a 30B-class MoE model, has been released under the Apache 2.0 license.
This analyzes the mathematical structure and operating principles of various linear attention variant models, including DeltaNet.
Nvidia has announced SANA-Video 2.0, which accelerates high-resolution long-form video generation through hybrid attention technology.
HOLA, a model that introduces a hippocampal structure to prevent information loss occurring during the compression process of linear attention, has been released.
Qwen accelerated linear attention by up to 3x with TileLang-based FlashQLA.
FlashQLA optimizes GDN Chunked Prefill with CP and fused kernels, achieving up to 3x speedup over FLA Triton.
Moonshot released FlashKDA, which is up to 2.22x faster than Triton on H20.