AI Briefing#flash-attention
The latest AI and developer news about #flash-attention, with the original source and a short summary.
The latest AI and developer news about #flash-attention, with the original source and a short summary.
The new kernel leverages native matrix core shapes and SLM architecture, delivering significant speedups on ARL-H, PTL-H, and B70 Pro platforms.
The Tang project uses an IR-based compiler to derive GPU kernels, achieving 5-12% faster decode steps for 80B MoE models at high concurrency and reducing 44k-token prefill time by a third on H200 GPUs.
llama.cpp has improved prompt processing speed by up to 31% through vectorization using F16C intrinsics.
The VIDRAFT team boosted Gemma inference speed to 510.58 TPS on an A10G while maintaining quality.
llama.cpp has rolled out an update adding Flash Attention support on Intel SYCL and improving Qwen model inference speed.
An open-source kernel has been released that enables efficient training of MiniMax Sparse Attention (MSA) on Hopper and Blackwell GPUs.
A GPU programming guide covering Blackwell architecture optimization techniques, based on a Carnegie Mellon University course, has been released.
MiniMax has released MSA, a Sparse Attention technology that supports processing long contexts of over a million tokens.
Stanford's CS336 course covers the entire process of LLM development, from data curation to Triton-based FlashAttention2 implementation and DPO alignment.
Parallax attention has been proposed, overcoming the limitations of existing LLA while achieving performance on par with FlashAttention.
llama.cpp has applied an update that reduces VRAM usage by processing the KQ mask in f16 when using Flash Attention.
In llama.cpp, an optimization was made to reduce VRAM usage by using an f16 mask during Flash Attention.
The llama.cpp b9158 release includes Flash Attention fixes for AMD RDNA3 GPUs.
Selective hierarchical attention was about 17x faster than standard attention at 512K context.
When an agent read papers and forks before writing code, llama.cpp optimizations improved significantly.
Low-bit inference reduces memory and computation to make AI models faster and cheaper.
Rebellions unveiled a high-efficiency LLM serving technology compatible with vLLM by optimizing FlashAttention and other techniques for NPUs.
Hugging Face has unveiled Kernel Hub, which lets developers easily load optimized compute kernels to boost model performance.
Covers quantization, Flash Attention, and architectural innovations for efficient production deployment of LLMs.