AI BriefingRTPurbo: Converting Full Attention into Sparse
Key point
RTPurbo, a technique that converts a Full Attention LLM into a highly efficient Sparse model with only a few hundred short training steps, has been unveiled.
Details
Existing efficient long-context inference methods have relied on sparse training or token eviction approaches, resulting in trade-offs among efficiency, training cost, and accuracy.
RTPurbo leverages the fact that Full-attention LLMs already possess intrinsic sparsity, converting a model into a highly efficient Sparse model with just a few hundred training steps.
This technique is designed based on three key observations:
- Only a subset of Attention heads are actually necessary for processing long contexts.
- Long-range retrieval is governed by a low-dimensional subspace, enabling efficient token retrieval with a 16-dimensional indexer.
- The useful token budget varies by query, making dynamic top-p selection more suitable than a fixed approach.
RTPurbo implements Sparse attention by maintaining a Full KV cache only for retrieval heads and introducing a lightweight token indexer. Experimental results show that at 1M context, Prefill speed improved by up to 9.36x and Decode speed by about 2.01x, while accuracy was preserved with almost no loss.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Stanford and NVIDIA Release 'Contrastive Language Model' CLM with 9x Faster Inference