AI BriefingReformer: A Transformer That Handles 500,000 Tokens
·2020.07.03 09:00
Key point
Reformer uses techniques like LSH attention to minimize memory usage while processing long sequences of up to 500,000 tokens.
Details
Reformer is a memory-efficient transformer model designed to process up to 500,000 tokens at once, surpassing the limitations of existing BERT models (512 token limit). This model can train on long sequences with less than 8GB of RAM.
The key innovative technologies are as follows:
- Reformer Self-Attention Layer: Implements efficient self-attention using LSH (Locality Sensitive Hashing).
- Chunked Feed Forward Layers: Optimizes the time-memory tradeoff of large feed forward layers.
- Reversible Residual Layers: Dramatically reduces memory consumption during training through a smart residual structure.
- Axial Positional Encodings: A positional encoding method usable even with very long input sequences.
Through these structural improvements, the model shows superior performance compared to existing models in NLP tasks that require processing long context, such as summarization and question answering.
