Hugging Face Supports Ulysses SP

As long-context processing becomes critical for document analysis, code understanding, RAG, and more, techniques for training sequences ranging from hundreds of thousands to millions of tokens have become essential. However, existing attention mechanisms have a limitation in that memory usage increases quadratically with sequence length.

Ulysses Sequence Parallelism (SP) addresses this problem by distributing attention computation across multiple GPUs through attention head parallelization. The main mechanism works as follows:

• Sequence Sharding: The input sequence is divided and allocated across multiple GPUs.
• All-to-All Communication: After the QKV projection, data is redistributed so that each GPU handles a specific subset of attention heads for the entire sequence positions.
• Local Attention: Each GPU computes attention for its assigned heads using FlashAttention or similar methods.
• Data Restoration: Another All-to-All communication restores the data to the original sequence sharding format.

Hugging Face has integrated this technology into Accelerate, Transformers Trainer, and TRL's SFTTrainer, making it easier for developers to train long-context models.