Hugging Face Releases Block Sparse Library

Hugging Face has released the pytorch_block_sparse extension to solve the problem of inefficient sparse algebra computation in PyTorch. This library aims to reduce model size and increase inference speed, supporting the construction of low-cost production environments.

Built on NVIDIA CUTLASS, this extension implements high-performance block sparse matrix multiplication using C++ CUDA templates. Users can easily replace existing torch.nn.Linear with BlockSparseLinear, and using BlockSparseModelPatcher allows instant conversion of existing models into sparse models without modifying the source code.

Key Performance and Features:

• Efficiency: When applying 75% sparsity, memory usage is reduced by 4x, and computation speed is improved by about 2x compared to dense models.
• Combinability: Can be combined with distillation and quantization techniques to maximize model optimization effects.
• Future Plans: Aims to optimize for NVIDIA Ampere Tensor Core and develop tools to find optimal sparse patterns during the training process.