AI BriefingBreaking the Limits of Ternary LLM Weight Compression: Achieving 1.485 bits/weight with BITCOS
Key point
Intel researchers proposed the BITCOS compression technique, which exceeds previous limits by analyzing the weight distribution of Ternary LLMs.
Details
Intel researchers proposed a new compression technique called BITCOS that overcomes existing information-theoretic limits by leveraging the weight distribution characteristics of Ternary LLMs. The previous practical deployment format, five-trit packing, had a limitation of rounding up to 1.625 bits/weight due to the power-of-two nature of group sizes, but BITCOS improved this with a layout that adapts to the weight distribution.
Key Findings and Technique
The researchers measured 29 Ternary LLM models and discovered that zero accounts for up to 51.5% of all weights. Based on this, they introduced the BITCOS technique, which combines a dense presence bitmap and a compacted presence bitmap. This technique has a cost of 2 - z bits/weight depending on the zero density (z), and showed a more compact compression ratio than previous methods in 26 of the tested models.
Hardware Efficiency and Performance
BITCOS provides unpacking sequences optimized for hardware such as AVX-512 and Intel Xe2 GPU. Measurements showed up to 1.28x faster computation compared to production kernels based on the zero density of actual Ternary models, and end-to-end inference tests confirmed up to 1.18x improvement in decode throughput on CPU and up to 1.27x on GPU.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
