How to compress LLM weights by 22% without quality loss

Unweight is a lossless compression technique that touches only model weights, shrinking LLM weights by 15~22% without any quality degradation. In initial results, it achieved roughly 30% compression on MLP weights alone for Llama-3.1-8B, saving about 3 GB VRAM on a full-model basis.

The core problem isn't compute but memory bandwidth. On NVIDIA H100 GPUs, tensor cores compute far faster than memory can feed them, so reducing the number of bytes read from HBM each time a token is generated is the key to improving performance. Unweight decompresses the compressed weights in on-chip shared memory and feeds them directly to the tensor cores, avoiding round trips to slow main memory.

The compression exploits the structure of BF16 weights. The sign and mantissa are essentially random, but the exponent is heavily skewed—the top 16 exponents account for more than 99% of a typical layer. Unweight applies Huffman coding only to this exponent byte, leaving the sign and mantissa untouched.

The scope of application is also selective. It leaves attention, embeddings, and layer norms untouched, applying compression only to the gate, up, and down projections of MLP weights, which account for a large share of memory traffic during decode. Additionally, if even a single rare exponent outside the top-16 palette appears within a 64-weight row, the entire row is stored verbatim, eliminating per-element branching on the hot path.

There isn't just one execution path. Unweight selects among the following 4 pipelines depending on the workload:

• Full Huffman decode: Fully reconstructs the original BF16, then runs matmul via cuBLAS
• Exponent-only decode: Decodes only the exponent byte, cutting preprocess traffic in half
• Palette transcode: Converts to 4-bit palette indices at runtime, reducing traffic further
• Direct palette: Pre-converts to 4-bit format at model load time, so matmul consumes it immediately

At small batch sizes, custom kernel overhead is significant, so full decode + cuBLAS is favorable. Conversely, as batch size grows—such as 256+ tokens—reconstructive matmul takes longer, making the lighter-preprocessing palette/exponent paths faster. Even within the same layer, gate, up, and down projections differ in shape and computation order, so different pipelines may be optimal for each.

Three of the four pipelines use reconstructive matmul. This kernel is split into a producer, which reads compressed data from HBM, and a consumer, which reconstructs BF16 in shared memory and passes it to Hopper's WGMMA tensor-core instruction. The reconstructed weights never exist in main memory—they flow directly from shared memory into computation.

This creates resource contention. Each SM on Hopper has 228 KB of shared memory; reconstructive matmul uses about 227 KB, while the decode kernel also needs about 16 KB for its Huffman lookup table. The two can't coexist on the same SM, so allocating more SMs to decode speeds up preprocessing but slows down matmul. The autotuner therefore also adjusts the SM allocation ratio.

The compression gains are amplified further by exploiting the transformer's layer structure. Unweight classifies layers as hard or easy, and while an easy layer is being computed, it runs preprocessing for the next hard layer in the background on a separate CUDA stream. In particular, since the down projection is consumed last in the MLP sequence, its decode can be finished earliest, producing a large overlap effect.

In the end, Unweight's design philosophy is simple. Rather than picking one fixed solution, it optimizes pipeline selection, matmul variant, and SM allocation all based on empirical measurement, using the fastest combination for each workload. The result is that on H100, more models can be fit into less VRAM while maintaining bit-exact quality.