H100 3GB Savings

Cloudflare has released Unweight. The goal is to make LLM weights smaller while maintaining bit-identical output, reducing the memory bandwidth bottleneck in H100-based inference.

The core idea is compressing only the exponent byte of BF16 using Huffman coding. The sign and mantissa are left untouched, and only the heavily skewed exponent is reduced, achieving about 30% MLP weight compression. This method is selectively applied to MLP layers such as gate / up / down projection.

As actual results, a 15-22% reduction in overall model size and about 3GB VRAM savings were reported for Llama-3.1-8B. Cloudflare explains that this allows more models to be loaded on the same GPU, reducing inference cost and latency.

The execution path is divided into four types depending on the workload.

• full Huffman decode + cuBLAS
• exponent-only decode
• palette transcode
• direct palette

The cuBLAS path is advantageous for small batches, while reconstructive matmul and lighter preprocessing paths are advantageous for large batches. So the runtime performs autotuning by batch size and weight matrix to select the optimal combination.

Along with this post, the technical paper and GPU kernels have also been open-sourced.