Finding the Wasted Bits: How Much Information Do LLM Weights Actually Carry

Remeasuring the actual information content of LLM weights using Shannon entropy showed that in BF16, only an average of 10.6 bits out of 16 bits were filled with information. Nearly all of the actual slack was concentrated in the exponent, while sign and mantissa were almost completely full.

Comparing open-weight models from multiple labs across scales from 0.6B to 1.4T, this pattern barely changed. The weights appeared statistically almost i.i.d., and the absolute value distribution was largely sharply concentrated around 2^-7 to 2^-6. Very small values (|w| < 2^-16) were separated into a distinct small bin, so that dead experts in some MoE models wouldn't distort the mean and standard deviation. When normalized, the distributions of different models overlapped into nearly the same curve.

• BF16: Only about 2.6 bits out of 8 bits were used for the exponent, and its width stayed very narrow at about 0.05 bits, even across different models.
• FP8: About 6.5 bits out of 8 bits were used, raising efficiency, but slack still remained mainly on the exponent side.
• FP4: With only 2 bits of per-element exponent, it became hard to handle the distribution, requiring block-wise scale as seen in MXFP8, MXFP4, NVFP4, and INT4.

In sub-byte formats, the slack shifts from element bits to scale. MXFP8/MXFP4 use 32-element blocks with E8M0 scale, while NVFP4 uses a structure with 16-element blocks with E4M3 scale plus an FP32 per-tensor scale. Qwen block-FP8 uses BF16 scale for 128-element blocks, and INT4 absorbs residual information through group-wise scale.

As a result, MXFP4 leaves a bit of room, with per-block scale using only 0.03 to 0.10 bits/element, while INT4 and NVFP4 fill element bits almost to the limit, then pack the remaining roughly 0.26 bits/element of information into the scale. Overall entropy utilization rises to about 66% for BF16, 80% for FP8, and 93% for the FP4 family.

The conclusion is clear: LLM weights still have 7 to 30% of spare bits remaining, and as long as inference stays bound by memory bandwidth, further reducing this slack will require compression-and-reconstruction approaches that turn memory into computation, going beyond fixed-length formats.