4-bit floating point FP4

Unlike general float, which has grown up to 64-bit, neural networks want lower-precision floating point in order to fit more parameters. That's why ultra-low-precision formats like FP8 and FP4 are used.

FP4 uses 1 of its 4 bits for sign, and splits the remaining 3 bits between exponent and mantissa. The possible combinations are E3M0, E2M1, E1M2, and E0M3, among which E2M1 is the most common and is also supported by Nvidia hardware.

Values are generally interpreted in the form (-1)^s × 2^(e-b) × (1 + m/2), where bias is used to represent positive/negative exponents. The bias affects the range, but does not itself change the relative spacing.

The character of each format is as follows.

• E3M0: entirely exponent, so it's close to a log scale.
• E0M3: entirely mantissa, so it's close to a linear scale.
• E1M2, E2M1: compromises between the two, both with non-uniform spacing.

In particular, when e = 0, an exception rule applies: m = 0 becomes 0, and m = 1 becomes 0.5. Also, just like full float, FP4 has two zeros, +0 and -0.

As an example, the 16 representable values of E2M1 come out as follows.

• Positive: +0, +0.5, +1, +1.5, +2, +3, +4, +6
• Negative: -0, -0.5, -1, -1.5, -2, -3, -4, -6

The article also shows Python code that generates this table using the pychop library. Finally, it notes that FP4 isn't the only option—various 4-bit float formats exist—and previews that the next article will cover NF4.