How Low-Bit Inference Makes AI Efficient

Large AI models demand more memory, computation, and power as their performance grows. Dropbox treats low-bit inference as a core efficiency tool in products like Dash to reduce these constraints.

In attention-based models, the costly parts are mainly in two places. One is the linear layers, where repeated matrix multiplication occurs in the attention block, the MLP, and the final output stage, and the other is the attention mechanism itself, whose cost grows significantly with long context.

GPU Tensor Cores and Matrix Cores can handle more computation as precision decreases. Typically, halving precision can nearly double throughput, which makes quantization one of the most important methods for improving performance, memory, and power efficiency at the same time.

Lowering precision isn't just about reducing the number of bits, but about matching what the actual hardware can process. Dropping to 8-bit or 4-bit reduces memory usage, and below 4-bit, bitpacking becomes necessary. Also, formats like FP4 show greater potential energy savings on Blackwell compared to H100.

Conversely, extremely low-bit approaches like binary or ternary could theoretically be more efficient, but current GPUs struggle to fully utilize Tensor/Matrix Cores with them. There is research targeting custom hardware or specialized accelerators, but due to ecosystem support and model quality issues, these haven't seen wide adoption yet.

Formats are broadly divided into pre-MXFP and MXFP. Before MXFP, combinations like A16W4 and A8W8 were mainly used, and 4-bit weights maintained quality through techniques like AWQ or HQQ. When activations and weights have different precisions, dequantization is typically performed first before MMA is executed; this approach is favorable for memory bottlenecks but can actually be slower in compute-bound environments due to the additional computation.

This difference leads to a choice between weight-only quantization and activation quantization.

• weight-only quantization is advantageous for reducing memory bandwidth in small-batch and inference-style workloads.
• activation quantization is more suitable for compute-bound scenarios like long-context prefill or high-throughput serving.

To improve accuracy, linear quantization with grouping is widely used. Grouping shares scale across small blocks like 32, 64, or 128, rather than individual tensor values, reducing quantization error. On the activations side, channel-wise quantization and per-block quantization are commonly used, with the latter—as seen in JetFire and DeepSeek V3—being strong at reducing outlier impact through finer block-level scales.

As non-linear approaches aiming for even lower bits, there are QuiP# and GPTVQ, but these face significant limitations in real deployment due to implementation complexity and the need for custom fused kernels. In contrast, 4-bit linear quantization already shows strong accuracy, and with techniques like HQQ it's easy to apply on-the-fly, making it the most practical choice in the current GPU environment.

Attention optimization is evolving in the same direction. Flash Attention 3 and Sage Attention accelerate attention computation through 8-bit quantization while boosting both throughput and memory efficiency together. In conclusion, the key to low-bit inference isn't simply reducing bits, but how well the format aligns with Tensor Core and inference frameworks.