LittleBit-2: Maximizing Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometric Alignment

LittleBit-2 is a method that simultaneously improves compression efficiency and performance of large language models (LLMs) in the sub-1-bit regime. A 70B model requires about 140GB of VRAM in FP16, and even a fully 1-bit model demands about 15GB, which is a burden for deployment on edge and consumer devices.

Samsung's existing LittleBit decomposed weight matrices into low-rank binary latent factors, achieving compression down to as little as 0.1 bits per parameter (bpp). However, at the 1bpp range, its performance fell short of OneBit, with a perplexity of 9.08 on Llama-2 7B compared to OneBit's 8.36.

The cause was a geometric structure mismatch in the latent factors initialized via standard SVD. Singular vectors had a 'spiky' structure concentrated on a few coordinates, which did not align with the binary hypercube targeted by binary quantization, and values near zero had their signs easily flipped by small perturbations. In addition, a few outliers caused the shared floating-point scale to overestimate the overall value range.

LittleBit-2 addresses this issue with the following two techniques:

• Internal Latent Rotation: Rotates the latent factors to align them with the diagonal direction of the binary hypercube.
• Joint Iterative Quantization (Joint-ITQ): Jointly and iteratively quantizes the latent factors to produce a distribution suited to binary representation.

This process incurs no additional cost at inference time. It improved performance across the entire 1 to 0.1bpp range on Llama-2 and Llama-3, matching or surpassing major 1-bit baselines and setting a new state of the art in this regime.