128x compression

Macrocosmos has released the ResBM (Residual Bottleneck Models) paper.

The core idea is to insert a residual encoder-decoder bottleneck between pipeline boundaries to reduce inter-stage communication, while preserving a low-rank identity path.

The paper claims the following:

• Achieves 128× activation compression
• Convergence degradation compared to the uncompressed baseline is not significant
• The strongest compression results come when using Muon

The goal is to reduce communication bottlenecks in distributed environments, especially in low-bandwidth pipeline-parallel training and decentralized / internet-grade training.