xHC: Residual Stream Expansion Technique Unveiled

xHC has been announced, overcoming the limitations of Hyper-Connections (HC) technology, which expands a Transformer model's Residual Stream into N parallel streams to provide a new memory scaling axis beyond the model's width and depth.

The existing HC method had a problem where, once the number of streams exceeded 4, performance gains diminished while computational cost surged. To address this, xHC introduces two key mechanisms.

• Temporal Feature Augmentation: Enables richer information to be written back to the streams.
• Sparse Residual-Stream Architecture: Uses a sparse structure that updates only k=4 out of N=16 streams, improving computational efficiency while maintaining dense access to the overall state.

In experiments with 18B and 28B MoE models, xHC improved the average downstream score by 4.0 points over existing mHC on the 18B MoE model, with only a negligible increase in FLOPs. Additionally, xHC-Flash, designed to solve the memory traffic problem, reduced memory traffic by about 45% compared to before (73.5C → 40C), securing practicality for large-scale model pre-training.