Microsoft's Full-bandwidth Transformer

Existing autoregressive Transformers excel at horizontal access between tokens, but the vertical feedback channels along the model's depth are extremely narrow. During decoding, only sampled tokens are fed back into the input, while the hidden states from the top layer are discarded.

Full-bandwidth Transformer addresses this issue by introducing a Latent Feedback mechanism. At each decoding step, the top-layer hidden state from the previous step is combined with the sampled token embedding via a Gated Linear Unit and passed back as the next input. This allows non-verbalized computational results to be reused within the model's depth budget.

A key advantage of this structure is that it preserves the existing Transformer architecture, KV Cache, and language modeling objective function. Training a 1B parameter model on up to 400B tokens resulted in significant improvements in mathematics, coding, and instruction-tuned performance.

Notably, Full-bandwidth Transformer achieves performance comparable to training approximately 1.5x more tokens than existing models, while incurring very little overhead during decoding. It also demonstrates the ability to generate shorter reasoning traces while maintaining the same or higher accuracy.