Recurrent Looped Transformer Released: Architecture That Expands Depth with Sequence Length

Recurrent Looped Transformer (RLT) is a new architecture combining a causal encoder and a recurrent decoder, passing the final hidden state and layer-wise Sliding Window Attention (SWA) caches across all prompt and response tokens. As sequences grow longer, the depth of the decoder's recurrent path increases linearly, providing infinite temporal paths; however, this refers to scalable paths within a sequence and does not imply increased computation per token.

RLT features model-hardware co-design and model-RL algorithm co-design. The encoder processes known tokens in parallel to build global key-value memory, while the decoder performs independent updates and reuses memory. Notably, during RL training, the entire history is reconstructed using current parameters, eliminating structural discrepancies between sampling and replay.

In the specific implementation, 48 encoder layers and 48 decoder layers are used, with each token executing 96 logical blocks. After t tokens, the temporal path traverses 48t decoder blocks. However, improvements in actual inference performance, hardware efficiency, and RL scaling remain challenges that need to be verified through empirical evidence.