AI BriefingRLT: Proposed Recurrent Decoder Structure with Computational Depth Increasing with Token Length
Key point
The RLT structure, where the recurrent path lengthens and computational load increases during token generation, along with preliminary experimental results, has been released.
Details
RLT (Recurrent Looped Transformer), proposed by Yifan Zhang et al., introduces a decoder structure where the recurrent path lengthens proportionally to the number of generated tokens to overcome the fixed computational depth limitations of existing Transformers. The encoder generates global memory through parallel processing, while the decoder processes sequentially by gate-summing the previous hidden state.
Preliminary experimental results show a significant improvement in accuracy for the modular arithmetic (excluding parentheses) task in an 8-layer model, rising from a baseline of 20.57% to 95.44%. However, a sharp drop in performance was observed in length extrapolation experiments, and validation regarding hardware efficiency or RL performance has not yet been conducted.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Researchers Release 'Turnbench', a Turn-Taking Evaluation Benchmark by Dialogue TypeStepAudio 3 Gen Achieves SOTA in TTS and Multimodal Audio Generation via RVQ-Based Discrete Generation
Anthropic Reveals Mathematical Framework Showing Induction Heads Are Key to In-context Learning in 2-Layer Transformers