DeepMind's Looped Transformer

Elastic Looped Transformers (ELT) is a recursive generative model that repeatedly reuses the same block instead of a deep Transformer stack. This aims to significantly reduce the number of parameters while maintaining visual generation quality.

The core is a recursive structure based on weight sharing. Instead of each layer having different weights as in typical generative models, the same Transformer block is looped multiple times during computation, increasing parameter efficiency.

For training stability, Intra-Loop Self Distillation (ILSD) is also proposed. From the teacher configuration trained with the maximum number of loops, the intermediate student configurations are distilled within the same single training process, aligning output consistency across depths.

This approach makes it possible to create an elastic model family tailored to different compute budgets with just a single training run. In other words, Any-Time inference is possible, adjusting quality and cost according to the number of loops while keeping the same number of parameters.

The paper mentions a 4x parameter reduction under iso-inference-compute conditions, and states that it achieved FID 2.0 on class-conditional ImageNet 256×256, and FVD 72.8 on class-conditional UCF-101. In summary, ELT is an approach that pushes the balance point between performance and efficiency further in visual generation.