Parcae, a stable looped model that achieves higher performance with fewer parameters

Parcae is an architecture designed to stably train looped language models, aiming for similar or better quality than same-sized Transformers while using fewer parameters. Notably, 770M Parcae came close to the performance of a 1.3B Transformer trained on the same data, demonstrating new possibilities for parameter efficiency.

Existing looped models suffered from unstable training due to residual state explosion and loss spikes caused by their recurrent structure. Parcae analyzed this from the perspective of a linear dynamical system, designing the model to keep the spectral radius, which governs the stability of the recurrent block, below 1.

The architecture divides the Transformer into prelude ($\mathcal{P}$), recurrent ($\mathcal{R}$), and coda ($\mathcal{C}$).

• The prelude converts the input into a latent state
• The recurrent block reuses the same layer multiple times to update the state
• The coda produces the final output

For stability, the input injection parameters are represented in continuous form as A, B, which are discretized using ZOH and Euler methods. Additionally, they are constrained to a negative diagonal matrix of the form A = Diag(-exp(log_A)), preventing the recurrence from diverging.

In experiments, Parcae reduced validation perplexity by up to 6.3% compared to existing large-scale looped recipes. Also, even when RDM was used as-is, applying Parcae's constraints stabilized training, and adding further tricks improved performance even more.

The work also produced important results on scaling. The authors presented the first scaling law showing that looping and data must be increased together to be compute-optimal, and observed that both the average recurrence and token budget follow a power law. This result creates a stricter Pareto frontier than fixed-depth models, showing that better quality can be achieved with the same resources.

Finally, the authors have released the training code and models, laying the groundwork for further exploration of parameter-reuse-based models in on-device or heavily memory-constrained environments.