Parameters cut in half

Parcae is a new architecture that directly addresses the stability problems of looped language models.

The core idea is to interpret the residual dynamics of the recurrent structure as a stable LTI system, and to turn input injection into a continuous parameterization + ZOH/Euler discretization. By constraining A to be a negative diagonal matrix, the design keeps the spectral radius below 1, greatly reducing the residual state explosion and loss spikes commonly seen in existing looped models.

The experimental results are concrete as well.

• Up to 6.3% improvement in validation perplexity compared to existing large-scale looped recipes
• 770M Parcae achieves quality comparable to a 1.3B Transformer trained on the same data
• Across the 140M / 370M / 770M / 1.3B scales, it records better perplexity and downstream scores than parameter- and data-matched Transformers

It also presents important conclusions from a scaling perspective.

• When the compute budget is fixed, it is better to increase recurrence and adjust the data together
• The optimal recurrence and token budget each follow a power law
• As a result, looped Parcae forms a denser Pareto frontier than fixed-depth models

In conclusion, this work presents a path to boosting quality without increasing parameters in on-device or memory-constrained environments, and the code and models have also been released.