ParaRNN: Large-Scale Nonlinear RNNs That Can Be Trained in Parallel

Apple researchers presented ParaRNN, a framework that parallelizes the training of nonlinear RNNs for large language models (LLMs), achieving 665× faster speed than traditional sequential training methods, and for the first time showing that a traditional RNN at the 7B parameter scale can match transformer and Mamba2 in language modeling performance. The paper was accepted as an Oral at ICLR 2026, and the code was also released as open source.

RNNs still have strengths at the inference stage. Unlike attention-based models, they don't need to store and reference the entire long context, so token generation is close to O(1), and the memory and compute burden is also small. The problem is that during training, parallelization along the sequence-length direction isn't possible, making large-scale scaling difficult.

Existing parallelizable recurrent models have restricted the recurrence to be linear. Selective state space models (SSMs) like Mamba take the form h_l = A_l h_{l-1} + B_l x_l, which can be computed via parallel scan, but this comes at the cost of reduced expressiveness.

ParaRNN circumvents this limitation using Newton's method. It treats the entire hidden state as a single system of nonlinear equations, linearizes it at each iteration using the Jacobian, then converts it into an SSM-form problem and solves it in parallel. The researchers implemented this as ParaGRU and ParaLSTM, making the Jacobian diagonal and block-diagonal respectively, and boosted efficiency with custom CUDA kernels.

• Preserves the core nonlinearity while only parallelizing the training
• In experiments, convergence was generally achieved with 3 Newton iterations
• Structured Jacobians and GPU optimization made large-scale training possible