ParaRNN Released, Accelerating Nonlinear RNN Training by 200x

To address the sequential bottleneck of nonlinear RNNs, the open-source PyTorch/Triton library ParaRNN, which combines Newton-Raphson iterations, has been released. This tool dramatically improves training speeds for major nonlinear architectures such as sLSTM, RWKV-7, CfC, and Titans.

Key Performance and Features

• Training Speedup: Achieved approximately 200x wall-clock time reduction compared to sequential BPTT for the CfC model on long sequences (T=2048) (decreasing from 643ms to 2.79ms on an RTX 3060).
• Convergence Stability: The number of Newton iterations remains 3 or fewer even with long contexts of 131,072 tokens, exhibiting convergence characteristics identical to sequential unrolling.
• Supported Architectures: Provides fused kernels for sLSTM, RWKV-7, CfC, Titans, Modern Hopfield, M²RNN, and others.
• Production Compatibility: Supports full compatibility with torch.compile (0 graph breaks), deterministic VJP implementation, and vLLM plugins.