WarpState Reveals Dual-Memory Architecture for Long-Context LLMs

WarpState, an experimental architecture that maintains compact recursive memory instead of token-level KV caches for long-running LLMs, has been released. The model combines two associative matrix memories (Fast/Slow) with different decay rates, alongside exact causal self-attention within 128-token chunks.

Core Architecture and Memory Structure

• Dual Memory System: Separates short-term and long-term information via Fast memory (β≈0.90) and Slow memory (β≈0.99). An input-dependent gate dynamically determines the ratio of local attention to recursive memory usage.
• Fixed-Size State: The recursive state does not grow with the number of processed tokens, using only approximately 49 MiB of persistent tensor cache for a 2B-parameter model.
• Stability Guarantee: Bounded writes and convex combination updates constrain memory values to the [-1, 1] range, ensuring numerical stability.

Performance and Limitations

• Reference Scale: A preset with approximately 2.005B parameters, 28 logical layers, and a 128-token chunk size is provided.
• Implementation Note: The parallel forward/training implementation has quadratic complexity with respect to the number of chunks, meaning it is not currently linear-time for arbitrary long sequences. Future application of optimized scan/tiled recurrence is required.
• Validation Status: Causal ordering was confirmed with a maximum absolute error within 1.79e-7 between serial/parallel paths in an FP32 environment, but discrepancies occur in BF16 environments due to precision differences. Validation data from the past 150M-parameter model should be treated as a historical report that cannot be independently reproduced.

Future Plans

The research team plans controlled experiments comparing local attention alone, single-memory, and WarpState dual-memory models under identical conditions. This will evaluate how much useful information fixed-size recursive states preserve per byte and FLOP.