Attention-Free Grassmann Flow Architecture Achieves Perplexity Within 10-15% of Transformers on Small-Scale Benchmarks

• Proposes an attention-free architecture using Causal Grassmann layers that encode local token pairs as two-dimensional subspaces via Plucker coordinates instead of forming L by L attention matrices.
• On the Wikitext-2 language modeling benchmark, models with 13 to 18 million parameters achieve validation perplexities within about 10 to 15 percent of size-matched Transformers.
• On the SNLI natural language inference task, a Grassmann-Plucker head on top of DistilBERT achieves best validation and test accuracies of 0.8550 and 0.8538, compared to 0.8545 and 0.8511 for a Transformer head.
• The design offers linear scaling in sequence length for fixed rank and aims to provide a more structured route toward geometric and invariant-based interpretations of neural reasoning.

Technical Mechanism

• Information propagates through controlled deformations of low-rank subspaces over multi-scale local windows.
• The core computation lives on a finite-dimensional manifold rather than in an unstructured tensor space.
• The Causal Grassmann layer linearly reduces token states, encodes local pairs, and fuses geometric features back into hidden states through gated mixing.