💡Attention-Free Architecture Proposed in New Paper
A new paper challenges the dominance of attention mechanisms in deep learning
TL;DR
A new paper proposes an attention-free architecture based on Grassmann flows, achieving comparable performance to Transformers. The Causal Grassmann layer could shift the paradigm in neural network design.
A new paper submitted on December 22, 2025, proposes an attention-free architecture based on Grassmann flows, challenging the dominance of attention mechanisms in deep learning. The Causal Grassmann layer, consisting of linear reduction of token states, encoding of local token pairs as two-dimensional subspaces, and fusion of features back into hidden states, achieves validation perplexities within 10-15% of size-matched Transformers on the Wikitext-2 language modeling benchmark. This architecture could impact teams working on natural language processing tasks, offering a more structured approach to neural reasoning with linear complexity scaling.
Key Points
Zhang Chong's paper 'Attention Is Not What You Need' was submitted on December 22, 2025, to arXiv.
The proposed Causal Grassmann layer achieves validation perplexities within 10-15% of size-matched Transformers on Wikitext-2.
A Grassmann-Plucker head on top of DistilBERT outperforms a Transformer head on the SNLI task, with best validation and test accuracies of 0.8550 and 0.8538.
The paper analyzes the complexity of Grassmann mixing, showing linear scaling with sequence length for fixed rank.
Manifold-based designs offer a more structured route toward geometric and invariant-based interpretations of neural reasoning.
Why It Matters
If you're working on natural language processing tasks, the Causal Grassmann layer could offer a more structured approach to neural reasoning with linear complexity scaling. Teams using Transformers may want to explore this new architecture for tasks like language modeling and natural language inference.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,567 builders reading daily.