🤖Transformers: The Engine Behind LLMs Since '17
LLMs hit a bottleneck with transformers, but alternatives are emerging
TL;DR
Since the 'Attention Is All You Need' paper in 2017, transformers have powered major LLMs. But their scalability issues and computational costs are pushing researchers to explore alternatives like sparse attention and power retention.
In 2017, Google's AI researchers published a groundbreaking paper introducing the transformer neural network. Today, these networks underpin every major large language model (LLM) on the market. However, as LLMs grow in size and complexity, transformers are hitting their limits. Subquadratic claims its sparse attention mechanism rivals top-tier models, while Manifest AI is experimenting with power retention to reduce data overload. Liquid AI's hybrid LFMs, combining 20% transformers with liquid neural networks, can run on a Raspberry Pi for $50. The industry is at a crossroads as researchers seek more efficient and sustainable alternatives.
Key Points
Subquadratic's sparse attention mechanism rivals top-tier models on a handful of tasks
Manifest AI uses power retention to store only relevant data for LLM tasks
Liquid AI builds LFMs with hybrid architecture: 20% transformers + 80% liquid neural networks
Liquid AI's models can run on Raspberry Pi, costing just $50 per unit
The company has seen almost 34 million downloads of its models to date
Why It Matters
If you're working with massive LLMs like GPT-4 or Claude, the limitations of transformers are becoming painfully clear. Subquadratic's sparse attention and Manifest AI's power retention offer promising alternatives that could reduce computational costs and improve efficiency. Liquid AI’s hybrid LFMs show how smaller, more energy-efficient models can still tackle complex tasks.
Frequently Asked Questions
Why does this matter?
If you're working with massive LLMs like GPT-4 or Claude, the limitations of transformers are becoming painfully clear. Subquadratic's sparse attention and Manifest AI's power retention offer promising alternatives that could reduce computational costs and improve efficiency. Liquid AI’s hybrid LFMs show how smaller, more energy-efficient models can still tackle complex tasks.
What happened?
Since the 'Attention Is All You Need' paper in 2017, transformers have powered major LLMs. But their scalability issues and computational costs are pushing researchers to explore alternatives like sparse attention and power retention.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 2,763 builders reading daily.