🤖Mini-AGI Continual Learner on 8GB GPU
A tiny AGI that learns on the fly
TL;DR
A new model learns continually from a single stream of data on an 8GB GPU, showing the potential for continual learning without catastrophic forgetting. It's a small toy model, but the implications for hardware constraints are huge.
A new mini-AGI model is learning continually on an 8GB GPU, assembling its own architecture as it trains. This model, designed to run on modest hardware, demonstrates that continual learning from a single stream of data without catastrophic forgetting is possible. For developers, this means that almost anyone can train their own version of the model, regardless of hardware limitations. The model's architecture is unique, with parameters bounded by free disk space and the ability to grow and prune its capacity during training. It reads through the same code path it serves on, making it highly adaptable to different tasks. The model's capabilities are limited by the hardware, data quality, and training time, but it opens up new possibilities for continual learning on resource-constrained devices.
Key Points
Mini-AGI trains from scratch on a single 8GB VRAM GPU, assembling its own architecture.
The model stores weights as ordinary files on disk, paging them onto the card as needed.
Parameter count is bounded by free disk space, not VRAM, allowing for continual learning.
Model grows new capacity while training and prunes unused weights, adapting to data.
Architecture passes characters through dense prelude blocks and a recurrent block up to 24 times.
Why It Matters
If you're working with limited hardware, this model shows that continual learning is possible without high-end GPUs. For example, a developer with an 8GB GPU can now train a model that continually learns from data streams, adapting its architecture as it goes. This is a game-changer for teams with resource constraints, as it opens up new possibilities for AI development.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,484 builders reading daily.