🤖4-bit Model Runs on Mac with 48 GB RAM
48 GB Macs can now run a 104 GB model
TL;DR
A 104 GB 4-bit model now runs on Macs with 48 GB RAM, thanks to streaming from SSD. Key: 110 GB disk space, 4-bit quantization, and auto-sized memory usage.
A new 4-bit model, weighing in at 104 GB, can now run on Macs with just 48 GB of RAM. This is a big deal for developers with limited hardware, as it means they can run large models without needing massive amounts of RAM. The model streams from SSD, allowing it to run in whatever memory is available, down to an 8.1 GB floor. It requires a one-time 104 GB download and ~110 GB of free disk space. The model's memory usage scales based on the number of tokens processed, from 8 GB at low tok/s to 33 GB at high tok/s.
Key Points
4-bit model requires 104 GB SSD storage, 110 GB free disk space.
Model runs on Mac with 48 GB RAM, 8.1 GB memory floor.
Memory usage scales from 8 GB to 33 GB based on tok/s.
Weights are stored on disk, streamed as needed during runtime.
Model supports Ollama and OpenAI chat/generate endpoints.
Why It Matters
If you're developing on a Mac with limited RAM, this model allows you to run large AI models without needing a beefy machine. The 104 GB model runs on just 48 GB of RAM, thanks to streaming from SSD. This is a game-changer for teams with resource constraints, enabling them to experiment with large models without breaking the bank.
Frequently Asked Questions
Why does this matter?
If you're developing on a Mac with limited RAM, this model allows you to run large AI models without needing a beefy machine. The 104 GB model runs on just 48 GB of RAM, thanks to streaming from SSD. This is a game-changer for teams with resource constraints, enabling them to experiment with large models without breaking the bank.
What happened?
A 104 GB 4-bit model now runs on Macs with 48 GB RAM, thanks to streaming from SSD. Key: 110 GB disk space, 4-bit quantization, and auto-sized memory usage.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,436 builders reading daily.