🤖FreeToken Boosts MoE Models on Consumer Hardware
Edge AI just got a major upgrade
TL;DR
FreeToken, a new open-source inference engine, boosts MoE models on consumer hardware. It achieves 3-4x faster decode and 6-30x faster prefill, making frontier-class reasoning agents more accessible.
FreeToken, an open-source inference engine from UC Berkeley and MIT, is revolutionizing how MoE models run on consumer hardware. It replaces static expert offloading with a dynamic co-scheduling policy, significantly speeding up decode and prefill processes. This means developers can now run advanced AI models on affordable consumer GPUs, reducing the need for expensive cloud services. FreeToken achieves 3-4x faster decode and 6-30x faster prefill on equivalent MoE models, making it a game changer for anyone looking to self-host AI agents. It supports RTX 30, 40, and 50 series GPUs on Linux and Windows, with benchmarks showing Qwen3.6-35B running at ~39 tokens/sec on an 8GB RTX 4060 laptop.
Key Points
FreeToken introduces a dynamic co-scheduling policy called the q* policy, splitting token computation between CPU cores and GPU tensor cores based on real-time interconnect throughput.
FreeToken achieves 3-4x faster decode and 6-30x faster prefill on equivalent MoE models compared to other runtimes, making it a major improvement for consumer hardware.
The system supports RTX 30, 40, and 50 series GPUs on Linux and Windows, with benchmarks showing Qwen3.6-35B running at ~39 tokens/sec on an 8GB RTX 4060 laptop.
FreeToken integrates semantic anchor checkpointing, allowing reuse of existing sub-sequence states when editing intermediate tool arguments or injecting external execution output.
The project aims to bridge the gap between frontier MoE models and consumer-grade hardware, making advanced AI more accessible to developers.
Why It Matters
If you're running MoE models on consumer hardware, FreeToken is a must-try. It achieves 3-4x faster decode and 6-30x faster prefill, making it a game changer for anyone looking to self-host advanced AI agents. Developers can now run Qwen3.6-35B at ~39 tokens/sec on an 8GB RTX 4060 laptop, reducing the need for expensive cloud services.
Frequently Asked Questions
Why does this matter?
If you're running MoE models on consumer hardware, FreeToken is a must-try. It achieves 3-4x faster decode and 6-30x faster prefill, making it a game changer for anyone looking to self-host advanced AI agents. Developers can now run Qwen3.6-35B at ~39 tokens/sec on an 8GB RTX 4060 laptop, reducing the need for expensive cloud services.
What happened?
FreeToken, a new open-source inference engine, boosts MoE models on consumer hardware. It achieves 3-4x faster decode and 6-30x faster prefill, making frontier-class reasoning agents more accessible.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,382 builders reading daily.