🤖DeepSeek V4 Flash Runs on AMD MI300X
MI300X can run DeepSeek with full model in HBM
TL;DR
A new repo shows how to run DeepSeek V4 Flash on AMD's MI300X GPU, including all weights in HBM. It costs half as much as NVIDIA's H100 and handles up to 64 concurrent streams.
DeepSeek V4 Flash now runs natively on AMD's MI300X GPU with no hacks needed. The model fits entirely within the GPU’s HBM, eliminating PCIe streaming or layer offloading. This setup costs roughly half as much at list price compared to NVIDIA's H100 SXM5 and supports up to 64 concurrent streams without errors. Developers running large language models on budget hardware will find this a game-changer.
Key Points
Repository includes Docker Compose stack for DeepSeek V4 Flash on MI300X (29 facts).
Single-stream decode median per-stream: DSpark-7, 168.6 tok/s; Prefill with tuned kernels: 7.9–8.5K tok/s.
Running the model reliably required fixes for FP8 format and MoE routing at high concurrency (facts 8, 14).
MI300X has 192 GB HBM3 and 5.3 TB/s memory bandwidth, twice the capacity of an NVIDIA H100 SXM5.
Official vLLM recipe targets newer AMD GPUs; this repo adds correctness overlays for pinned ROCm nightly.
Why It Matters
If you're running DeepSeek V4 Flash on a budget and need high concurrency, MI300X is now an option. The model fits entirely within HBM without PCIe streaming or layer offloading, making it cost-effective at roughly half the price of NVIDIA's H100 SXM5.
Frequently Asked Questions
Why does this matter?
If you're running DeepSeek V4 Flash on a budget and need high concurrency, MI300X is now an option. The model fits entirely within HBM without PCIe streaming or layer offloading, making it cost-effective at roughly half the price of NVIDIA's H100 SXM5.
What happened?
A new repo shows how to run DeepSeek V4 Flash on AMD's MI300X GPU, including all weights in HBM. It costs half as much as NVIDIA's H100 and handles up to 64 concurrent streams.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 2,588 builders reading daily.