Skip to content
Source: ankitg.me

🔧DeepGrove Used a Coding Agent to Run a Ternary LLM on FPGAs at 213 Tokens/s

Coding agents just made FPGA inference a weekend project

TL;DR

DeepGrove let a coding agent grind through hundreds of FPGA experiments for $10,000 in API tokens. Throughput went from 6.43 to 213.13 tokens/s. If you're stuck writing C++ and Verilog for low-level hardware, this is the workflow shift to watch.

A cloud coding agent took a ternary-weight LLM from a bit-exact baseline at 6.43 tokens per second to 213.13 tokens per second, a 33x gain, on AWS F2 FPGA instances. DeepGrove's founders kicked off the run and nudged the agent periodically over several days in Conductor, mostly with prompts like "go for 2x faster, you got this" plus the occasional ML hint such as "time to consider speculative decoding." Why should you care? Writing low-level C++ and Verilog for FPGAs is cumbersome, and that's exactly the barrier that keeps most teams off this hardware. The result suggests coding agents can open up new domains, letting a generalist engineer drive hardware work they'd never have attempted by hand. The catch: the agent needed human direction to stay on track. The numbers: roughly 3M lines of code and experiment history, 30M lines of output parsed, hundreds of experiments, and about $10,000 burned across Claude and OpenAI tokens. The optimizations that moved the needle included a head loader reading both memory banks at once, a runtime expert-prefetch policy, and a router-job timing fix. The F2's 16 GB of HBM is tight, which is why the ternary architecture matters here.

DeepGrove Used a Coding Agent to Run a Ternary LLM on FPGAs at 213 Tokens/s — ankitg.me

Key Points

1

The benchmark rose from 6.43 to 213.13 tokens per second, a 33.13x improvement over the first full-model bit-exact baseline on FPGA hardware.

Keep reading — it's free

Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.

Also get
or

Already a member? Sign in

FPGAternary-weightsAWS-F2LLM-inferencecoding-agents

Comments

Subscribe to join the conversation...

Be the first to comment