🔬AgentSWE: No Coding Agent Builds the Agent You Asked For
TL;DR
A new 25-task benchmark tests whether coding agents can build, edit and optimize production AI agents. The authors report that none comes close to delivering what a requirement asks for.
A new 25-task benchmark tests whether coding agents can build, edit and optimize production AI agents. The authors report that none comes close to delivering what a requirement asks for.
Key Points
Covers three stages: create an agent from a requirement, edit production agent codebases, optimize toward a target
Tested across multiple coding agents, models and harnesses
Failure modes: overfitting to visible cases, hollow runtimes behind complete-looking code, unsupported completion claims
Submitted October 9, 2026 by Jiahao Wang, Hongjin Qian and colleagues
Why It Matters
If you let a coding agent build your agent, check that the runtime actually works. The paper names where the demo-to-delivery gap hides.
Quick Facts
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,568 builders reading daily.