🔬New Benchmark Finds Top LLMs Fail at Team Agent Work
TL;DR
AgentWorld tests up to 20 AI agents coordinating through 50+ rounds in a shared MMORPG sandbox, and even the best model tops out at 52% task success. Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini and DeepSeek R1-70B all broke down on communication and shared planning.
AgentWorld tests up to 20 AI agents coordinating through 50+ rounds in a shared MMORPG sandbox, and even the best model tops out at 52% task success. Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini and DeepSeek R1-70B all broke down on communication and shared planning.
Key Points
The benchmark covers 100 human-annotated tasks (plus 100 variants) requiring 3 to 20 agents with asymmetric roles to coordinate for 50+ rounds
Best result: 52.0% task success, from testing Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini and DeepSeek R1-70B
A new metric, Causal Collaboration Effectiveness, traces which agent actions actually contributed to the outcome versus wasted effort
Failure modes include communication breakdowns, role confusion and agents losing track of shared plans
The benchmark and code are fully open-source
Why It Matters
Multi-agent demos look impressive in short clips, but a 52% ceiling on genuine long-horizon teamwork is a reality check for anyone building agent swarms for real enterprise workflows.
Quick Facts
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,568 builders reading daily.