🤖GPT-6 Astra Tops StarSkirmish Bench v0.2
GPT-6 Astra leads, but Sol offers best value
TL;DR
GPT-6 Astra leads StarSkirmish Bench v0.2, scoring 45. But GPT-6.1 Sol, at just 10% of Astra's cost, offers the best value. Human-written bots still outperform all models.
GPT-6 Astra leads StarSkirmish Bench v0.2 with a score of 45, a clear step ahead of GPT-6.1 Sol in second. But Sol offers unmatched value, costing just over a tenth of Astra's price per run. This matters for developers choosing between models based on performance and cost. StarSkirmish measures how well LLMs can write bots for StarCraft: Brood War in one hour, with scores reflecting win rates against human-written bots. The strongest human-written bots still outperform all models, winning 93% of their games against the best LLMs.
Key Points
GPT-6 Astra scores 45, a clear step ahead of GPT-6.1 Sol's 35.
GPT-6.1 Sol offers unmatched value, costing just over a tenth of GPT-6 Astra.
Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 score between 37 and 39.
LLMs start from a skeleton bot that only mines, with access to three demo bots and two standard bot libraries.
Games are Protoss versus Protoss on three ladder maps: Heartbreak Ridge, Benzene, and Destination.
Why It Matters
If you're choosing between GPT-6 Astra and GPT-6.1 Sol, Sol offers unmatched value. It scores second at just over a tenth of Astra's cost per run. But Astra's 45 score is a clear step ahead, making it the top performer. This affects teams deciding on LLMs based on cost and performance.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,568 builders reading daily.