📊Terminal-Bench 2.0 Scores Swing 6 Points on Resource Limits Alone

Your agent benchmark scores might be measuring your cluster

Resource limits can move a Terminal-Bench 2.0 score by 6 percentage points, and that's more than the gap separating most leaderboard spots. Anthropic's team ran the benchmark on a Google Kubernetes Engine cluster and got scores that didn't match the official leaderboard. The culprit was enforcement: their Kubernetes setup treated per-task resource specs as both a floor and a hard ceiling. If you run agentic evals or pick models based on coding benchmarks, this should bother you. An agent writing code, running tests, and installing dependencies over many turns is a full system test, not a static answer check. Two agents with different RAM and time budgets aren't taking the same exam. Tight limits reward lean, fast code, while generous limits reward brute-force tooling, and a single score hides which one you're looking at. The numbers: infra error rates fell from 5.8% at strict 1x enforcement to 2.1% at 3x headroom and 0.5% uncapped, with the first drop significant at p < 0.001. Success rates rose with headroom, reaching +6 points over 1x at uncapped resources (p < 0.01). On SWE-bench the effect was smaller, about 1.54 points at 5x RAM. Up to roughly 3x, extra resources mostly fix flaky infrastructure. Past 3x, they let agents try approaches that were impossible before.