Skip to content
Source: anthropic.com

🧪Safety Evals Are Easy to Game, Researchers Find, and Most Can't Be Reproduced

Your model's safety score might be meaningless

TL;DR

Researchers found AI safety evals are easy to game, don't always measure what they claim, and are poorly documented. Results can flip based on question wording or answer order. If you ship models or rely on eval scores, treat them as a starting point, not proof.

Safety evals are the tests teams use to decide whether a model is safe to deploy. Researchers just found that many of them are easy to game, don't always measure what they claim, and aren't documented well enough for anyone else to reproduce. So that green checkmark on your model card? Ask what it actually tested. If you're shipping LLM features, this hits your release process directly. Evals drive decisions about deployment and safeguards, so a flawed eval means a flawed go or no-go call. The catch is that a model can score well by refusing everything, which fails users with legitimate requests. Good evals measure both harmful compliance and over-refusal, and they need separate rubrics per topic. The numbers are the problem. Eval results can shift depending on the type of question asked and the order of answer options, so a small wording change can move the score. Multi-turn evals should report an appropriate response rate, the share of conversations where the model behaved well. Designing an eval around the question you actually want answered is the step most teams skip. Worth watching: whether eval suites start publishing full methodology and grading details. Until then, ask your team how each eval was built before you trust it.

Safety Evals Are Easy to Game, Researchers Find, and Most Can't Be Reproduced — anthropic.com

Key Points

1

Eval results can change based on the type of questions asked and the ordering of answer options, so identical models may score differently under small formatting changes.

Keep reading — it's free

Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.

Also get
or

Already a member? Sign in

AI safetymodel evaluationLLM testingover-refusalreproducibility

Comments

Subscribe to join the conversation...

Be the first to comment