🧪Safety Evals Are Easy to Game, Researchers Find, and Most Can't Be Reproduced
Your model's safety score might be meaningless
TL;DR
Researchers found AI safety evals are easy to game, don't always measure what they claim, and are poorly documented. Results can flip based on question wording or answer order. If you ship models or rely on eval scores, treat them as a starting point, not proof.
Safety evals are the tests teams use to decide whether a model is safe to deploy. Researchers just found that many of them are easy to game, don't always measure what they claim, and aren't documented well enough for anyone else to reproduce. So that green checkmark on your model card? Ask what it actually tested. If you're shipping LLM features, this hits your release process directly. Evals drive decisions about deployment and safeguards, so a flawed eval means a flawed go or no-go call. The catch is that a model can score well by refusing everything, which fails users with legitimate requests. Good evals measure both harmful compliance and over-refusal, and they need separate rubrics per topic. The numbers are the problem. Eval results can shift depending on the type of question asked and the order of answer options, so a small wording change can move the score. Multi-turn evals should report an appropriate response rate, the share of conversations where the model behaved well. Designing an eval around the question you actually want answered is the step most teams skip. Worth watching: whether eval suites start publishing full methodology and grading details. Until then, ask your team how each eval was built before you trust it.

Key Points
Eval results can change based on the type of questions asked and the ordering of answer options, so identical models may score differently under small formatting changes.
Keep reading — it's free
Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.
Already a member? Sign in
Comments
Be the first to comment