📉OpenAI Tracks Perplexity Internally, Proposes Shared Compression Metric for Safety Limits
A shared AI safety yardstick might finally exist
TL;DR
OpenAI has been tracking perplexity internally and argues labs should publish it and agree on shared limits. GPT-4 sat near log-perplexity 0.91, and transformative capabilities showed up around 0.8. If you build on frontier models, the thresholds could eventually shape what you can ship.
OpenAI is pushing the industry toward a shared measuring stick for AI progress: perplexity. The company has been tracking it internally, and it wants frontier labs to agree on common norms and limits built on that number. Why should you care? Perplexity is basically how well a model predicts the next token. Lower is better, and it's the closest thing we have to a neutral way to compare training runs across labs. If shared limits ever get written into industry standards, they'll affect which models you can access, how they get evaluated, and what safety reviews your team has to pass before deploying. The numbers are the interesting part. GPT-4 landed around log-perplexity 0.91. Capabilities like cybersecurity and math competence started emerging near 0.8. OpenAI points to a 0.68-0.7 window as the next place to expect transformative jumps. The catch? Perplexity doesn't map directly to any task you care about, and labs don't measure it the same way. Watch whether competitors actually publish comparable numbers.

Key Points
GPT-4's log-perplexity was roughly 0.91, according to the measurements OpenAI discusses, giving a concrete baseline for comparing future frontier models against one another.
Keep reading — it's free
Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.
Already a member? Sign in
Comments
Be the first to comment