Skip to content
Source: OpenAI

📉OpenAI Tracks Perplexity Internally, Proposes Shared Compression Metric for Safety Limits

A shared AI safety yardstick might finally exist

TL;DR

OpenAI has been tracking perplexity internally and argues labs should publish it and agree on shared limits. GPT-4 sat near log-perplexity 0.91, and transformative capabilities showed up around 0.8. If you build on frontier models, the thresholds could eventually shape what you can ship.

OpenAI is pushing the industry toward a shared measuring stick for AI progress: perplexity. The company has been tracking it internally, and it wants frontier labs to agree on common norms and limits built on that number. Why should you care? Perplexity is basically how well a model predicts the next token. Lower is better, and it's the closest thing we have to a neutral way to compare training runs across labs. If shared limits ever get written into industry standards, they'll affect which models you can access, how they get evaluated, and what safety reviews your team has to pass before deploying. The numbers are the interesting part. GPT-4 landed around log-perplexity 0.91. Capabilities like cybersecurity and math competence started emerging near 0.8. OpenAI points to a 0.68-0.7 window as the next place to expect transformative jumps. The catch? Perplexity doesn't map directly to any task you care about, and labs don't measure it the same way. Watch whether competitors actually publish comparable numbers.

OpenAI Tracks Perplexity Internally, Proposes Shared Compression Metric for Safety Limits — OpenAI

Key Points

1

GPT-4's log-perplexity was roughly 0.91, according to the measurements OpenAI discusses, giving a concrete baseline for comparing future frontier models against one another.

Keep reading — it's free

Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.

Also get
or

Already a member? Sign in

perplexityAI safetyOpenAIGPT-4model evaluation

Comments

Subscribe to join the conversation...

Be the first to comment