OpenAI's Pachocki urges frontier pause at 0.68-0.7 perplexity
TL;DR
- OpenAI chief scientist Jakub Pachocki publicly places GPT-3.5 at ~0.99 log-perplexity, GPT-4 at ~0.91, and recent models below 0.8.
- A model at log-perplexity around 0.76, he writes, produced groundbreaking results on open problems in mathematics.
- He proposes frontier labs commit not to cross a 0.68-0.7 log-perplexity window until a successor to chain-of-thought monitoring exists.
A model at log-perplexity around 0.76, writes OpenAI chief scientist Jakub Pachocki, "started producing groundbreaking results" on open problems in mathematics. In a post dated Nov. 22, he asks other frontier labs to commit not to cross a window from 0.68 to 0.7 until monitoring techniques catch up.
The post is the first time OpenAI has lined up its own models on the loss curve publicly. GPT-3.5 sat at roughly 0.99, "capable of fluent conversation." GPT-4 was at ~0.91, "proficient at utilizing chain-of-thought reasoning productively." Recent models have passed below 0.8, where Pachocki says the company has seen "the emergence of transformative capabilities in fields like cybersecurity and mathematics." Extrapolating, he expects "further transformative capabilities as we enter the ~0.68-0.7 log-perplexity window."
The ask, stated plainly: "common limits, based on perplexity and other measurements, that developers are committed to not crossing until our ability to understand and monitor AI behavior achieves a certain concrete milestone (such as the development of a successor to CoT monitors)." Chain-of-thought monitoring is currently the main technique labs use to inspect a reasoning model's intentions, and Pachocki expects it to weaken as capability grows.
He does not expect safety research to keep pace.
"I expect AI will also significantly contribute to progress on alignment and monitoring, I think that progress is unlikely to be fast enough to allow the full potential acceleration in pretraining," he writes. That gap is why he puts "as soon as 2027" on a possible "avalanche of fundamental pretraining advancements," paralleling what has already arrived inside mathematics.
The post names no specific math problems the ~0.76 model resolved, and proposes no enforcement mechanism beyond the hope that "frontier model developers should jointly endeavour to establish shared norms for how to evaluate such models and red-team safety cases for them." OpenAI has filed 423 items into our tracker over the past 90 days; this is one of the rarer entries where the lab publicly names a number it wants to stop at.
Originally reported by openai.com
Read the original article →Original headline: OpenAI Opens Up Pretraining Perplexity Scores, Calls for Industry Safety Limits at 0.68–0.70 Threshold