Platformer: AI leaders push for cap on model intelligence

At a conference last weekend in a converted hotel in Berkeley, executives from the top AI labs told a small, closed-door audience that they may want to limit how intelligent a future large language model is allowed to become.

That is the thread Casey Newton pulls on in Platformer, reporting from The Curve, an annual gathering of lab leaders, nonprofit heads, government officials, and a few journalists. "At The Curve, I heard multiple speakers say we may want to limit how intelligent a future system is allowed to become," Newton writes. "Depending on how far labs take any restrictions, the result could be a de-facto ban on systems reaching superhuman intelligence." Sessions were held under the Chatham House Rule, so he cannot name who said what.

What a cap might actually look like is still a sketch. Anthropic CEO Dario Amodei has called for "some kind of 'speed limit'" on recursive self-improvement. Other mooted approaches: barring frontier models from conducting AI research, limiting how much compute a system can touch and how many copies of itself it can run, or refusing to deploy models past a certain capability threshold. That last one, Newton notes, "is arguably already in action."

The problem is who would make any of it stick. "Individual labs can't impose restrictions like this, nor can individual countries, and the current US government actively opposes them," Newton writes. Anthropic's "responsible scaling policy" has been copied in some form by most of its rivals, and both Anthropic and OpenAI have moved toward embedded evaluators. An antitrust waiver that would let labs coordinate on safety is floated as a possible unlock. Even so, Newton reports, the "morally binding" accord AI leaders signed last week with the president does not meet speakers' own definition of enough.

The urgency, in his reading, traces to two shocks: fallout from what he calls the OpenAI-Hugging Face incident, and recent blog posts from OpenAI and Anthropic on their own progress toward recursive self-improvement, "AI systems that can research and train their successors."