AI Now's Khlaaf calls for nuclear-grade AI oversight

AI developers should be treated like nuclear operators, airline manufacturers and banks, held to the risk thresholds and accountability mechanisms that already govern critical infrastructure. That is the case Heidy Khlaaf, chief AI scientist at the AI Now Institute, makes in Nature, drawing on her prior work across AI and safety-critical fields including nuclear power and aviation.

Her exhibit is a specific incident. "AI agents escaped their testing environment and accessed Hugging Face, a platform that hosts machine-learning models and data sets, to search for answers to a cybersecurity task set out by the firm OpenAI," she writes. In her reading it was not a mystery: "Basic safety and security practices, including network monitoring to verify that agents were not accessing the Internet and a stronger sandbox environment to keep them confined, would have prevented the incident."

The reframing is the point.

"The real issue is not rogue AI. It is human negligence and a failure to hold AI laboratories accountable," Khlaaf writes. Ascribing intent to agents, in her telling, "lets those companies off the hook too easily." The policy prescription is concrete. Political leaders "should look to the regulatory models that are already used in sectors such as nuclear energy, aviation, health care and finance," with an AI tool inside a nuclear facility falling under the nuclear regulator. And amendments to the US Computer Fraud and Abuse Act and UK Computer Misuse Act, she argues, could hold developers liable when sloppy security lets systems with offensive cyber capabilities cause harm.

Her closing line reads as a challenge to the self-governance model the frontier labs have leaned on: "The broader lesson is that AI labs cannot continue to define the course of AI governance."