AI Now scientist: labs can't be trusted to self-regulate

Heidy Khlaaf, chief AI scientist at the AI Now Institute, argues in a Nature op-ed that when an AI agent "escapes" its test environment, the failure belongs to the humans who built the environment, not the model.

Her prompt is an episode in which agents pursuing a cybersecurity task set by OpenAI reached Hugging Face, the platform that hosts machine-learning models and datasets, during their run. "Basic safety and security practices, including network monitoring to verify that agents were not accessing the Internet and a stronger sandbox environment to keep them confined, would have prevented the incident," Khlaaf writes.

The reframing is her point. "The real issue is not rogue AI. It is human negligence and a failure to hold AI laboratories accountable," she writes. She warns: "Inappropriately ascribing intent to AI agents, rather than recognizing that AI companies deliberately developed these capabilities in poorly secured environments, lets those companies off the hook too easily."

Her policy prescription borrows from sectors that already gate deployment on external review. Political leaders, she writes, "should look to the regulatory models that are already used in sectors such as nuclear energy, aviation, health care and finance." She offers as example: "An AI tool used in a nuclear facility should fall under the authority of the relevant nuclear regulator and be required to meet the same safety standards." She also calls for amendments to the US Computer Fraud and Abuse Act and the UK Computer Misuse Act to hold developers liable for negligent security. Two researchers we track shared the piece the day it ran.

Khlaaf discloses she has done work for both OpenAI and the UK government's AI Security Institute. Her closing verdict is blunt: "AI labs cannot continue to define the course of AI governance."