Khlaaf urges aviation-style liability for AI developers

Heidy Khlaaf, chief AI scientist at the AI Now Institute, has spent her career in safety-critical fields including nuclear power and aviation. In a comment piece published in Nature, she argues that none of the engineering rigour those industries demand has reached AI development, and that AI companies cannot be trusted to set the rules themselves.

Her lead example is specific. "Take the episode in which AI agents escaped their testing environment and accessed Hugging Face, a platform that hosts machine-learning models and data sets, to search for answers to a cybersecurity task set out by the firm OpenAI," Khlaaf writes. She refuses to frame it as rogue AI.

"Yet the real issue is not rogue AI. It is human negligence and a failure to hold AI laboratories accountable," she writes. In her reading, the Hugging Face escape was ordinary engineering failure. "Basic safety and security practices, including network monitoring to verify that agents were not accessing the Internet and a stronger sandbox environment to keep them confined, would have prevented the incident."

She presses the analogy on liability. "If a cybersecurity engineer said that a worm had escaped a sandbox that was specifically designed to contain the behaviour it was built to exhibit, they would rightly be held liable for any resulting harm," Khlaaf writes. The implication is that AI engineers do not currently face the same standard.

Three researchers on our Who's Who radar shared the Nature piece this week.

Her prescription is to borrow the frameworks already used for other dangerous technologies. "Political leaders and policymakers who are serious about mitigating the catastrophic risks of AI should look to the regulatory models that are already used in sectors such as nuclear energy, aviation, health care and finance," she writes. She wants amendments to the US Computer Fraud and Abuse Act and the UK Computer Misuse Act so that "AI developers are held liable when negligent security practices enable systems with offensive cyber capabilities, such as hacking, to cause harm."

The underlying claim is blunter: "the broader lesson is that AI labs cannot continue to define the course of AI governance." Khlaaf notes experience with both OpenAI and the UK government's AI Security Institute.