Khlaaf in Nature: AI labs need aviation-style oversight

The real problem with artificial intelligence, a new Nature comment piece argues, is not rogue machines. It is the people running the labs that build them.

Heidy Khlaaf, chief AI scientist at the AI Now Institute, writes that AI development has escaped the rigour that governs other high-risk industries. "As a computer scientist who has worked in both artificial intelligence and safety-critical fields — such as nuclear power and aviation — I've long been struck by how little of the rigour that is required for critical infrastructure has been applied to AI development," she writes. Her diagnosis: "the real issue is not rogue AI. It is human negligence and a failure to hold AI laboratories accountable."

The example she reaches for is an OpenAI cybersecurity exercise in which, as she describes it, AI agents "escaped their testing environment and accessed Hugging Face, a platform that hosts machine-learning models and data sets, to search for answers to a cybersecurity task set out by the firm OpenAI." Basic network monitoring, she argues, would have caught it.

From there she makes a regulatory pitch. "From aviation to banking, high-risk industries are subject to independent oversight and meaningful penalties," she writes, and policymakers "serious about mitigating the catastrophic risks of AI should look to the regulatory models that are already used in sectors such as nuclear energy, aviation, health care and finance." The specific lever: amendments to the US Computer Fraud and Abuse Act and the UK Computer Misuse Act so that "AI developers are held liable when negligent security practices enable systems with offensive cyber capabilities, such as hacking, to cause harm."

The piece is a comment, not research, and the Hugging Face episode is the only specific case it names. Three researchers on our Who's Who roster posted the link the day it ran.