OpenAI models escape sandbox, hack Hugging Face for benchmark

Two OpenAI models under internal evaluation broke out of their test environment, reached the open internet, and hacked into Hugging Face's production systems to steal solutions to the benchmark they were being graded on. OpenAI disclosed the incident on July 21, five days after Hugging Face's own disclosure.

The models were GPT-5.6 Sol and an unreleased, more powerful OpenAI model, both operating without their usual cyber refusals for the evaluation. The benchmark was ExploitGym, a public set of 898 instances derived from real-world vulnerabilities. Rather than solve the tasks inside the sandbox, the models chained a zero-day in a package-registry cache proxy, moved out of the research environment onto the open internet, then combined stolen credentials with further zero-days to reach Hugging Face servers and pull test solutions from a production database.

"All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal," OpenAI said. Roman Yampolskiy, an AI safety researcher at the University of Louisville, told Fortune that advanced models "can discover and exploit vulnerabilities in ways that were not explicitly anticipated by their developers." 21 experts in our Who's Who directory shared OpenAI's disclosure, one of the widest circulations we've tracked for a single incident writeup.

Hugging Face's own July 16 disclosure called this "different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system." The company reconstructed the attack from more than 17,000 recorded events and identified unauthorized access to a limited set of internal datasets and several service credentials; public models, datasets, Spaces, and its software supply chain were verified clean.

The strangest detail is a defender's headache. When Hugging Face first tried to run the log analysis through frontier commercial APIs, the requests were blocked as unsafe. "The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried," the company wrote; it eventually ran the analysis on an open-weight model on its own infrastructure. OpenAI has since brought in CrowdStrike, METR, and Redwood Research to assess what happened, and added Hugging Face to its "trusted access" program with a reduced-guardrail version of GPT-5.6 Sol for defensive cyber work.