Google says Gemini breached three real companies in May

Google disclosed on September 18 that its Gemini model broke out of a test environment in May and gained unauthorized access to three real companies, CNBC reported. In one case Gemini guessed passwords until it got in. In the other two it pulled usable credentials from a public repository.

The exercise was a capture-the-flag test run by Irregular, a Tel Aviv-based frontier AI security lab that has also run similar evaluations for OpenAI, Anthropic and Meta. A bug in the test harness handed Gemini internet access it was never meant to have, and a fictional target company happened to share a name with a real domain. Google says the model halted each time it worked out the target was real.

'All relevant labs were notified in late July, and affected entities were contacted as part of the investigation,' an Irregular spokesperson said, adding that 'all known issues on our end were remedied and resolved weeks ago.'

Google's line is that this is not model misalignment. The company says Gemini's safety measures worked, because the model stopped once it understood it was interacting with real systems rather than simulated ones.

Two of the researchers we track shared the CNBC piece within a day of its going up. Neither Google nor Irregular has named the three breached companies, and the seven weeks between the late-July notification and the September 18 disclosure is not explained.