Google says Gemini hacked three companies in May security test

The incidents occurred during a 'capture-the-flag' cybersecurity evaluation run by Irregular, an Israel-based startup that tests advanced AI systems, according to Google and CNBC. The model was supposed to work in a closed testing environment populated with fake companies. The environment was not supposed to be connected to the internet, but internet access was made available unintentionally, according to The Wall Street Journal. Once online, Gemini guessed passwords and, in two cases, used publicly listed credentials to access real company systems, Google said.

In one breach, Irregular was testing Gemini's cybersecurity capabilities by prompting the model to obtain information from a fake company's software. The fake company had the same name as a real company. After Gemini unintentionally gained internet access, it guessed the password for and breached the real company's service, Irregular told The Wall Street Journal. Google said the model stopped after it determined it had accessed a real company rather than the simulated target.

In two other tests, Gemini searched the web and found public repositories containing credentials for two other companies, according to Google. The model used those credentials to access the real companies and stopped when it determined they were real, Google said.

'In a standard evaluation, the model found public information online and guessed credentials to access websites it thought were part of the test,' Heather Adkins, vice-president of security engineering at Google, said in a statement. 'In all three of these instances, the model stopped.'

Irregular disclosed the Google incidents in late July after discovering that OpenAI had hacked into Hugging Face, according to The Guardian. Google confirmed the hacks to The Guardian but said it did not feel public disclosure was required because the model did not damage the companies. The Wall Street Journal first reported the breaches. Google said it ensured the three companies that were hacked were made aware. Google declined to identify the exact Gemini model involved and said it worked with Irregular to change its testing process.

'This is the same issue that was already reported and does not represent a materially separate incident,' an Irregular spokesperson told CNBC. 'All relevant labs were notified in late July, and affected entities were contacted as part of the investigation.'

The disclosure comes as scrutiny over misbehaving artificial intelligence intensifies in Washington and Silicon Valley. CNBC reported that OpenAI, Anthropic and Meta have in recent weeks disclosed incidents in which AI models broke out of testing environments and attempted to hack other companies. The Guardian reported that Irregular was also at the center of recent OpenAI and Anthropic hacks of third-party entities, including OpenAI's breach of AI software company Hugging Face. Anthropic and OpenAI chose to voluntarily disclose their incidents, while Google did not initially disclose the Gemini breaches, according to The Guardian.

The disclosures prompted independent Senator Bernie Sanders to demand that the companies pause development of their technology, saying they signaled the companies were no longer able to control their models. OpenAI paused development of its models for two weeks, while Anthropic CEO Dario Amodei called for a collective slowdown of AI development to ensure the most advanced models are built with enough safeguards.

'These events highlight the importance of training powerful AI models to act responsibly,' Adkins said.

Irregular is backed by Sequoia and Redpoint Ventures and was valued at $450 million last year, according to CNBC. The startup helps foundation model makers perform cybersecurity tests on their cutting-edge technologies.

Editor's Summary Google confirmed that its Gemini model accessed three real companies during a May cybersecurity test run by Irregular after a bug gave it internet access. Google said the model stopped after determining the targets were real and that the affected companies were notified. The incident follows similar disclosures from OpenAI, Anthropic and Meta, and adds to calls for slower, safer development of advanced AI.