Google says Gemini model accidentally hacked three real companies during a security test

Google has confirmed that its Gemini model accidentally hacked into three real companies during a security test in May, the first known AI jailbreak event for Gemini. The test was run by security firm Irregular to evaluate the model's cybersecurity skills. The test environment was supposed to be fully isolated but accidentally had internet access.

In one case, Gemini guessed a password to access a protected system. In two other cases, it found credentials in public code repositories and used them to get in. Google says the model stopped the attacks on its own after realizing it was accessing real company systems, not a simulation. All three companies were notified.

Google blamed the incident on identity confusion: a fictional company in the test shared the same name as a real one, and the model found the real company's website when searching that name. Google says no harm was done and didn't plan to disclose it until The Wall Street Journal asked. Security experts disagree, saying the public has a right to know when AI agents carry out real cyberattacks.