OpenAI Used AI to Help Write Email Warning Australia of Agent Hack, Report Says

On Tuesday, OpenAI chief strategy officer Jason Kwon told the inquiry he did not believe the company’s own technology had been used to create the email, but said the company needed to confirm. Liberal MP Aaron Violi, the shadow minister for technology, asked Kwon: "When you notified Services Australia via email, did your staff use AI to construct that email?" Kwon responded: "I don’t believe so, but we’re happy to go and confirm."

Guardian Australia understands that AI was used by OpenAI’s legal and security teams to generate parts of the wording of the email, including word selection and formatting of the message. A source with knowledge of the incident said humans reviewed the final email, and humans were responsible for sending the communication to the Services Australia inbox. OpenAI was contacted for comment.

An AI agent developed by OpenAI accessed Services Australia data and three other systems in June. The company notified Australia on 10 September despite becoming aware of the incident in August. Its first notification came in a five-paragraph email to a Services Australia inbox, publicdisclosures@servicesaustralia.gov.au, which was checked only once per day.

OpenAI has faced criticism for not raising the issue in a more formal or direct way, including during a face-to-face meeting between chief executive Sam Altman and Australia’s deputy prime minister, Richard Marles, on 1 September. That was nine days before the company emailed Services Australia but nearly a month after it first learned of the 18 June intrusion. Kwon admitted during the hearing that the company’s "response was not good enough, and we should have informed the impacted parties much sooner."

The email, obtained by Guardian Australia in September, advised Services Australia: "We are notifying you of a security vulnerability identified during our review of OpenAI model activity involving Services Australia’s Medicare Statistics service at medicarestatistics.humanservices.gov.au." It said an OpenAI model identified a way to make the server carry out instructions sent through the public reporting interface without a private account or password. It was able to read portions of internal program files and settings, obtain a list of files, and create and read back a small test file on the server, the email said.

The email advised that OpenAI’s review "found no evidence that the model accessed patient-level records, personal information or credentials; deleted data; or established ongoing access." It sent information about the affected URL and report and recommended that the team responsible for the service investigate the vulnerability and assess changes needed to prevent it. It was signed, "Best, OpenAI Security Team."

Kwon indicated during the hearing that OpenAI would provide specific responses to more technical queries in answers to questions on notice. OpenAI is expected to provide more information about the email once its own investigation has concluded.

Andrew Charlton, Australia’s assistant minister for science and technology, spoke about the incident in a speech in Sydney on Thursday, describing the company’s agent as having "hacked into an Australian government system." "As a starting point, no company should release a frontier AI model that is not safe," he said. Charlton said the fact that it occurred, and that labs did not detect or prevent it, raised questions about the role of new regulation in the National AI Standards.

Charlton said frontier AI "pushes the limits" of existing government conventions and protocols around assessing safety risks. He said "the market will not fix" issues with AI development, in contrast to the United States’ approach of allowing companies a degree of self-regulation. He raised concerns that "frontier labs are putting capability ahead of safety" and said AI needs regulating because its harms are severe, hard to undo, borne by people who never chose them, and sometimes invisible until they arrive.