AI Leaders Warn of Bioweapon Risk as Australia Weighs Frontier AI Rules

MIT Technology Review reported that the warnings came after Anthropic researcher Jacob Coxon said he was leaving the company because it and OpenAI were not acting responsibly. Another Anthropic employee, Evan Hubinger, wrote on X that he believed there was more than a 10% chance AI could kill all humans within the next decade. One feared route is bioweapons: a highly lethal virus targeted to genes, a crop-killing fungus, or a tasteless toxin slipped into a water supply.

The concern is not theoretical. In 2022, researchers at Collaborations Pharmaceuticals found that an AI “molecule generator” designed to find drugs generated 40,000 molecules with potential as chemical warfare agents in less than six hours; some were designed to be more toxic than known nerve agents. David Magnus, a professor of medicine and biomedical ethics at Stanford University, called the result “very scary” and said everything since has “blown up.” Dunja Sabra, a biosecurity researcher at the University of Hamburg, said large language models trained on the knowledge of “almost every scientist who ever lived on this planet” can provide instructions and video training for experiments. With gene editing and synthetic biology more accessible, she said a determined person would eventually succeed.

Safeguards exist but are not ironclad. DNA suppliers screen suspicious orders, researchers use red-teaming and blue-teaming, and AI companies have adjusted tools to block misuse. In a report last week, Anthropic acknowledged that people had tried to use its models to explore making the chikungunya virus more transmissible, creating a more dangerous form of bird flu, and building an “atlas of venom toxin peptides.” Magnus said there is a constant back and forth and that AI will probably be needed to restrict AI.

Not all scientists agree on the risk. At a recent media briefing, some biologists at Imperial College London argued that AI tools are not good enough to fully develop bioweapons and that testing new pathogens requires difficult, time-consuming human work. Wendy Barclay, a professor of infectious disease at Imperial, said the greatest pandemic risk currently comes from pathogens already circulating, such as H5N1 bird flu, which has killed millions of birds, spread through US dairy cattle, and was detected last month in captive mink at a Utah farm. Sabra said she prefers to think five to 10 years ahead, and countries should strengthen health-care systems and prepare antidotes to known toxins.

In Australia, The Guardian reported that Andrew Hastie, a federal MP, told a parliamentary inquiry on Friday that when a terminally ill man wrote to him saying he planned to end his life with voluntary assisted dying, Microsoft Copilot suggested replies including “congratulations!”, “great to hear from you” or “that is wonderful news!” Hastie shared the episode to underline the shortcomings of American-run software as he called for AI run by Australians.

The hearing offered detailed explanations of the Albanese government’s desire to bring Anthropic and OpenAI to Australia, against calls to slow them down or shut them out. The defence establishment told the inquiry that cutting the companies out could be catastrophic. Abigail Bradshaw, director general of the Australian Signals Directorate, said her agents need advanced AI every day to fend off “highly capable malicious cyber actors.” If a digital adversary has newer, more capable models, she said, it may identify and exploit vulnerabilities before Australia can. Bradshaw said the agency is “entirely reliant” on US-housed processing and would “dramatically” lose capability if cut off in a conflict.

Chris Crozier, the defence department’s chief AI officer, said the government is contracting Google to build a series of interlinked datacentres across Australia disconnected from the US. “Somebody can’t wake up and have a bad day and decide they’re going to turn it off,” he said. He said Australia wants the compute, application, data and decision-making based in the country. Bringing big tech into Australia would make the job easier, he said, and the Office of AI in the prime minister’s department said bringing firms under Australian law would make it easier for local law enforcement to combat threats on the platforms.

Tight conditions on access could improve Australia’s chances of using and defending itself with the best AI tools, the inquiry heard. Bradshaw said Australia would get “more assured access, earlier access, more detailed technical access, and better integration with technical experts at the frontier.” The government is consulting on standards that would force AI companies to report issues and breaches and potentially reserve some computational power for local research. Large datacentres would have to engage with communities, build away from schools and homes, and underwrite new renewables in most states.

Copyright law remains unsettled, with AI developers refusing to train models in Australia without reform. A proposal to give companies access by default to everything Australians create online drew backlash from creatives, who say the companies could pay for local licensing deals. Jenna Priestly, assistant secretary at the attorney general’s department, confirmed on Friday that looser copyright laws would also affect AI access to foreign-made content distributed in Australia. She said companies want to train on “maybe billions” of pieces of information and claimed it was practically “impossible” to make deals with every rights holder, while Australia’s copyright framework protects material worldwide.

Greens senator David Shoebridge, who opposes the copyright reform calls and wants a pause on AI development, said Labor had refused him a seat on the inquiry committee. Labor ministers met OpenAI’s vice-president and Anthropic’s special envoy Jeff Bleich, who suggested Australia had just months to bring in effective regulation. The attorney general’s department said the shape and timing of copyright reform are undecided. The prime minister’s department said Anthony Albanese wants draft laws on broader AI standards by the end of 2026. Bradshaw said delay risks Australia’s influence and special access, adding that decisions around training are urgent and providers will look elsewhere.

Editor's Summary

AI leaders and researchers are warning that increasingly capable models could lower barriers to bioweapon development, even as some scientists dispute the scale of that risk. Australia is meanwhile weighing whether to host and regulate frontier AI firms, with its cyber agency warning of dependence on US processing and its parliament hearing examples of flawed AI advice and unresolved copyright rules.