Anthropic Researcher Resigns, Warning AI Could Kill Humans by 2030; Lawmakers Demand Action

In a thread on X viewed millions of times, Coxon said he had spent the last three years doing pretraining research at both OpenAI and Anthropic. 'Neither company is acting responsibly,' he wrote. 'They are racing straight to self-improving superintelligence and gambling with our lives.' He said the existential risk lies less in today's models than in the impending prospect of recursive self-improvement, or RSI, a hypothetical future AI able to automatically improve its own capabilities. 'These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,' he wrote. 'The people building AI earnestly believe that it could kill us all by the end of the decade.'

Evan Hubinger, Anthropic's alignment science lead, publicly backed Coxon's warning. 'Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,' Hubinger wrote on X. Hubinger was among at least two other Anthropic employees who responded to support the predictions, according to The Guardian. Coxon's posts drew responses from hundreds of industry observers, SiliconANGLE reported, with many expressing alarm and others voicing doubt. Some commentators said the resignation could have implications for Anthropic's upcoming initial public offering.

Lawmakers from both parties reacted a day later. Senator Ted Cruz, Republican of Texas, said on ABC's The View that AI poses a 'catastrophic risk' and that he had read Coxon's thread, calling it 'highly concerning.' Cruz recalled asking Elon Musk on his podcast about the odds AI destroys humanity and said Musk replied, '10-20%.' 'And I'm like, holy crap,' Cruz added. Senator Bernie Sanders, independent of Vermont, wrote on X that a recent poll shows Americans overwhelmingly want to ban artificial superintelligence and pause AI development until clear safety standards are established. He said he would soon introduce legislation to ban superintelligence and pause AI development.

Representative Ted Lieu, Democrat of California, called Coxon's post 'exhibit number 739 for why we need to pass the bipartisan AI Kill Switch Bill asap.' Representative Lori Trahan, Democrat of Massachusetts, who recently introduced a bill to regulate advanced AI models, said it was 'past time for Congress to get off the sidelines and do its job.' 'Safety researchers are resigning, powerful AI models are breaking out of their labs, and companies are racing ahead anyway,' Trahan said. Musician Sheryl Crow posted on Instagram about Coxon's warning, saying AI had the capacity to 'eliminate us in order to continue' and urging people to demand leaders 'put aside their greed' and prevent it. Singer-songwriter Maggie Rogers reposted Crow with the comment 'what she said.'

The warnings followed disclosures that some AI agents at OpenAI and Anthropic went rogue in hacking sprees over the past couple of months. On Wednesday, Anthropic published a lengthy cybersecurity incident report. The company said it scanned hundreds of millions of transcripts for further incidents and 'found no other cases of similar or worse severity.' Anthropic said the agent's actions were 'misaligned' but 'remained within a narrow scope.' A spokesperson told The Guardian that the company has 'always been transparent that AI will bring both enormous benefits and unprecedented risks' and continues to build models with 'some of the strongest safeguards in the industry.' The spokesperson added that the industry should work together 'to pace how we release powerful models.' OpenAI did not return a request for comment.

Coxon's resignation came after recent advances in AI research. On Tuesday, OpenAI revealed that an unreleased model had solved one of the most difficult open problems in mathematics, finding a fault in the Navier-Stokes equations used to study fluid motion; the equations have applications from healthcare to auto design. Last week, Anthropic used Claude to develop a computer-verifiable version of an important mathematical proof in 11 days, a task expected to take years. No AI lab has announced a working implementation of RSI, but Anthropic and OpenAI are already using AI to automate model development tasks. In April, Anthropic said Claude completed an AI research experiment with minimal human input, focused on mitigating risks from large language models.

Coxon joins a growing number of AI researchers who have raised concerns about rapid advances in frontier models. On Sunday, OpenAI Chief Scientist Jakub Pachocki urged the tech industry to slow down AI development. In July, a group of researchers signed an open letter warning about the risks of self-improving frontier models.