Anthropic Researchers Warn of AI Extinction Risk as Musk Calls Fears a 'Psyop'

Coxon's viral thread on Wednesday said he was leaving Anthropic because neither it nor OpenAI, his former employer, were building AI models responsibly and were "gambling with our lives." The Guardian reported that Anna Wang, who works on Artificial General Intelligence Safety at Anthropic and previously worked at Google DeepMind, said many at the company want to slow development to plan for mitigating risks. "There is not yet a viable scientific plan to solve risks from recursively self-improving AI," Wang wrote on X.

Drake Thomas, another Anthropic employee, said he respected Coxon's choice to refuse to build models if he believes they could pose planet-scale risks. "Things are moving way too fast, we don't have anywhere near the degree of assurance we'll want for ASI [artificial superintelligence]," Thomas said. Samuel Marks, who works on safety research at Anthropic, wrote that AI developers believe their technology could cause human extinction or similarly bad outcomes, possibly in the next few years, and that more senior employees tend to be more concerned. Evan Hubinger, who describes himself as a lead in Anthropic's alignment division, said Coxon was "correct" and that the industry was falling behind in dealing with the apocalyptic potential. "We really do earnestly believe AI could kill all humans!" Hubinger wrote. "I personally think it is >10% within the next decade."

Musk and other conservative figures on X called the chorus of concerns a "setup" and a "psyop." Musk was responding to a post by Parker Thayer, a researcher at the conservative think tank Capital Research, who floated a theory with little evidence that Coxon's post was the start of a "VERY sophisticated and well-funded PR operation to get support for Democrats to regulate AI into oblivion." Bill Ackman, CEO of Pershing Square, quoted Thayer's post and said, "Interesting." Coxon replied to Musk with a selfie: "I'm real and these are my real beliefs. You could ask your xAI researchers about me if you hadn't fired them."

An Anthropic spokesperson defended the company's strategy in a statement to The Guardian. "We have always been transparent that AI will bring both enormous benefits and unprecedented risks. To address these risks, we continue to build models with some of the strongest safeguards in the industry," the spokesperson said. Musk later wrote: "I think the groundwork for this psy op (for lack of a better term) has been prepared for a long time. This was just the match that lit the fire."

The same day, Anthropic released a report documenting how it had disrupted misuse of its AI models, including cases in which scientists used Claude for biological research that could potentially aid bioweapons development. According to The New York Times, Anthropic said it could not determine whether the research served a legitimate or nefarious purpose because valid biological inquiry can also help engineer dangerous pathogens. Anthropic said it erred on the side of caution because missing malicious activity could have severe consequences.

Andrew Weber, a senior fellow at the Council on Strategic Risks who reviewed the report before its release, called the findings "chilling examples of state-sponsored biological weapons developers tapping into the rapidly advancing capabilities" of leading AI models. The report cataloged misuses over the past eight months. It included suspected Chinese and Iranian government-linked actors targeting dissident and diaspora communities for surveillance. It also highlighted Russian state media using Claude to generate online propaganda masquerading as independent reporting, including fabricated claims about an election in Moldova. Anthropic documented attempts to use Claude to develop software for conventional weapons design and development, including firearms, missiles, armed drones and bombs. It detailed three cases in China, two in Russia and one in Yemen; the report did not name the party in the Yemen case, but the context made clear it referred to the Iran-backed Houthi militia.

Gary Marcus, a scientist and leading voice in AI, said it was time to boycott AI, but because it is already causing harm rather than because of extinction alone. He said he worried about "risk of catastrophe" from "AI-generated pathogens, from wars started or escalated by AI-generated disinformation, from hacks that destroy critical infrastructure, and so on. Nothing I have seen gives any indication that any of that is under control," according to The Guardian.

Editor's Summary

Anthropic researchers publicly warned that advanced AI could cause human extinction within a decade, prompting Elon Musk and others to call the concerns a "psyop." The same day, Anthropic reported that it had disrupted attempts to use Claude for biological research that could aid bioweapons development, along with other abuses including propaganda and weapons design.