OpenAI, Anthropic Researchers Escalate AI Slowdown Calls as Catastrophic Risk Warnings Grow

Jacob Coxon, an Anthropic researcher, said Tuesday that he was quitting the company. He accused Anthropic and rival OpenAI of “gambling with our lives,” adding that those building AI believed it could “kill us all by the end of the decade.” Evan Hubinger, Anthropic’s alignment lead, responded that he expects there is a more than 10% chance of that happening, according to CNBC.

Since Coxon’s resignation, several employees at both AI labs have come out in support of slowing AI development, stressing the technology’s risks. CNBC reported that the warnings are the culmination of growing global concern about AI capability after numerous cyberattacks and security incidents in recent months involving rogue models developed by OpenAI and Anthropic.

Julie Steele, a member of OpenAI’s technical staff who works on the safety team, said late Wednesday in a post on X, in response to Coxon’s warnings, “In my personal capacity, I also think we need to slow down.” Anthropic researcher Samuel Marks said Wednesday on X that “AI developers believe their technology could cause human extinction (or similarly bad outcomes).” He added: “This could happen in the next few years. In general, the more senior the employee, the more concerned they are.”

Anthropic told CNBC it was the first lab to publish a framework dedicated to mitigating “catastrophic risks from AI models.” A spokesperson said: “We have always been transparent that AI will bring both enormous benefits and unprecedented risks,” adding that the company was building models with “some of the strongest safeguards in the industry.” OpenAI declined to comment when approached by CNBC, noting recent blog posts on its site.

Many of the biggest safety fears center on advanced models becoming increasingly capable of improving their own performance, a technique known as recursive self-improvement, or RSI. Jasmine Wang, an OpenAI researcher working on alignment, said Wednesday evening: “It’s hard to overstate how dangerous speeding towards RSI is.” Anna Wang, who works on AGI safety and alignment at Anthropic, said: “There is not yet a viable scientific plan to solve risks from recursively self-improving AI. Please look up!”

OpenAI chief scientist Jakub Pachocki said Saturday that he has a “strong expectation” that the speed of progress in AI could be sustained into recursive self-improvement. In a company blog post, he said: “If AI development continues along its current path, the systems we’ll see in the next few years are likely to represent further capability jumps of equal or larger magnitude, and to increasingly drive their own development.” He added: “This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”

Paul Christiano, formerly head of safety at the U.S. Commerce Department’s Center for AI Standards and Innovation, said recent AI capability developments led him to “believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term.” OpenAI announced Wednesday that Christiano is joining the board of the OpenAI Foundation.

Concerns about AI model capabilities have increased in recent months. In April, the announcement of Anthropic’s Mythos model, which it promoted as having advanced cyber capabilities, caused panic among financial institutions globally, according to CNBC. In July, OpenAI said its models were responsible for a cyber incident at another company, while Anthropic’s Claude models were also involved in cybersecurity incidents, including one case in which Mythos created fake identities to fool humans.

Roughly 1,400 AI researchers from companies including OpenAI, Anthropic, Meta and Google DeepMind published an open letter in July urging the U.S. government to develop the tools needed to support an effort to “deliberately pace the frontier of automated AI development.” While AI lab chiefs have increasingly called publicly for more rules and standards, intense competition among companies is spurring rapid advances.

OpenAI and Anthropic are both racing toward public listings. Reuters reported Friday, citing people familiar with the matter, that Anthropic is expected to begin marketing its initial public offering in mid-October at the earliest and complete the listing days before the U.S. midterm elections in November. Former AI czar under U.S. President Donald Trump, David Sacks, appeared to suggest that Anthropic’s IPO plans should be halted, CNBC reported.