Anthropic Exit and a '>10%' Doom Warning Set Off AI Industry's Loudest Risk Debate

The exchange began after AI researcher Jacob Coxon said he had resigned from Anthropic because he worried that leading AI companies are "gambling with our lives." Anthropic's alignment lead then shared Coxon's post and thread on X, writing, "We really do earnestly believe AI could kill all humans!" and adding that he personally put the chance at ">10% within the next decade."

On the latest episode of TechCrunch's Equity podcast, Sean O'Kane said he was hard-pressed to think of something that had blown up so fast. He pointed to the combination of the young researcher's warning, Coxon has also worked at OpenAI, and its immediate amplification on X by Anthropic's alignment lead, which he described as one of the best misplaced exclamation marks ever. O'Kane said the timing turned the post into a powder keg, coming after a Hugging Face hack involving an OpenAI internal model and after capability gains in recent models from Anthropic and from OpenAI's Astra a few weeks earlier.

Anthony Ha disagreed about the punctuation. "If you believe that AI could destroy all humanity, that does deserve an exclamation point," he said. His objection was to the "we" in the post, and to the how far the AI research community can be described as a monolith. He also called the greater-than-10% figure a made-up number, saying the tech industry and others have a habit of throwing out percentages that are not calculated from anything. He said he realized in retrospect that the tweet was probably referencing the concept of P(doom), but that he still considered it silly.

Ha said Coxon's decision set him apart. He noted a recurring theme on the podcast: when figures such as Sam Altman or Dario Amodei use doomer narratives, there is always the question of why they continue the work if they believe the risk is real. Coxon, by contrast, put his professional trajectory where his mouth is, and Ha said he deserved credit for having the courage to do so.

Kirsten Korosec said she placed Coxon in a separate camp from others who talk about the dangers. She asked whether the growing number of blog posts about AI agents breaking through unintentionally, or about humanity being at risk, might be "a weird way of flexing to show how far advanced their company's AI model is," a question she raised as the companies prepare to go public. O'Kane wondered how the warnings would appear in Anthropic's S-1 filing for its IPO, imagining junior lawyers rewriting a section to state that it is officially Anthropic's position that there is a more than 10% chance it could develop something that would eradicate all of humanity and be materially bad for its business.

Ha said he had wondered about the same dynamic, but did not see it as entirely cynical. He said he did not think it was a conscious marketing ploy across the board, and that researchers and executives voicing these concerns hold real worry, while acknowledging that such statements also align with their business interests.

The episode was recorded before Anthropic CEO Dario Amodei published his plan for more cautious AI development, TechCrunch said.