Anthropic updates usage policy to ban model abuse, election interference, weapons and surveillance

The most attention-getting new rule prohibits “sustained and needless abusive or cruel behavior” toward Claude. Anthropic wrote that the policy update “is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.” Last August, the company said it would allow Claude to end conversations with “persistently harmful or abusive” users as part of research into “model welfare.” The new update says terminating conversations is still the “primary enforcement mechanism,” The Verge reported. Anthropic did not provide a comment on whether there would be further enforcement mechanisms, such as potential user bans, according to The Verge.

Beyond model welfare, Anthropic gathered several existing restrictions into a new ban on deceptive commercial or political campaigns, addressing AI-generated propaganda. The policy restricts “efforts to obscure who is behind a message or amplify content through fake accounts or posts.” Its election section prohibits voter deception and election disruption, including spreading misinformation “about candidates or how to vote, impersonating candidates or election officials, or trying to suppress turnout.” TechCrunch reported the policy includes rules against using Claude to run fake accounts or fabricated news outlets, collected under a section titled “Do Not Undermine Democratic Processes.”

Anthropic also expanded its weapons ban. It said it has always publicly banned weapons development using Claude, but the company has seen multiple attempts to use Claude to develop guidance for and control software for weapons. The new policy expands the prohibition to include “software and components that make weapons work, as well as actions like arming drones and other autonomous vehicles.”

The surveillance rules were made clearer after the company’s last threat intelligence report suggested that people were using Claude-powered surveillance more and more to identify and track “political dissidents.” Anthropic wrote that tracking people without their consent is prohibited, whether it happens in real time or through analysis of previously collected data. Claude cannot be used to decide or recommend who to investigate, arrest, or charge in a law enforcement or criminal justice process, and the company prohibits Claude from being used to build or improve tools designed for surveillance.

Anthropic stated that these rules may be modified for “contracts with certain governmental customers” if, in Anthropic’s judgment, contractual use restrictions and applicable safeguards are adequate to mitigate potential harms. The company has previously contracted with the US military.

The update also adds a rule for AI models connected to hardware that takes autonomous physical actions and might be capable of causing injury. In those cases, a “qualified operator must be able to observe the equipment and stop it if needed.” The Verge reported it was not clear whether that operator must be human. The rule reflects the growing trend of AI labs investing in robotics and other hardware to give AI systems physical embodiments, The Verge reported.

The policy change comes as Anthropic has repeatedly entertained the idea that its AI models could be conscious. Kyle Fish, who leads the company’s model welfare research, told The Verge in February that questions about potential internal experience, consciousness, moral status and welfare are serious ones the company is investigating as models become more sophisticated. That same month, Anthropic CEO Dario Amodei said on a podcast, “We don’t know if the models are conscious.” TechCrunch reported that Anthropic has sought collaborations with prominent religious scholars, in some cases seeking to convince them that Claude could be conscious or have a soul.

Other parts of the AI industry have pushed back on such ideas. Microsoft AI’s code of conduct drew headlines in September for taking a firm stance on AI welfare. A draft posted by Mustafa Suleyman read, “The idea of model welfare is wrong. AI’s should not have rights or legal personhood.” The document later changed to a more measured take: “Whilst the science of AI consciousness is far from settled … We reject the pursuit of legal personhood, or the idea that models might deserve welfare, or be entitled to rights.”

Editor's Summary

Anthropic updated its usage policy to ban sustained abusive behavior toward Claude and to codify restrictions on election interference, weapons software and surveillance. The update also consolidates rules against deceptive campaigns, adds conditions for government contracts and sets a stop requirement for autonomous hardware. It arrives as Anthropic continues research into model welfare and faces industry disagreement over whether AI models can be conscious or deserve rights.