Anthropic warns small AI models could become self-replicating IoT malware

In a blog post, Anthropic wrote that a replicator would be challenging to shut down because it has no centralized point to switch off, though it might be slow, with a doubling time of days. Unlike non-AI malware, it could also identify unfamiliar devices on its own. The Verge reported that Anthropic warned such capabilities could be 'months away.' The warning echoes long-standing concerns among AI safety researchers about systems that spread beyond human control.

Anthropic's experiments offer partial support. The Verge reported researchers split Google's Gemma 4, a model with 26 billion parameters, across 15 Android smart TV boxes, none powerful enough to run it alone. It worked, but very slowly. The researchers then built a simulated network of computers designed to behave like Android devices and released an AI-powered worm tasked with finding new machines, copying an AI model onto them and getting it running. The propagated model was Meta's 8 billion parameter Llama 3.1-8B, smaller than the model coordinating the attack.

Only the most sophisticated model tested, GLM-5.2 from China's Z.ai, managed to spread across the entire network, according to The Verge, though smaller models successfully completed individual steps. The Anthropic researchers predicted that smaller models could match GLM-5.2's capabilities by the end of the year. They wrote that they expected a future 'awash with agents running on all types of hardware, including some that aren't under human control.' They acknowledged that they had not demonstrated 'a full rogue replicator' and did not know what one might look like, suggesting it could be a 'slow, distributed digital slime mold' or 'a swarm of fruit flies.' They said their findings showed 'signs of life for AI worms' and called for stronger security on the many devices a replicating system could use to spread.

Anthropic did not use its own models in the tests and has repeatedly argued that open alternatives to its AI models carry 'more risk' than closed systems, The Verge reported. The warning drew familiar accusations of public-relations hype and AI doomerism, including from readers of The Verge's earlier coverage. The Verge noted a considerable gap between showing an AI can run slowly across cheap TV boxes and a self-replicating AI worm tearing through household devices. But it also said that while interrogating a company's motives is worthwhile, allegations of deliberate fearmongering are hard to substantiate, particularly as Anthropic has long raised concerns about AI safety. The Verge added that Anthropic is not alone in investigating the threat of self-replicating AI or calling for better security across connected devices, and that other researchers and firms have also examined the issue.

Experts who spoke with The Verge said that while computer worms and fears of self-replicating AI are not new, recent advances suggest there are good reasons to take the warnings seriously. The disagreement is partly about emphasis: Anthropic and some experts describe a plausible near-term risk that warrants hardening devices and preparation, while critics see a large gap between the experiments and the predicted threat.