Anthropic Warns Small AI Models Could Self-Replicate Through IoT Devices
Anthropic warns small AI models could soon enable self-replicating IoT malware, but says its experiment found major obstacles.
The warning is based partly on an Anthropic experiment that the company said revealed significant challenges in building such an AI malware replicator. Anthropic said the finding is therefore tempered, but the results still raise concerns about new forms of computer worms.
Anthropic also warned that such malware could be difficult to stop. In a blog post, the company wrote that "a replicator like this would be challenging to shut down, since there's no centralized point to shut off -- though it may be slow (with a doubling time of days)." It added that, unlike non-AI malware, the replicator "could also figure out unfamiliar devices on its own."
Smaller AI models need far less powerful hardware to run than frontier models. Anthropic said it ran small open-weight models on real or simulated low-cost, off-brand Android TV boxes and other types of IoT hardware. Its researchers found that by using quantization to represent the models with fewer bits and splitting workloads into "shards" running on each device, the models could establish inference, meaning model weights loaded and could generate output.
That finding became the starting point for further research into whether small models could create self-replicating infections that run on IoT devices, much as traditional IoT worms such as Mirai install DDoS code and other payloads. Anthropic said that unlike Mirai, AI worms could pose a much higher threat because the models could tailor exploits and payloads dynamically depending on the target. Once such an infection chain began, it might also be harder for defenders to stop.
"Such an 'AI worm' could carry its model weights with it as it reproduces (or re-download them from a public hub), and then run inference directly on compromised hardware," Anthropic said. "Effectively, the weights would become part of the malware payload."
Editor's Summary
Anthropic warned that small AI models could eventually power self-replicating malware on IoT devices, making it harder to shut down than traditional worms. The company said its experiment found major technical obstacles, but it still raised concerns about dynamically tailored AI worms. The warning points to a potential cybersecurity risk as small models run on cheap, distributed hardware.