AI Hallucination Nearly Triggered U.S. Strike on Chinese Vessel, CNN Reports
A U.S. armed operation against a Chinese vessel was aborted at the last minute this spring after officials found the intelligence behind it had been hallucinated by an AI chatbot, CNN reported.
The intelligence report, which circulated during the war with Iran, said the vessel was carrying components for a nuclear weapons program. The false intelligence originated with a Special Operations Command analyst who queried an AI chatbot to synthesize open-source data with classified signals intelligence. The chatbot misidentified the ship’s cargo manifest. The analyst then used the tool a second time to format the erroneous findings into an official-looking summary, which was circulated across command channels.
The near-miss comes as the U.S. military races to integrate AI to accelerate decision-making and maintain its edge over China. The Pentagon has described AI as delivering a significant advantage in speeding up its kill chain so commanders can respond in the right time. But the same speed that makes AI attractive may also allow hallucinations with insufficient human oversight.
“It’s important for service members to understand the uncertainty inherent to LLMs,” said Jake Steckler, research scholar at GovAI and veteran U.S. Army officer, in a written response to TechCrunch. “But it’s especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions.”
Steckler said the incident should serve as a call to add more safeguards to AI, not a reason to avoid it. “These tools can be useful in the right contexts and with the right safeguards in place,” he said. “But prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems, which ultimately is only going to slow adoption.”
Editor's Summary The aborted operation shows how an AI chatbot’s misreading of a cargo manifest moved through command channels as apparently official intelligence before human review stopped the mission. It also highlights the tension between the Pentagon’s push to speed decision-making with AI and the safeguards needed for targeting, intelligence analysis, and operational planning.