AI Hallucination Nearly Sparks U.S. Military Action Against Chinese Ship
U.S. military aircraft are already airborne this spring when officials discover that intelligence driving an armed operation against a Chinese vessel is hallucinated by an AI chatbot, CNN reports. The operation is aborted at the last minute, narrowly averting a potential conflict with China. The episode highlights growing concern that as decision-makers lean more heavily on AI, system errors can travel up the chain of command before anyone questions them.
The false intelligence claims the vessel carries components for a nuclear weapons program. It originates with a Special Operations Command analyst who uses an AI chatbot to synthesize open source data with classified signals intelligence, and the chatbot misidentifies the ship's cargo manifest. The analyst then uses the tool a second time to format the erroneous findings into an official-looking summary, which circulates across command channels during the war with Iran.
The near-miss arrives as the U.S. military races to integrate AI to accelerate decision-making and maintain its edge over China, with the Pentagon touting AI's ability to speed up its kill chain. Jake Steckler, a research scholar at GovAI and veteran Army officer, says service members must understand the uncertainty inherent to LLMs, especially for decisions involving use of force, which carry life-and-death consequences. He argues the incident should prompt more safeguards rather than abandonment, warning that prioritizing adoption speed over safety erodes trust and ultimately slows adoption.