The Department of Defense’s experimental, unpredictable use of artificial intelligence nearly caused a catastrophic diplomatic incident in the middle of the Iran war this spring, when an AI-generated intelligence report suggested that a Chinese vessel
in the region was carrying nuclear weapons components.
According to a report by CNN, the U.S. military scrambled jets and armed servicemembers prepared to board the vessel. But right before the mission went underway, analysts at the DOD finally checked their work: the report, sources told CNN, was “entirely false.” The internal chatbot that a specialist command analyst had used to prepare the report had mistakenly identified the ship’s cargo. The military assets stood down. No one boarded the boat. But the incident came close enough to catastrophe that sources who spoke to CNN were shook, saying that the bogus AI report “almost started a war.”
Secretary of Defense Pete Hegseth has been exuberant about integrating AI chatbots and other tools into the military. “The United States will continue to be AI DOMINANT!” an account belonging to the Pentagon posted on X earlier this week.
But it’s not entirely clear what exactly that means. In January, the Pentagon released its “AI Acceleration Strategy,” a vaguely muddled document that did little to specify what exactly the military would be using AI for. There were mentions of “Swarm Forge” and “Agent Networks,” but little in plain english to explain how large language models and AI tools would fit into actual combat capabilities.
“We will unleash experimentation, eliminate bureaucratic barriers, focus our investments and demonstrate the execution approach needed to ensure we lead in military AI,” Hegseth said in a statement at the time. “We will become an ‘AI-first’ warfighting force across all domains.”
From CNN’s report, it seems there are still plenty of domains that AI shouldn’t be the first tool that soldiers reach for. CNN reports that the specific flub happened when an analyst asked a chatbot — it’s not specified which one, though the military has used several — to analyze intelligence reporting about a ship’s manifest. The report it spit out allegedly was a mishmash of open-source information and secret government intel that brought the bot to a faulty conclusion.
While other officials checked their work and avoided disaster, it’s probably just a matter of time before something like this happens again. In April, a senior Pentagon official told the website DefenseScoop that Hegseth’s department was “cattle-driving, as we say, everything in GenAI.mil.”
The military’s procurement and implementation process — basically, how they adopt new tools — is notoriously slow. But it seems like AI technology has effectively circumvented the slow wheels of traditional bureaucracy and is being thrust into every aspect of the Pentagon’s workflow. CNN’s report is one of the first major documented cases of it making a serious error — one that progressed to the point that real live human soldiers were about to get involved in a major, diplomatically-sensitive operation.
The Pentagon checked their work this time, and no one got hurt. But in an “AI-first military,” anything that happens in the future is just a roll of the dice.
Click here to read the full article.
