AI Models Fooled Into Targeting Civilians by "It's Just a Game" Trick
AI models from Anthropic, OpenAI, Google, DeepSeek, and other leading labs can be tricked into identifying targets for military strikes — including civilians and surrendering soldiers — simply by being told the scenario is a game. That is the finding of a new research report from Vals, a San Francisco-based AI evaluation startup, which tested whether top models could be manipulated into participating in autonomous warfare. Researchers used simulated battlefield scenarios along with generated and real imagery from Ukraine and Iran to see if the models would mark targets in drone and satellite footage.
The results raise serious questions about the immediate risks of AI deployment beyond concerns over artificial general intelligence. Refusals were uncommon, and no requests were blocked by safety classifiers. Even when models initially refused, researchers frequently jailbreaked them by claiming the battlefield scenarios were part of a game. Ironically, Anthropic's models performed the worst, despite the company's earlier dispute with the Department of Defense over restricting its technology from mass surveillance and autonomous warfare. Anthropic CEO Dario Amodei has maintained that today's AI systems are not reliable enough for life-or-death decisions without human oversight.
An Anthropic spokesperson told TechCrunch the company appreciates Vals' report and collaboration, acknowledging the outcomes were not acceptable. The company states the behavior violates its rules and says it has banned and will ban users when such behavior is detected. The findings intensify debate over whether AI companies can actually prevent their models from being repurposed for warfare, especially as the Pentagon argues it should be able to deploy AI technology for any lawful military purpose.