OpenAI, Google, Microsoft, and Anthropic Back New Standard for AI Scraping Opt-Outs
OpenAI, Google, Microsoft, and Anthropic announce support for a new standard that lets publishers opt out of AI scraping across multiple companies at once, rather than setting preferences with each AI firm individually. The shared "policy token" signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF).
The signal lives within a website's robots.txt file, which has long told web crawlers whether they can scrape a site for search indexing, archiving, or AI training. Currently, sites must signal their AI training preferences separately to each AI company, creating extra work for publishers.
The announcement comes as publishers struggle with a growing influx of AI crawlers feeding their content into AI models. Some organizations are fighting back in other ways: the Really Simple Licensing (RSL) standard lets publishers charge visiting crawlers, and Cloudflare now blocks AI crawlers by default.