Cloudflare Forces AI Crawlers to Pay or Face Blocking Starting September 2026
Cloudflare announces a major policy shift that requires AI companies to disentangle their web crawlers from traditional search bots by September 15, 2026. Under the new default settings, mixed-use crawlers that blend search indexing with AI training and agent services face automatic blocking from any pages that host advertisements. The changes apply to new customers, new sites from existing customers, and all current free-tier users unless they manually adjust their preferences.
The infrastructure giant specifically targets what it describes as an imbalance in how the web's largest search engine leverages its position to access significantly more data than competing AI firms. Cloudflare argues that Google makes it difficult for publishers to remain discoverable in search results without simultaneously surrendering their content for AI applications. Google maintains that its Google Extended bot allows sites to opt out of AI training while preserving search visibility, though its primary Googlebot still crawls for AI-powered features like AI Overviews and AI Mode.
CEO Matthew Prince frames the initiative as a necessary step toward a sustainable digital ecosystem now that bot traffic officially exceeds human traffic on the internet. The company positions its new tools and partnerships as a way to give website owners better visibility and commercial opportunities while rewarding AI companies that operate with transparent intent. Prince expresses hope that the default changes push mixed-use crawlers to clearly separate search functionality from agent use and content training.