🔒Cloudflare Sets Deadline for AI Crawler Separation: September 15, 2026
Your site's ad revenue vs. AI training data access
TL;DR
Starting September 15, 2026, Cloudflare will block mixed-use crawlers from pages with ads, impacting how AI models can train on web content. Publishers gain control over their data and ad revenue.
Cloudflare has set a new deadline for separating web crawlers used for traditional search purposes from those used for AI training to September 15, 2026. This move aims to protect website owners' intellectual property while allowing discoverability through search engines. Starting this date, Cloudflare's default settings will block 'mixed-use' crawlers from any pages that host ads, affecting new and existing free customers. Website owners now have more control over their content in the AI era with tools like Pay Per Crawl and Pay Per Use, which allow them to charge for access or usage of their data.
Key Points
New Cloudflare policy blocks 'mixed-use' crawlers from pages with ads starting September 15, 2026
Over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages, driving inefficiencies
Cloudflare's Pay Per Crawl and Pay Per Use tools let publishers charge for access or usage of their data
Google Extended bot lets site owners opt out of content being used for training without impacting search inclusion
Initial partners include Ceramic.ai and You.com; publishers earn when content appears in AI search results
Why It Matters
If you're a publisher with ad revenue, this change flips the economics of your website. Blocking mixed-use crawlers protects your ads but limits AI training access. Google's Extended bot offers an opt-out for training data without affecting search inclusion.
Frequently Asked Questions
Why does this matter?
If you're a publisher with ad revenue, this change flips the economics of your website. Blocking mixed-use crawlers protects your ads but limits AI training access. Google's Extended bot offers an opt-out for training data without affecting search inclusion.
What happened?
Starting September 15, 2026, Cloudflare will block mixed-use crawlers from pages with ads, impacting how AI models can train on web content. Publishers gain control over their data and ad revenue.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 2,211 builders reading daily.