🔒Cloudflare Blocks AI Crawler Requests by Default
AI crawlers face a new blockade
TL;DR
Starting July 1, Cloudflare blocks AI crawler requests by default. This move affects data integrity and highlights the risks of inaccurate agent summaries.
As of July 1, 2025, Cloudflare automatically blocks AI crawlers on all new domains, turning away hundreds of billions of bot requests. The primary concern isn't about training data but an agent that confidently provides incorrect information after reading a source and checking prices. This issue is easy to miss because the system doesn't flag anything as wrong. Twelve major sites are currently blocked by Cloudflare, including Indeed and Glassdoor. Benchmark tests show that while open-source Fortress can bypass five out of eight anti-bot vendors, Tilion Cloud is needed for more robust protection against DataDome and Amazon's click-wall.
Key Points
As of July 1, Cloudflare blocks AI crawler requests by default on every new domain.
Cloudflare has turned away hundreds of billions of bot requests since this policy was implemented.
Twelve major sites are currently blocked: Indeed, Glassdoor, ZipRecruiter, Crunchbase, Product Hunt, Capterra, FreelanceUpwork, Fiverr, StockX, DoorDash, Coinbase, and Udemy.
Benchmark tests show that open-source Fortress bypasses five out of eight anti-bot vendors but fails against DataDome and Amazon's click-wall.
Tilion Cloud is on the waitlist for the DataDome tier; star the repo and leave your email to join.
Why It Matters
If you're using an AI agent that relies heavily on web scraping, this change by Cloudflare could severely impact data integrity. Twelve major job sites are now blocked, making it harder for agents to accurately summarize or fetch information from these sources without proper protection.
Frequently Asked Questions
Why does this matter?
If you're using an AI agent that relies heavily on web scraping, this change by Cloudflare could severely impact data integrity. Twelve major job sites are now blocked, making it harder for agents to accurately summarize or fetch information from these sources without proper protection.
What happened?
Starting July 1, Cloudflare blocks AI crawler requests by default. This move affects data integrity and highlights the risks of inaccurate agent summaries.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 2,066 builders reading daily.