Summary
Cloudflare launched a Disallow AI Training setting that lets any site refuse AI training while staying fully indexed in search, plus an Accountable designation for crawler operators. Apple, Google, and Microsoft are labeled Accountable after meeting or committing to opt-out, URL-level transparency, and no-search-penalty criteria. Cloudflare says mixed-use crawlers are 36.6% of verified crawler traffic on its network.
What changed
Cloudflare shipped a Disallow AI Training control that separates AI-training consent from search indexing, and an Accountable operator designation with defined criteria; Apple, Google, and Microsoft received the designation.
Why it matters
The 'crawl for search vs. crawl for AI training' distinction has been unenforceable because mixed-use bots blur it; letting publishers decline training without losing search visibility, and holding operators to accountability criteria, reshapes the leverage between content owners and AI firms over training data.
Evidence excerpt
To receive the Accountable designation, an operator must provide a way to opt out of AI training, permit an opt-out from AI-generated search summaries, offer URL-level visibility into how content is used, and confirm publicly that declining training will not affect traditional search rankings. Mixed-use crawlers account for 36.6 percent of verified crawler traffic on Cloudflare's network.