Summary
Cloudflare expanded AI Crawl Control on April 17 with new tools for the 'agentic Internet,' including Content Format insights and a renamed Directives tab linked to an external agent-readiness score. The update is aimed at helping site owners understand and shape how AI agents consume their content.
What changed
AI Crawl Control added content-format analytics for AI requests and renamed the Robots.txt tab to Directives, with a link to an agent-readiness checker.
Why it matters
This reflects a shift from crawler blocking toward AI traffic optimization. Cloudflare is treating AI agents as a distinct traffic class that sites may want to measure and tune for, not just resist.
Evidence excerpt
Cloudflare says AI Crawl Control now includes Content Format insights showing what content types AI systems request and a Directives tab linked to an Agent Readiness score.