Summary
During Agents Week 2026, Cloudflare launched the ability to manage automated traffic by purpose — Search, Agent, and Training crawlers — rather than as one undifferentiated bot class, and made the controls available to every customer including the Free tier. For each category, owners can block on all pages, block only on ad-serving pages, or allow. Starting September 15, 2026, newly onboarding domains block Training and Agent crawlers by default on pages that display ads, while Search remains allowed.
What changed
Cloudflare introduced three purpose-based AI bot classifications (Search, Agent, Training) with per-category allow/block controls and enhanced analytics for all plans including Free, and set Training and Agent to block by default on ad-serving pages for new domains starting September 15, 2026.
Why it matters
This turns 'block all bots or none' into a policy surface where publishers can keep search discoverability while gating training and agent access, and it establishes purpose-declaration as the price of admission for AI crawlers — a lever that shapes how agent traffic reaches the open web. Extending the controls to the Free tier pushes agent-aware access policy toward a web-wide default.
Evidence excerpt
Cloudflare groups AI traffic into three behaviors you can control independently: Search, Agent, and Training; for all new domains onboarding to Cloudflare, Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed.