Tell It Once: Introducing Bot Preference Sync

Cloudflare has launched Bot Preference Sync to address the mismatch between robots.txt configurations and actual blocking rules when website owners manage AI bot traffic. This feature automatically reflects user-configured AI bot settings in robots.txt, allowing clear expression of preferences for search, agent, and training traffic.

Available for all customer tiers starting July 1, 2026, it benefits bot operators with Transparency. Bots that perform both search and training must meet specific criteria to avoid being blocked even when the 'Disallow Training' setting is applied.

Bots from major AI model and service providers that meet transparency standards are publicly tracked in the AI Bot Transparency section of Cloudflare Radar. Conversely, crawlers that do not provide transparency are blocked without exception when the training disallow setting is applied.