Cloudflare Introduces 'Disallow AI Training' Setting to Separate AI Training Blocking from Search Indexing

Cloudflare announced a 'Disallow AI Training' setting that allows website owners to opt out of AI training while maintaining search indexing. Previously, mixed-use crawlers that served both search and AI training created a trade-off where blocking one would also block the other; this update enables separate control of these two functions.

The new setting is applied at the domain level and works by publishing training opt-out preferences in robots.txt. Cloudflare categorizes bot behavior into Search, Training, and Agent. Crawlers from Apple, Google, and Microsoft (Applebot, Googlebot, Bingbot) are classified as Accountable crawlers, which comply with this setting and allow search crawling even when training is refused. Crawlers from Amazon, Anthropic, Meta, and OpenAI are separated for training-only purposes, so blocking training does not affect search visibility.

The existing 'Block' and 'Block on pages with ads' settings now apply to all Training crawlers, including mixed-use crawlers. Therefore, to block only training while maintaining search visibility, the new 'Disallow AI Training' setting must be used. Cloudflare stated that the two values of crawler access and content creator control impact businesses in different ways.