Redirects for AI Training Enforce Canonical Content

Instead of leaving outdated documentation in place, Cloudflare has unveiled Redirects for AI Training, a feature that reads the canonical tag in HTML when a verified AI training crawler accesses a page and redirects it to the latest URL with a 301 Moved Permanently. The judgment is that existing measures alone—warning banners, noindex, and canonical tags—are not enough to stop AI crawlers from continuing to ingest outdated content.

The feature works using cf.verified_bot_category together with the page's <link rel="canonical">. When a bot in the AI Crawler category makes a request, the redirect is performed only if there is a non-self-referencing canonical, while human traffic, search indexing, and other automated traffic are left untouched. Cross-domain canonicals and self-referencing canonicals are excluded, so as not to interfere with loop prevention or domain consolidation use cases.

Cloudflare also shows, using its own documentation, why this feature is necessary. On developers.cloudflare.com, AI crawler category bots made 4.8 million visits over the past 30 days, consuming deprecated content and current content at roughly similar rates. In March 2026, legacy Workers documentation was crawled about 46,000 times, 3,600 times, and 1,700 times by OpenAI, Anthropic, and Meta respectively.

As an example, an AI assistant that had seen the outdated Wrangler CLI documentation still answered with kv:key put, but the correct command is actually wrangler kv key put, and kv:key put was deprecated in Wrangler 3.60.0. Cloudflare stated that during the first 7 days after turning on this feature, 100% of AI training crawler requests to pages with a canonical were redirected to the latest URL, so deprecated content was not served.

Cloudflare also considers that simply solving this with a plain redirect rule doesn't scale well. A rule would need to be added for every newly deprecated path, user-agents would need to be tracked manually, and the content hierarchy already declared by canonical tags would end up being duplicated in rules again.

In addition, Cloudflare has added Response status code analysis to Radar AI Insights. This lets you view the distribution of 2xx, 3xx, 4xx, 5xx responses received by AI crawlers, checkable both for overall traffic and per individual bot. In the example graph, about 70%+ of all AI crawler requests received a 200 response, 10.1% were 301/302 redirects, 3.7% were 404, and 8.3% were 403.