Cloudflare and OpenAI Pilot Discovers Hundreds of Millions of New Pages for ChatGPT

Cloudflare and OpenAI have shared early results from a research pilot launched in July to improve AI content discovery. The initiative tested whether Cloudflare’s network signals could help OpenAI’s search crawler find more useful pages for ChatGPT users. The pilot successfully brought previously unknown pages into OpenAI’s search index, expanding the sources available for relevant, timely information.

Expanding the Search Index

The pilot utilized a participating cohort of ~1% of Cloudflare’s total network. This limited scope allowed OpenAI to discover and index hundreds of millions of new web pages. These pages, which may have had few external links, are now accessible to ChatGPT, helping users find specific details like product manuals, local business hours, or official requirements. OpenAI independently evaluates and ranks these pages based on relevance and helpfulness, creating a feedback loop that improves both discovery and answer quality.

Efficiency and Safety Signals

Beyond simple discovery, the pilot focused on reducing wasted crawling effort and enhancing safety. Cloudflare’s signals help identify:

• Non-existent or duplicate pages: Filtering out bots asking for missing files or sites that redirect to homepages.
• Change signals: Determining when a page is worth re-fetching, reducing unnecessary load on website owners.
• Harmful content: Flagging specific compromised pages, such as fake login forms, without penalizing the entire domain.

These mechanisms ensure that OpenAI spends its search resources on information that actually helps people, while Cloudflare customers experience less unnecessary traffic. The research uses metadata for search and discovery, not page content for model training. Domain owners retain control over access via Cloudflare’s AI Crawl Control.