Cloudflare AI Search: A Data Search Engine for Agents

Cloudflare AI Search improves the developer experience by allowing developers to build search solutions without directly combining multiple Cloudflare primitives. It helps agents search structured and unstructured data and websites to generate better answers.

Key updates include:

• Indexing data collections, including files and websites, for agents
• Crawling websites without sitemaps using the Discover parsing method
• Providing single /search and /mcp public endpoints that search across entire namespaces
• Connecting custom domains to public endpoints and configuring private search instances with Cloudflare Access
• Adding semantic search to content via the AI Search plugin for EmDash

AI Search also unveiled an initial pricing model that includes embedding and reranking costs. When using certain base models from the Workers AI catalog, embeddings and reranking are free, eliminating the need to separately predict token counts and making it easier to forecast costs as usage increases.

Cloudflare has also applied AI Search to the Cloudflare Dev Stack MCP. It indexed 10 surfaces managed by Cloudflare—Docs, Blog, API Docs, Community, Astro, Vite, Vitest, Hono, Replicate, and OpenNext—as individual AI Search instances, then unified them into a single search. This setup allows coding agents to retrieve the latest documentation and updates along with their sources.

For community sites without sitemaps, you can configure the crawler to follow links and find pages using the --parse-type discover option as follows:

npx wrangler ai-search instance create cloudflare-community \
  --namespace dev-stack \
  --source https://community.cloudflare.com \
  --type web-crawler \
  --parse-type discover

This approach leverages the /crawl feature of Browser Run to automate crawling, collection, embedding, and search.