From Search Rankings to Recommendations: Preparing Sites for the AI Agent Era

AI agents are beginning to answer customer questions and recommend products and services in place of search engines. According to Cloudflare, less than half of current HTML page requests are generated directly by humans, with the remainder including bots and agents.

Existing search rankings and page views alone make it difficult to measure discoverability in the agent era. Whether a site is read by agents, interpreted as trustworthy information, and recommended in final answers becomes the new core metric.

Cloudflare has integrated its existing Agent Readiness feature into the dashboard and added Answer Engine Optimization (AEO) tools, allowing site operators to check the following two things:

• Can agents actually use the site?
• How often are the site and its products recommended in AI answers?

The Diagnostics of Agent Readiness inspect the site from the agent's perspective. It checks access permissions and content discoverability via robots.txt and sitemaps, and also examines machine-readable Markdown content, callable interfaces, authentication, and tool-related metadata.

Diagnostic results are aggregated by hostname and displayed in stages from Not Ready to a fully agent-native state. Each item is categorized as pass, fail, or neutral, and provides explanations of importance along with evidence showing actual requests and responses.

Check items are structured according to task difficulty as follows:

• Quick wins: Crawler-readable robots.txt, XML sitemaps, AI crawler rules, and providing Markdown for agents
• Technical foundation: Content Signals specifying content usage, API catalogs, link headers, and agent login guidance
• Advanced configuration: Additional setups for agents to interact with the site more reliably

Just as early SEO favored sites built for search engines, sites that are easily discovered, readable, and trusted will be more likely to receive recommendations from AI agents in the future.