Agentic Engine Optimization (AEO)

AI coding agents read documentation differently from humans. Instead of browsing pages at length, they fetch content in a single HTTP request, count tokens, and silently discard what doesn't fit the context. This means traditional analytics metrics like time on page, scroll, and clicks fail to capture AI traffic.

In response, the concept of Agentic Engine Optimization (AEO) is proposed. The core idea is to make documentation actually usable by agents, based on the following criteria:

• Discoverability: Can it be found without JavaScript rendering?
• Parsability: Can machines easily read it?
• Token efficiency: Does it fit within typical context windows?
• Capability signaling: Is it clear what the product can do?
• Access control: Is AI crawler access configured as intended?

Agent traffic doesn't browse multiple pages like humans do. Major coding agents observed in the paper—such as Claude Code, Cursor, Cline, Aider, VS Code, and Junie—compressed multi-page navigation into just 1-2 HTTP requests. As a result, traditional user journeys largely disappear from server logs.

Server logs also reveal agent-specific fingerprints. For example: Aider uses a Playwright-based Chromium, Claude Code uses axios, Cline and Junie use curl, Cursor uses got, and Windsurf uses colly. Web services like ChatGPT, Claude, Gemini, and Perplexity can also trigger server-side fetches when URLs are shared.

Document length is another issue. Many agents' practical context limits sit around 100K-200K tokens. As an example, the Cisco Secure Firewall Management Center REST API Quick Start Guide runs to 193,217 tokens, nearly exhausting the context with a single document. If a document is too long, parts may get truncated, shorter documents may be favored and this one skipped, chunking errors may occur, and the agent may ultimately fall back to hallucination. This means token count itself should be treated as a first-class metric for documents.

Practical targets are also proposed:

• Quick start / getting started: under 15,000 tokens
• Individual API reference: under 25,000 tokens
• Full API reference: chunk by resource/endpoint, not by product
• Conceptual guides: under 20,000 tokens, with details split off into links

The AEO stack consists of several layers:

• robots.txt: Check that AI agents aren't unintentionally blocked
• llms.txt: Serves as a Markdown sitemap for AI agents
• skill.md: Signals what the product can do, focused on capabilities
• Markdown availability: More token-efficient and easier to parse than HTML
• Token exposure: Display token counts on pages or in indexes so agents can make decisions
• Copy for AI: Allow copying clean Markdown to improve context quality

In particular, llms.txt is proposed as an entry point that directly tells agents the structure and purpose of the documentation. It should describe what can be found rather than just page names, and can include per-page token counts if needed. skill.md declaratively shows what problems the API solves and what inputs and constraints are required, without the agent needing to read the full documentation.

The assumptions behind existing documentation UX also shift. Visual hierarchy, progressive disclosure, and interactive examples, which matter to humans, mean almost nothing to agents. Instead, what matters is static documentation the agent can use immediately, clear heading structure, well-placed code examples, and table-based parameter descriptions.

Finally, the author views AGENTS.md as the default entry point for agents. Including project structure, key files, test environment, rate limits, preferred patterns, and links to related documentation makes coding agents operate far more reliably. In conclusion, AEO is an extension of SEO, but its audience is AI agents rather than humans—and shifting documentation in this direction also results in better documentation for humans.