AI BriefingBuilding Agent-Friendly Pages Using Content Negotiation
Key point
By using Content Negotiation to serve AI agents optimized Markdown, token efficiency is maximized.
Details
There is a problem where when AI agents crawl web pages, they end up fetching all the unnecessary markup meant for browsers—HTML, CSS, JavaScript, tracking scripts, and so on. This wastes the agent's Context Window and drives up request costs.
By using Content Negotiation, a client can specify its preferred format via the Accept header, and the server can return optimized data accordingly. When an agent requests Accept: text/markdown, the server returns structured Markdown instead of browser-oriented HTML at the same URL.
Vercel implemented this using Next.js. A rewrite rule in next.config.ts detects the header, and a Route Handler converts the CMS's rich text into Markdown on the fly and returns it. Here, the Next.js 16 remote cache is used to keep the HTML and Markdown versions in sync.
The key benefits of this approach are as follows:
- Payload optimization: Compared to the HTML version (about 500KB), the Markdown version (3KB) achieves roughly a 99.37% size reduction, maximizing token efficiency.
- Markdown Sitemap: Unlike a conventional XML sitemap, a Markdown sitemap that includes hierarchy and headings is provided, helping agents more easily understand the site structure.
- Discoverability: For agents that don't use the
Acceptheader, a<link rel="alternate">tag is added to the HTML<head>to point to the Markdown path.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.