Why We Need to Rethink Caching for the AI Era

32% of Cloudflare network traffic already comes from automated traffic. Alongside search engine crawlers, uptime checkers, and ad networks, AI agents and scrapers that read the web for RAG-based responses have now joined this flow. The problem is that, unlike humans, they send requests in high-frequency, parallel, exhaustive-scan patterns that quickly shake up caches.

In particular, AI crawlers sequentially scrape broad, scattered content like documents, images, and knowledge documents rather than a site's popular pages, and training traffic shows this tendency even more strongly. According to analysis by Cloudflare and ETH Zurich, AI crawler traffic combines a high unique URL ratio, content diversity, and inefficient crawling, which significantly increases cache misses. Quite a few requests also trigger 404s or redirects, and since multiple independent instances don't share sessions, the same content can repeatedly appear as if it were a new visit.

This pattern degrades the efficiency of existing approaches like LRU cache, prefetching, and cache speculation. AI agents improve accuracy by repeating RAG loops, but since they fetch new long-tail assets each time, reuse rates are low and cache churn only grows. As a result, cache hit rate drops, and misses turn into origin requests, increasing response latency, egress costs, and origin load.

The real-world impact has already shown up at multiple sites. Wikipedia saw multimedia bandwidth usage increase by 50% due to mass image scraping, and SourceHut experienced service instability and latency from LLM crawlers. Read the Docs saw bandwidth increase significantly as large files were downloaded hundreds of times a day, and Fedora and Diaspora also experienced slower response times for human users and downtime.

Addressing this requires not just simple blocking but AI-aware caching. Latency-sensitive AI traffic like real-time Q&A or summarization can be routed to a cache tier that allows larger capacity and moderate latency, while mass collection for training can be mitigated with deeper caches, origin-side SSD cache, queue-based admission, or rate limiting. Cloudflare already offers tools like AI Index and Markdown for Agents, and together with ETH Zurich is experimenting with AI-aware cache algorithms and new cache layers to explore ways to separate AI traffic.