AI Briefing
KO

I asked ChatGPT, Claude, Perplexity, and Gemini questions and watched the Nginx logs

·2026.04.21 00:22

Key point

Broke down how major AI assistants access the web and their identifiable User-Agents through Nginx logs.

Details

By adding a custom log format to Nginx and asking ChatGPT, Claude, Perplexity, Gemini, and others queries pointing to the same page, I separately observed whether the model fetches the original content directly and whether the user reads the answer and clicks through.

The key point is that AI traffic should not be treated as a single lump.

  • Provider-side fetch: requests where a bot/agent reads the origin directly
  • Real clickthrough visit: normal browser visits where a user clicks a link in the answer

According to the observations, some services clearly left retrieval-specific User-Agents.

  • ChatGPT: ChatGPT-User/1.0
  • Claude: Claude-User/1.0, and it checked robots.txt first every time
  • Perplexity: Perplexity-User/1.0
  • Meta AI: meta-webindexer/1.1
  • Manus: Manus-User/1.0

On the other hand, Gemini had no Google-family requests coming into the origin during the prompt window, and it appeared to answer from its index. Copilot and Grok did actually fetch the page, but each came in indistinguishable from a regular Chrome/Safari browser, making it hard to identify them as AI visits from the logs alone.

For practical purposes, the following conclusions matter.

  • Logs need to separate AI provider fetches from human click visits.
  • Google-family requests are structurally difficult to identify, so it's hard to tell from HTTP logs alone whether Gemini is doing a live fetch.
  • Search indexing bots (Googlebot, Bingbot, PerplexityBot, Claude-SearchBot, OAI-SearchBot) and training bots (GPTBot, ClaudeBot, CCBot) should be considered separate from answer-generation retrieval traffic.

This piece combines each vendor's documentation with log observations to lay out how to interpret AI agents' web access patterns from actual operational logs.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.