OpenAI to Honor Common robots.txt Signal for AI Training Preferences

·2026.10.30 18:30

Key point

The move allows website operators to communicate content usage preferences to multiple AI providers at once, reducing the need to maintain separate rules for each crawler.

Details

OpenAI will honor a common signal in robots.txt that lets website operators express preferences for foundational generative AI training.
The shared signal allows publishers to communicate the same preference to multiple participating providers simultaneously, replacing the previous need to maintain separate rules for individual crawlers.
Implementation is planned for the coming months, with existing crawler controls in robots.txt remaining in effect and respected.
The initiative draws on the work of the Internet Engineering Task Force’s (IETF) AI Preferences working group, with OpenAI encouraging other AI providers to honor the signal.

openai ai-training web-scraping ietf robots.txt content-privacy

Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.