Google's Illyes: Robots.txt Covers Crawling, Not AI Training

Publisher Open Graph preview · Search Engine Journal ↗
THE TLDR

The short version

Google clarifies that robots.txt governs crawling access, not AI content usage. A video discusses encoding paths and potential usage limitations.

Key takeaways

  • Robots.txt restricts crawling, not future content use.
  • Non-English paths should be percent-encoded in robots.txt.
  • To block AI crawlers, target specific user-agent tokens.
  • A draft proposes new usage preferences for robots.txt.

Overall sentiment

Neutral

The article provides factual information without expressing a clear opinion or positive/negative tone.

AI classification of the article’s tone. The response is based solely on the supplied publisher evidence.

Sources cited by the article

These references were linked or listed by the publisher, then selected for relevance. They are distinct from our original-source citation.