Google's Illyes: Robots.txt Covers Crawling, Not AI Training

The short version
Google clarifies that robots.txt governs crawling access, not AI content usage. A video discusses encoding paths and potential usage limitations.
Key takeaways
- Robots.txt restricts crawling, not future content use.
- Non-English paths should be percent-encoded in robots.txt.
- To block AI crawlers, target specific user-agent tokens.
- A draft proposes new usage preferences for robots.txt.
Overall sentiment
NeutralThe article provides factual information without expressing a clear opinion or positive/negative tone.
AI classification of the article’s tone. The response is based solely on the supplied publisher evidence.
Sources cited by the article
- RFC 9309 ↗
- Search Central video ↗
- robots.txt specification page ↗
- IETF Internet-Draft ↗
- Vocabulary Draft ↗
- The Non-Canonical Cabinet ↗
- AI Search ↗
- SEJ ↗
- list of common crawlers ↗
- Read Full Bio ↗
- News ↗
These references were linked or listed by the publisher, then selected for relevance. They are distinct from our original-source citation.