Robots Exclusion Protocol (robots.txt)

The core commands are Disallow and Allow, which define access paths relative to the site's root.

• Disallow: /private/: Prevents compliant crawlers from accessing any URL starting with /private/.
• Allow: /public/: Explicitly permits access to a subdirectory, often used to override a broader Disallow rule. Directives are processed sequentially, with the most specific matching rule taking precedence. This pattern-matching system is prefix-based, not regex, meaning Disallow: /data blocks both /data.html and /database/records.

The Sitemap directive points crawlers to an XML file listing canonical URLs for efficient discovery. While not an access control rule, it is a critical component of the protocol for AI governance. By specifying a sitemap, a site owner signals the preferred, authorized corpus of content for indexing. This contrasts with the implicit signal of a Disallow rule, providing a positive assertion of which data is intended for consumption. A sitemap can be referenced with a full URL: Sitemap: https://example.com/sitemap.xml.

The Crawl-delay directive specifies the minimum delay in seconds between successive requests from a crawler. This is a politeness mechanism to prevent server overload, not a strict access barrier. For AI data pipelines, a Crawl-delay: 10 can throttle aggressive scrapers that might otherwise degrade site performance for human users. It is important to note that this directive is advisory; many large-scale AI crawlers ignore it, making it an unreliable enforcement tool for data governance without server-side rate limiting.

The Robots Exclusion Protocol is a voluntary standard with no legal enforcement mechanism. Compliance is purely based on the crawler operator's adherence to the convention. Malicious actors and many AI training pipelines deliberately ignore robots.txt directives to scrape disallowed content. This fundamental limitation means the file serves as a signal of intent for data governance rather than a technical access control. Effective enforcement requires complementary measures like IP blocking, rate limiting, and legal terms of service.

The protocol supports limited pattern matching to efficiently manage large sites.

• Wildcard (*): Matches any sequence of characters. Disallow: /*.pdf$ blocks all PDF files.
• End-of-URL ($): Anchors the match to the end of the URL path. This syntax allows a single rule to govern entire classes of resources, such as blocking all dynamically generated pages (Disallow: /*?) or preventing access to specific file types. For AI governance, this enables blanket exclusion of structured data endpoints or API responses from crawler access.