Robots Exclusion Protocol

The protocol operates on user-agent tokens, allowing distinct rules for different crawlers. Each robots.txt record group begins with a User-agent: line, targeting a specific bot. The * wildcard applies a default rule to all crawlers not explicitly named. This enables granular control—allowing a search indexer while blocking an AI training bot—but relies entirely on the crawler's voluntary self-identification, making it vulnerable to user-agent spoofing.

The core functional directive is Disallow:, which specifies URL paths a crawler should not access. It is a prefix-matching rule: Disallow: /private/ blocks /private/data.html and /private/images/logo.png. An empty Disallow: grants full access. Critically, this is an advisory signal, not an enforcement mechanism. A malicious or non-compliant bot can ignore the directive entirely, which is why it must be paired with server-side bot management for security.

A non-standard but widely supported extension, Crawl-Delay: specifies a minimum delay in seconds between successive requests from a crawler. For example, Crawl-Delay: 10 forces a 10-second pause. This is essential for crawl budget management, preventing aggressive AI crawlers from overwhelming server resources or incurring excessive bandwidth costs. Not all bots honor this directive, making server-side rate limiting a necessary complementary control.

The Sitemap: directive points crawlers to an XML sitemap listing canonical URLs for efficient discovery. While traditionally used for search engines, it is increasingly relevant for AI-specific sitemaps that guide foundation model crawlers to high-quality, factual content for grounding. This directive is absolute, not path-restricted, and can appear anywhere in the file, providing a direct ingestion pipeline for authorized bots.

The REP is a voluntary, honor-based standard with no legal enforcement mechanism. It functions as a digital 'No Trespassing' sign, not a locked door. Malicious scrapers, unauthorized AI trainers, and state-sponsored bots routinely ignore robots.txt. True access control requires complementary layers: bot management systems, IP reputation filtering, and legal frameworks like robots.txt referenced Terms of Service that establish binding contract law for unauthorized access.

While basic REP uses simple prefix matching, Google and Bing support limited pattern matching extensions. The * wildcard represents any sequence of characters, and $ forces end-of-URL matching. For example, Disallow: /*.pdf$ blocks all PDF files. These extensions provide finer-grained control for content ingestion firewalls, allowing publishers to block specific file types or dynamic URL patterns from AI training datasets without blocking entire directories.