Robots.txt Directive

The Disallow directive instructs crawlers to avoid accessing specified URL paths. It supports prefix matching: Disallow: /private/ blocks all URLs starting with /private/, including /private/data and /private/images/logo.png. An empty Disallow: (no path) grants unrestricted access. The directive does not prevent indexing of URLs discovered through external links—it only controls crawling behavior. For complete de-indexing, combine with noindex meta tags or X-Robots-Tag HTTP headers.

The Allow directive creates exceptions within disallowed paths. It is most useful when a broad Disallow rule needs a narrow carve-out. For example:

• Disallow: /archive/
• Allow: /archive/public/

This permits crawling of /archive/public/report.pdf while blocking everything else under /archive/. The directive follows most-specific-match logic: longer matching paths take precedence over shorter ones, regardless of directive order.

The Crawl-delay directive specifies the minimum delay in seconds between successive requests from the same crawler. Syntax: Crawl-delay: 10 enforces a 10-second pause. This protects origin servers from resource exhaustion caused by aggressive AI training bots that may request thousands of pages per minute. Important caveat: Googlebot ignores this directive entirely—use Google Search Console for rate control instead. Bing and Yandex honor it natively.

The original Robots Exclusion Protocol supports only prefix matching, not full regex or glob patterns. Disallow: /*.pdf is not valid and will be interpreted literally as a path starting with an asterisk. Googlebot and a few advanced crawlers unofficially support limited wildcards (* and $ for end-of-URL anchoring), but this is non-standard. For complex access patterns, implement server-side logic via 403 responses or dynamic robots.txt generation based on request headers.

An unofficial but widely supported extension to the Robots Exclusion Protocol that specifies the minimum delay in seconds between successive requests from a crawler. While not part of RFC 9309, major search engines like Bing and Yandex honor this directive. It serves as a critical rate limiting mechanism to prevent server resource exhaustion from aggressive AI training crawlers that may ignore standard throttling signals.

A robots.txt field that points crawlers to XML sitemap locations, providing a machine-readable inventory of canonical URLs. This directive acts as a positive signal, guiding compliant crawlers toward authorized content rather than forcing them to discover pages through recursive scraping. For AI governance, it can be used to explicitly list content approved for ingestion while disallowing everything else.

The robots.txt specification supports two special characters for flexible path matching:

• Asterisk (*): Matches any sequence of zero or more characters. Example: Disallow: /api/*/private blocks all nested private API endpoints
• Dollar sign ($): Matches the end of a URL path. Example: Disallow: /*.pdf$ blocks PDF files specifically These patterns enable precise control over dynamic and parameterized content without enumerating every URL variant.