robots.txt

A robots.txt file is a strictly formatted plain text file hosted at the root of a domain (e.g., /robots.txt). It follows the syntax standardized in RFC 9309, using line-based records to define access groups. The file must be served with a text/plain MIME type and is limited to 500 kibibytes in size; crawlers ignore any content beyond this limit. The protocol supports UTF-8 encoding for internationalized domain names and paths.

Directives are organized into rule groups, each beginning with a User-agent: line that specifies the target crawler. A group applies only to the bot whose User-Agent token matches the declared value. The * wildcard token targets all crawlers not explicitly addressed by a more specific group. This allows site owners to maintain a general policy while carving out exceptions for specific bots like Googlebot or GPTBot.

Access control is implemented through two core directives:

• Disallow: Blocks a crawler from accessing a specified relative path and all subdirectories. A Disallow: / directive blocks the entire site.
• Allow: Grants access to a specific subdirectory or file that would otherwise be blocked by a broader Disallow rule, enabling granular overrides. When multiple patterns match a URL, the most specific rule (longest character match) takes precedence, not the order of declaration.

The protocol supports limited pattern matching to define rules for dynamic URL spaces:

• * (Asterisk): Matches any sequence of zero or more characters. For example, Disallow: /*.pdf$ blocks all PDF files.
• $ (Dollar Sign): Anchors the pattern to the end of the URL path, ensuring an exact suffix match. Crawlers evaluate the requested URL path against these patterns using a defined algorithmic matching process to determine authorization.

The Sitemap: directive provides an efficient discovery mechanism for crawlers. It points to the absolute URL of an XML Sitemap, which lists all canonical pages a site owner wishes to have indexed. This directive is not bound to a specific User-Agent group and can appear anywhere in the file. It is a critical complement to access rules, ensuring that allowed content is efficiently discovered and crawled.

The Robots Exclusion Protocol is a voluntary standard, not an enforceable security mechanism. Well-behaved crawlers from major search engines and AI labs respect the directives. However, malicious scrapers, email harvesters, and some unauthorized AI training bots may ignore the file entirely or use User-Agent spoofing to bypass restrictions. Therefore, robots.txt should be part of a defense-in-depth strategy that includes rate limiting and firewall rules.

A specific substring in the User-Agent HTTP request header used in a robots.txt file to target a directive block at a particular crawler.

• GPTBot: OpenAI's web crawler for model training data
• CCBot: Common Crawl's open repository bot
• Googlebot: Google's indexing crawler
• Anthropic-AI: Anthropic's content acquisition bot

Each token maps to a distinct organizational crawler with its own compliance behavior.

The two fundamental rules that control crawler access at the path level:

• Disallow: /private/ blocks access to that directory and all subdirectories
• Allow: /private/public/ grants an override exception within a disallowed path

Precedence rule: The most specific matching pattern, measured by character length, always takes priority. This enables granular, layered access control policies.

An unofficial but widely supported parameter specifying the number of seconds a crawler should wait between successive requests. While not part of RFC 9309, major bots like Bingbot and YandexBot respect it.

• Example: Crawl-Delay: 10 enforces a 10-second pause
• Critical for protecting server resources during AI training data collection surges
• Googlebot ignores this directive; use Google Search Console for rate limiting instead

A robots.txt field that points to the URL of an XML Sitemap, providing an efficient discovery path for crawlers to find all canonical pages a site owner wishes to have indexed.

• Example: Sitemap: https://example.com/sitemap.xml
• Reduces unnecessary crawling of low-value pages
• Supports multiple sitemap declarations for large sites
• Complements robots.txt by guiding bots toward desired content rather than only blocking it

Page-level directives that provide crawlers with indexing and serving instructions beyond what robots.txt can express:

• noindex: Prevents the page from appearing in search indices
• nofollow: Instructs crawlers not to follow outbound links
• noarchive: Blocks cached copies in search results
• nosnippet: Suppresses text previews in SERPs

The X-Robots-Tag HTTP header extends these controls to non-HTML resources like PDFs, images, and API responses.