robots.txt Grouping

Every rule group begins with a User-agent: line, which acts as the group's header and target selector. The token following this field specifies which crawler the subsequent directives apply to.

• Case-insensitive matching: The token is matched case-insensitively against the crawler's User-Agent HTTP header.
• Specificity: A group targeting User-agent: GPTBot applies only to OpenAI's crawler, while User-agent: * acts as a catch-all for any crawler not explicitly named in another group.
• Multiple declarations: A single group can list multiple User-agent lines consecutively to apply the same rules to several bots, e.g., User-agent: GPTBot followed by User-agent: CCBot.

Following the User-agent line, one or more directive lines form the body of the rule group. These directives are processed sequentially by a compliant parser to determine access rights for a given URL path.

• Disallow: The primary restrictive rule, e.g., Disallow: /private/, blocks access to any path starting with /private/.
• Allow: An override mechanism used to carve out exceptions from a broader Disallow rule, e.g., Allow: /private/public-report.pdf.
• Non-standard extensions: Directives like Crawl-delay: 10 can be included to set a politeness policy, though they are not part of the core RFC 9309 standard.

A rule group is terminated by the start of the next User-agent line or by the end of the file. The boundary between groups is critical for determining which policy applies to a crawler.

• First-match wins: When a crawler parses the file, it must find the most specific group that matches its User-Agent token. If GPTBot is explicitly defined, it uses that group and ignores the User-agent: * group entirely.
• Blank lines: Empty lines between groups are ignored by parsers and are used purely for human readability to visually separate distinct bot policies.
• Group isolation: Directives in one group have zero effect on crawlers targeted by a different group, ensuring complete policy isolation.

Within a rule group, the path specified in a Disallow or Allow directive is matched against the requested URL's path component using a defined algorithm from RFC 9309.

• Prefix matching: A rule like Disallow: /api matches /api, /api/v1/, and /apiary, because it functions as a prefix match.
• Wildcard character: The * token matches any sequence of characters. Disallow: /*.pdf$ blocks all PDF files when combined with the end-of-path anchor $.
• Precedence by specificity: If multiple directives match a URL, the most specific rule (the one with the longest matching character sequence) takes precedence. An Allow rule with a longer path overrides a shorter Disallow rule.

The interaction between a specific bot's rule group and the universal User-agent: * group forms the basis of a site's access control hierarchy.

• Explicit targeting: A group for User-agent: Googlebot provides a custom policy that overrides the default catch-all rules.
• Fallback mechanism: If a crawler's User-Agent token does not match any explicitly named group, it must obey the directives in the User-agent: * group.
• Complete denial: To block all compliant crawlers from a section, a single Disallow: /sensitive directive in the * group is sufficient, unless a specific group uses an Allow override for that path.

Although not a directive that controls access, the Sitemap field is often placed at the end of a robots.txt file, outside of any specific rule group, to point crawlers to an index of canonical URLs.

• Global scope: A Sitemap: https://example.com/sitemap.xml line is not bound to any User-agent and serves as a discovery hint for any crawler that supports the standard.
• Multiple sitemaps: Several Sitemap lines can be listed to reference multiple sitemap files or sitemap index files.
• Efficiency: This mechanism allows a crawler to bypass exhaustive site crawling for discovery, directly fetching the list of URLs the site owner wishes to have indexed.

The primary exclusion rule within a group that instructs compliant crawlers to avoid a specified relative path. An empty Disallow value grants full access to the entire site.

• Disallow: /private/ blocks the directory and all subdirectories
• Disallow: / blocks the entire site
• Disallow: (empty) permits unrestricted crawling

Multiple Disallow lines can stack within a single group to create cumulative exclusion zones.

A granular override mechanism that carves out exceptions from broader Disallow rules. The most specific matching pattern, measured by character length precedence, wins the conflict.

• Disallow: /archive/ combined with Allow: /archive/public/ permits access to the public subdirectory only
• Essential for complex content architectures where blanket blocks would inadvertently hide valuable resources
• Supported by all major crawlers per RFC 9309

An unofficial but widely adopted parameter specifying the minimum delay in seconds between successive requests. Prevents server overload from aggressive AI crawlers.

• Crawl-Delay: 10 enforces a 10-second pause
• Respected by GPTBot, CCBot, and Bingbot
• Googlebot ignores this directive; use Google Search Console for rate control
• Critical for protecting origin server health during large-scale AI ingestion events

A discovery optimization field that points crawlers to the canonical XML Sitemap URL. Unlike path rules, this directive is not group-specific and can appear anywhere in the file.

• Sitemap: https://example.com/sitemap.xml
• Provides an efficient alternative to link-graph discovery
• Reduces wasted crawl budget on low-value pages
• Multiple Sitemap directives are permitted for large-scale enterprise sites

The algorithmic core of robots.txt interpretation. Crawlers compare requested URLs against directive patterns using prefix matching with wildcard support.

• * matches any character sequence (including empty)
• $ anchors the pattern to the end of the URL path
• Longest match wins: a 25-character pattern overrides a 10-character one
• Disallow: /*.pdf$ blocks all PDF files site-wide

Understanding precedence prevents accidental data exposure through misconfigured overrides.