robots.txt Parser

The core logic determines whether a requested URL path matches a directive pattern. RFC 9309 specifies:

• Literal matching where the pattern is compared character-by-character against the beginning of the path
• Percent-encoding normalization — reserved characters like %2F must be decoded before comparison
• Wildcard matching using * to match any sequence of zero or more characters
• End-of-path anchoring using $ to require the pattern to match the end of the URL path
• Empty Disallow (Disallow:) means no restriction, granting full access to the group's user-agent

When multiple rules match a single URL, the parser must apply deterministic precedence logic:

• Most specific match wins — the matching pattern with the longest character length takes priority
• Allow overrides Disallow when both patterns have equal length and match the same path
• Group isolation — rules defined under one User-agent group do not apply to other groups
• Global wildcard (User-agent: *) serves as a fallback for any bot without its own specific group
• Duplicate groups for the same user-agent are merged sequentially, with later rules appended

Production parsers must handle malformed files without crashing or producing undefined behavior:

• Unknown directives (e.g., Visit-time:) must be silently ignored rather than causing parse failures
• Invalid lines that lack a colon separator are skipped entirely
• HTTP redirects (3xx status codes) must be followed up to a reasonable limit, typically 5 hops
• Unreachable robots.txt (4xx client errors) implies full crawl permission; 5xx server errors imply full disallow
• Cache invalidation — parsers should respect the Cache-Control header or apply a default TTL, typically 24 hours

Non-compliant parsers frequently introduce errors that cause unintended access or blocking:

• Case-sensitive User-agent matching — RFC 9309 requires case-insensitive comparison of token names
• Regex-based pattern interpretation — the standard uses simple prefix and wildcard matching, not regular expressions
• Ignoring the $ anchor — failing to respect end-of-path anchoring causes overly broad blocking
• Misinterpreting Disallow: / — this blocks the entire site, not just the root page
• Applying global rules to specific groups — rules under User-agent: * must not leak into named bot groups

The algorithmic core of any parser that determines if a URL is disallowed. The process follows strict rules:

• A Disallow rule applies if the URL path starts with the specified pattern.
• Wildcard matching uses * to match any sequence of characters.
• The $ character anchors a pattern to the end of the URL path.
• When multiple rules match, the most specific rule (longest character length) takes precedence.
• Escape sequences like %2F are decoded before comparison.

A parser must correctly associate directive blocks with specific crawlers. The structure follows:

• A User-Agent line starts a new group; all subsequent directives belong to that group until the next User-Agent line.
• A crawler identifies itself by matching its token against the User-Agent field.
• The * wildcard token matches any crawler not explicitly named.
• If a crawler has no specific group, it falls back to the global * group.
• Multiple User-Agent lines can share a single directive block.

An unofficial but widely supported directive that a parser must handle gracefully. Important considerations:

• The value specifies the minimum delay in seconds between successive requests.
• Not part of RFC 9309; Googlebot ignores it entirely.
• Respected by Bingbot, YandexBot, and many custom crawlers.
• A parser should treat non-numeric values as invalid and ignore the directive.
• Multiple Crawl-Delay values for the same bot should result in the parser using the first valid value encountered.

A parser extracts Sitemap directives to facilitate efficient content discovery. Parsing rules include:

• The Sitemap directive is not bound to any User-Agent group; it applies globally.
• Multiple Sitemap lines are valid; a parser should collect all URLs.
• The value must be an absolute URL pointing to an XML Sitemap.
• A parser should validate the URL scheme (must be HTTP or HTTPS).
• This directive is a crawler hint, not an access control rule.