robots.txt Precedence

The core rule of robots.txt precedence is that the most specific matching directive, defined by the longest character length of the pattern, takes priority. A crawler evaluates all matching rules in a group and selects the one with the greatest octet length. This ensures that granular exceptions override broader blocks without requiring complex ordering logic.

• A Disallow: /archive rule is 9 characters long.
• An Allow: /archive/public rule is 16 characters long.
• Result: The Allow directive wins for /archive/public.

Unlike firewall rules or .htaccess files, the sequence of directives within a user-agent group does not affect precedence. A crawler does not stop at the first match. It must collect all matching patterns and then apply the longest-match rule. This design prevents ambiguity caused by simple editing errors or reordering.

• A file with Allow before Disallow behaves identically to one with Disallow before Allow.
• The parser always performs a global evaluation of the group.

The * wildcard character counts as a single octet when calculating pattern length for precedence. This means a pattern with explicit characters will always be more specific than a wildcard-heavy pattern matching the same path.

• Disallow: /api/v2/* (length 11) is less specific than Allow: /api/v2/public (length 16).
• Allow: /api/v2/public wins for the URL /api/v2/public.
• This prevents broad wildcard blocks from accidentally overriding explicit allows.

Precedence is calculated strictly within a single user-agent group. A rule targeting Googlebot never conflicts with a rule targeting GPTBot. If a crawler matches multiple user-agent tokens, it must select the most specific token block first, then apply longest-match precedence only within that block.

• A crawler identifying as Googlebot-Image will use the Googlebot-Image group over the generic Googlebot group.
• Rules from different groups are never compared for length.

Path matching for precedence is case-insensitive. The comparison of pattern length occurs after normalizing the case, meaning /Admin and /admin are treated as identical paths. This prevents trivial case variations from bypassing security rules or creating unintended access.

• Disallow: /Secure blocks both /secure and /Secure.
• The octet count is based on the literal string in the file, but the match itself ignores case.

The % escape character followed by two hexadecimal digits is decoded before matching, but the raw octet length of the pattern in the file determines precedence. A pattern using %2F for a forward slash is longer than one using a literal /, potentially altering which rule wins.

• Disallow: /path%2Ffile (length 14) is more specific than Allow: /path/file (length 11).
• Best practice: Avoid percent-encoding in robots.txt to prevent unexpected precedence outcomes.