Path matching is the core logic of the Robots Exclusion Protocol (RFC 9309). When a compliant crawler like GPTBot or Googlebot prepares to fetch a URL, it isolates the path segment and evaluates it against the active rule group for its user-agent token. The comparison follows a defined precedence: the most specific matching rule, determined by character length, wins. This allows a broad Disallow: /private/ to be overridden by a more granular Allow: /private/public-reports/.
Glossary
Path Matching
The algorithmic process by which a crawler compares the requested URL path against the patterns defined in the Disallow and Allow directives of a robots.txt file to determine access authorization.
CRAWLER DIRECTIVE EVALUATION
What is Path Matching?
Path matching is the algorithmic process by which a web crawler compares a requested URL's path component against the pattern strings defined in the Disallow and Allow directives of a robots.txt file to determine access authorization.
The matching algorithm treats the * character as a wildcard representing any sequence of characters, while $ designates the end of the URL path. A directive lacking $ is treated as a prefix match. For instance, Disallow: /temp blocks both /temp and /temporary, but Disallow: /temp$ blocks only the exact path /temp. This deterministic, non-regex evaluation ensures consistent crawl budget allocation and prevents unintended access to staging environments or proprietary data repositories.
Algorithmic Access Control
Core Characteristics of Path Matching
Path matching is the deterministic engine of the Robots Exclusion Protocol, where a crawler evaluates its requested URL against defined patterns to enforce access policies.
01
Longest Match Precedence
When multiple Allow and Disallow directives match a URL, the most specific rule wins. Specificity is determined by character length of the defined path, not the order of declaration. For example, Allow: /blog/public overrides Disallow: /blog for the /blog/public directory because the allow pattern is longer and more precise.
02
Prefix-Based Evaluation
Robots.txt matching operates on prefix semantics, not regex. A Disallow: /admin rule blocks:
/admin/admin/login/administration
This is because /admin is a prefix of all three paths. To restrict matching to an exact directory, include the trailing slash: Disallow: /admin/.
03
Wildcard Expansion
The * character acts as a glob wildcard, matching any sequence of zero or more characters. Key applications:
Disallow: /*.pdfblocks all PDF files site-wideDisallow: /private/*/datablocks any path with/private/, then any directory, then/dataDisallow: /*?blocks all URLs containing query parameters
The * is not a true regex quantifier; it cannot specify character classes or repetition counts.
04
URL Normalization
Before matching, crawlers normalize the requested URL by:
- Decoding percent-encoded characters (
%20→ space) - Removing the fragment identifier (
#section) - Resolving relative path segments (
/../) - Converting to lowercase (scheme and host only, not path)
This ensures consistent comparison against robots.txt patterns, preventing trivial bypasses through encoding tricks.
05
Escape Character Handling
RFC 9309 reserves $ as a special end-of-match character. A pattern ending in $ matches only that exact path with no additional characters:
Disallow: /temp$blocks/tempbut not/templateor/temp/file
To match a literal $ or * in a URL path, use percent-encoding: %24 for $ and %2A for *. This prevents ambiguity between literal characters and pattern operators.
06
Case Sensitivity Rules
The path component of a URL is case-sensitive during matching, while the scheme and host are case-insensitive. This means:
Disallow: /Admindoes NOT block/admin- Both
HTTP://example.comandhttp://example.comare treated identically
This distinction matters for sites hosted on case-sensitive file systems (Linux) versus case-insensitive ones (Windows).
PATH MATCHING
Frequently Asked Questions
Clear, technical answers to the most common questions about how crawlers interpret robots.txt directives to determine access to specific URLs.
Path matching is the algorithmic process by which a compliant crawler compares a requested URL's path component against the patterns defined in Disallow and Allow directives within a robots.txt file to determine if it is permitted to fetch that resource. The process is formally defined in RFC 9309, which specifies that matching is performed by comparing the specified path pattern against the beginning of the URL's path. The match is case-sensitive, and the pattern is treated as a prefix unless wildcard characters are used. The most specific matching rule, measured by character length, takes precedence when multiple patterns apply to the same URL. This deterministic evaluation ensures that site owners can precisely control crawler access to their web resources.
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read more
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us