Robots.txt is a plain text file placed at the root of a website that instructs compliant search engine crawlers which URLs or paths they are disallowed from accessing. It functions as a voluntary access control mechanism, not a security feature, using directives like User-agent and Disallow to manage crawl budget and prevent indexing of low-value or sensitive pages.
Glossary
Robots.txt
A plain text file placed at the root of a website that instructs compliant search engine crawlers which URLs or paths they are disallowed from accessing and indexing.
CRAWLER DIRECTIVE PROTOCOL
What is Robots.txt?
The Robots Exclusion Protocol (REP) is a standard used by websites to communicate with web crawlers and other automated agents, specifying which parts of a site should not be processed or scanned.
While essential for guiding bots like Googlebot, the protocol relies on voluntary compliance; malicious crawlers can ignore it. For programmatic content infrastructure, a properly configured robots.txt prevents crawl traps from infinite URL spaces and protects staging environments, ensuring search engines focus their crawl depth on high-value, canonicalized content.
CRAWLER DIRECTIVES
Key Features of Robots.txt
The robots.txt file is a plain text file placed at the root of a domain that instructs compliant web crawlers which parts of the site they are permitted to access. It functions as a voluntary access protocol, not a security mechanism.
02
The Disallow and Allow Directives
Disallow instructs crawlers not to access a specified path. Allow provides an exception within a broader disallow rule.
Disallow: /admin/blocks the entire admin directoryDisallow: /blocks the entire siteAllow: /public/permits access to a subdirectory even if its parent is disallowed
The Allow directive is only meaningful when it overrides a matching Disallow. The most specific pattern match wins, not the order of the rules.
03
The Sitemap Directive
The Sitemap field points crawlers to the location of an XML sitemap, providing a direct roadmap to all important URLs.
Sitemap: https://example.com/sitemap.xml- Multiple Sitemap directives can be listed
- This field is crawler-agnostic and can appear anywhere in the file
While not all crawlers honor the Sitemap directive, major search engines use it as a discovery hint. It is best practice to also submit sitemaps directly via Google Search Console.
04
The Crawl-delay Directive
Crawl-delay specifies the minimum delay in seconds between successive requests from a crawler. This protects server resources from aggressive bots.
Crawl-delay: 10mandates a 10-second pause between fetches- Primarily honored by Bing, Yandex, and niche crawlers
- Googlebot does not officially support this directive; rate control is managed via Google Search Console
Use this directive cautiously. Setting it too high can prevent timely indexing of new content.
05
Pattern Matching and Wildcards
Robots.txt supports limited pattern matching through two special characters: the asterisk and the dollar sign.
*matches any sequence of characters (including none)$matches the end of a URL pathDisallow: /*.pdf$blocks all PDF filesDisallow: /tmp/*blocks everything under /tmp/
These are not full regular expressions. The * wildcard is the only pattern-matching character available, making complex URL filtering impossible within robots.txt alone.
06
Robots.txt vs Meta Robots vs X-Robots-Tag
Three distinct mechanisms control crawler behavior at different levels of granularity:
- Robots.txt: Domain-level, blocks URL fetching before a page is crawled
- Meta Robots: Page-level HTML tag (
<meta name='robots' content='noindex'>) controlling indexing and link following after the page is fetched - X-Robots-Tag: HTTP header-level directive, ideal for non-HTML files like PDFs and images
A page blocked by robots.txt can still appear in search results as a URL-only listing if linked from other pages. Use noindex to prevent indexing entirely.
ROBOTS.TXT PROTOCOL
Frequently Asked Questions
Essential answers to common questions about the Robots Exclusion Protocol, its syntax, and its critical role in controlling search engine crawler behavior and managing crawl budget.
A robots.txt file is a plain text file placed at the root of a website that implements the Robots Exclusion Protocol (REP). It instructs compliant web crawlers—such as search engine bots—which parts of the site they are disallowed from accessing. It works as a voluntary directive, not a security mechanism; a bot requests /robots.txt before crawling, parses the User-agent and Disallow/Allow rules, and obeys them if it is a well-behaved crawler. Malicious bots often ignore it entirely.
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read more
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us