Robots.txt

The User-agent directive specifies which crawler the subsequent rules apply to. It acts as a case-insensitive substring match against the bot's self-reported token.

• Universal Match: Using User-agent: * applies rules to all crawlers that do not have a specific block defined.
• Specific Bot: User-agent: Googlebot targets only Google's primary crawler.
• Precedence Logic: Bots follow the most specific matching block. If a block for Googlebot-News exists, it ignores the generic Googlebot block.
• Multiple Agents: You can list multiple user-agents on separate lines before a directive set to apply rules to a group.

The Disallow directive instructs compliant bots not to crawl the specified relative path. It is the primary mechanism for managing crawl budget and hiding non-public resources.

• Path Prefix: Disallow: /admin blocks /admin, /admin.php, and /admin/login.
• Root Block: Disallow: / blocks the entire site. This is distinct from a missing robots.txt file, which implies permission.
• Empty Value: Disallow: (with no path) allows crawling of everything, equivalent to Allow: /.
• Trailing Wildcard: While not standard, major engines treat Disallow: /*.pdf$ as a pattern match for file extensions.

The Allow directive creates exceptions within a broader Disallow rule. It is primarily used by Googlebot to grant access to a specific resource inside a blocked directory.

• Specificity Wins: Disallow: /blog followed by Allow: /blog/public ensures only the public subfolder is crawlable.
• Order Agnostic: Google evaluates the most specific path rule regardless of its position in the file.
• Parameter Stripping: Bots generally ignore query strings when matching Allow/Disallow paths unless explicitly handled.
• Non-Standard: While supported by major search engines, Allow is not part of the original 1994 standard.

The Sitemap directive is a non-blocking instruction that points bots to the absolute URL of an XML sitemap. It decouples discovery from crawling rules.

• Absolute URL: Sitemap: https://example.com/sitemap.xml is required. Relative paths are invalid.
• Cross-Domain: Sitemaps can point to URLs on other domains if verified via Search Console, enabling cross-domain sitemap hosting.
• Multiple Files: You can list multiple Sitemap directives or point to a Sitemap Index file containing up to 50,000 sitemaps.
• Location: The directive can be placed anywhere in the file and is not bound to a specific User-agent block.

The Crawl-delay directive specifies the number of seconds a bot should wait between successive requests. It protects server resources from aggressive crawlers.

• Integer Value: Crawl-delay: 10 requests a 10-second pause between hits.
• Google Ignored: Googlebot does not officially support this directive. Use Google Search Console to manage crawl rate.
• Bing/Yahoo Support: Microsoft's crawlers respect this directive for managing load on origin servers.
• Dynamic Alternative: For modern rate limiting, a 429 HTTP status code with a Retry-After header is more reliable than this static file directive.

While not part of the original standard, Google and Bing support limited pattern matching to reduce file bloat when blocking dynamic URLs.

• Asterisk (*): Matches any sequence of characters. Disallow: /search?q=* blocks all search query pages.
• Dollar Sign ($): Matches the end of a URL. Disallow: /*.pdf$ blocks PDF files but not /pdf-guide.
• Escaping: There is no official escape character. To block a literal * in a URL path, you must rely on the bot's heuristic parsing.
• Regex Limitation: Full regular expressions are not supported. Complex filtering logic should be handled via X-Robots-Tag HTTP headers.

The approximate number of URLs a search engine bot will crawl on a site within a given timeframe. Crawl budget is determined by two factors: crawl rate limit (how fast bots can fetch without degrading server performance) and crawl demand (URL popularity and freshness).

• A restrictive robots.txt that blocks low-value directories directly preserves crawl budget for critical pages
• Wasted budget on faceted navigation or infinite spaces reduces indexation of revenue-driving content
• Monitoring server logs reveals the crawl health score assigned by Googlebot

An HTTP header directive that provides crawler indexing instructions at the server level, offering granular control over non-HTML assets like PDFs, images, or video files. Unlike robots.txt which only suggests crawling behavior, the X-Robots-Tag can enforce noindex rules that bots must respect.

• Use for file types that cannot embed HTML meta tags
• Supports the same directives as the meta robots tag: noindex, nofollow, noarchive
• Can be applied conditionally via server configuration (e.g., Apache .htaccess or Nginx)

The forensic examination of server access logs to understand exactly how search engine bots interact with a site. Log file analysis reveals whether robots.txt directives are being honored, identifies wasted crawl budget, and surfaces anomalies like bots crawling blocked directories.

• Verify that Googlebot respects Disallow rules in practice
• Detect orphan pages that bots discover through sitemaps but users cannot navigate to
• Identify soft 404s consuming crawl budget despite returning 200 status codes

The process of selecting the preferred URL when multiple URLs serve identical or highly similar content. While robots.txt blocks crawling, canonicalization consolidates ranking signals through <link rel="canonical"> tags or HTTP headers.

• Blocking duplicate URLs via robots.txt prevents crawling but does not consolidate signals
• A blocked URL cannot pass canonical signals if the bot cannot access it
• Use robots.txt to block parameter-based duplicates, then canonicalize the accessible version