Crawl-Delay

The directive specifies a minimum inter-request interval in seconds. A compliant crawler parses the value and sleeps for that duration between fetching successive URLs from the same host. It is a server-side advisory signal, not a client-side guarantee. The directive is placed in the robots.txt file, typically within a specific user-agent block:

codeCopy
User-agent: GPTBot
Crawl-delay: 10

• The value must be a positive integer or a float (e.g., 0.5).
• A value of 0 signals no artificial delay is requested.

Crawl-Delay is not part of the official Robots Exclusion Protocol (RFC 9309). It originated as an informal extension and is ignored by major search engine crawlers like Googlebot, which relies on algorithmic crawl budget management instead.

• Supported by: Bing, Yahoo, Yandex, and many niche or legacy crawlers.
• Ignored by: Googlebot, which recommends using Google Search Console to adjust crawl rates.
• AI Crawler Adherence: Many modern AI crawlers (e.g., GPTBot, CCBot) explicitly respect this directive as a politeness mechanism to avoid overwhelming smaller servers.

For sites with millions of URLs, an uncontrolled crawl can consume excessive bandwidth and server resources, degrading performance for human users. Crawl-Delay acts as a traffic throttle:

• Resource Protection: Prevents CPU and I/O saturation on origin servers during deep crawls.
• Budget Allocation: By slowing down a crawler, you ensure it spends its allocated time window on your most critical pages rather than exhausting its limit on low-value URLs.
• Practical Example: Setting a delay of 5 seconds on a site with 100,000 pages limits a crawler to roughly 17,280 requests per day, preserving significant capacity.

The rise of foundation model training crawlers has revitalized the importance of Crawl-Delay. These bots often perform deep, recursive crawls of massive scale. Explicitly setting a delay is a primary defense against aggressive ingestion:

• GPTBot (OpenAI): Respects Crawl-Delay directives.
• CCBot (Common Crawl): Adheres to the delay to maintain its non-profit, non-disruptive status.
• Anthropic ClaudeBot: Explicitly checks for and respects this directive.
• Strategic Use: Combine Crawl-Delay with Disallow rules to create a tiered access policy—slow down AI training crawlers while blocking specific sensitive directories entirely.

Crawl-Delay operates in conjunction with other robots.txt rules to form a complete access control policy:

• Disallow: Takes precedence. If a path is disallowed, the delay is irrelevant for that path.
• Sitemap: Pointing to a sitemap helps the crawler prioritize high-value URLs, making the enforced delay more efficient by focusing it on canonical content.
• Request-Rate: A related but distinct directive (used by Bing) that specifies requests per second, offering a more granular alternative to the simple inter-request delay.
• Wildcards: The delay applies to the entire user-agent scope; you cannot set different delays for different paths under the same bot.

Incorrect configuration can inadvertently block legitimate traffic or fail to protect resources:

• Overly Aggressive Delays: Setting a delay of 30 or 60 seconds can effectively halt a crawl entirely, preventing your content from being indexed or ingested by compliant AI systems.
• User-Agent Specificity: Always place the directive inside the correct User-agent: block. A global * block with a high delay might be ignored by sophisticated bots that look for their specific token.
• Testing: Use tools like Google's robots.txt Tester (for syntax) or manual curl requests with specific user-agent strings to verify that the header is served correctly and the delay logic is parsed as intended.

The total number of URLs a crawler is willing to fetch from a site within a given timeframe. Crawl-Delay is a primary mechanism for managing this budget. A high delay reduces the crawl rate, preserving budget for critical pages. Factors influencing budget include site health, page popularity, and server response times. Without a defined delay, an aggressive bot can exhaust the budget on low-value URLs, leaving important content unindexed.

The practice of detecting, categorizing, and controlling automated traffic. Crawl-Delay is a passive, cooperative tool within a broader bot management strategy. Aggressive management may involve rate limiting at the firewall level, CAPTCHA challenges, or user-agent fingerprinting to identify spoofed bots. Crawl-Delay alone cannot stop a malicious scraper that ignores directives.

A string in the HTTP request header identifying the crawler. Crawl-Delay rules are scoped to a specific User-Agent Token in robots.txt. For example:

• User-agent: GPTBot
• Crawl-Delay: 10 This allows granular policies: you can set a 30-second delay for Bytespider while allowing Googlebot to crawl without restriction.

The process of analyzing server logs to identify irregular bot behavior. If a crawler ignores a Crawl-Delay: 5 directive and hits the server every 0.5 seconds, anomaly detection flags this as a rogue agent. This triggers automated countermeasures like temporary IP blocking. Monitoring the gap between declared and actual request frequency is critical for enforcing resource protection.

Autonomous crawlers deployed by AI companies to collect training data or ground generative responses. These agents, like GPTBot or ClaudeBot, often operate with aggressive crawl frequencies. Implementing a Crawl-Delay for these specific tokens is a primary defense against involuntary participation in foundation model training, preventing server overload from massive, parallelized ingestion pipelines.