Anthropic ClaudeBot

ClaudeBot identifies itself with a specific user-agent token in HTTP request headers. The full identification string follows the format Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +https://www.anthropic.com/claudebot. This transparent self-identification enables precise targeting in robots.txt directives and bot management systems. The token includes a verification URL pointing to Anthropic's official documentation, allowing webmasters to validate the crawler's authenticity and understand its purpose. Unlike some crawlers that use generic tokens, ClaudeBot's explicit naming convention supports granular access control policies.

ClaudeBot respects standard robots.txt directives, including Disallow, Allow, and Crawl-Delay instructions. The crawler operates from documented IP ranges published by Anthropic, enabling firewall-level verification. Key behavioral characteristics include:

• Respects 429 status codes for rate limiting
• Honors sitemap.xml references for efficient discovery
• Observes noindex and nofollow meta tag directives
• Implements exponential backoff when encountering server errors

Crawl frequency varies based on site authority and content update cadence, but ClaudeBot typically maintains a conservative request rate to minimize server impact.

To manage ClaudeBot's access, implement the following robots.txt directives:

Complete block:

codeCopy
User-agent: ClaudeBot
Disallow: /

Selective access:

codeCopy
User-agent: ClaudeBot
Disallow: /internal/
Disallow: /archive/
Allow: /public/
Crawl-Delay: 10

For non-HTML assets, use the X-Robots-Tag HTTP header with noindex, nofollow values. Anthropic also supports the Anthropic-ClaudeBot/1.0 token variant for backward compatibility with existing bot management rules.

Anthropic maintains a Crawl Transparency Report detailing ClaudeBot's activities and compliance. Infrastructure teams can verify legitimate ClaudeBot traffic through:

• Reverse DNS lookups matching Anthropic's published IP ranges
• User-agent string validation against the official format
• Verification URL checking at the included anthropic.com endpoint

This transparency infrastructure helps distinguish genuine ClaudeBot traffic from user-agent spoofing attempts. Anthropic publishes its IP ranges and updates them regularly, enabling precise firewall rules that prevent impersonation while allowing legitimate access.

The foundational standard (robots.txt) for communicating crawl directives. This file, placed at the root of a domain, is the first mechanism a server uses to declare which paths are disallowed for specific user-agent tokens. ClaudeBot respects this standard, making it the primary control point for managing its access. A misconfigured robots.txt can inadvertently grant full access to all AI crawlers.

OpenAI's distinct user-agent token for its training data crawler. Unlike OAI-SearchBot, which handles real-time ChatGPT browsing, GPTBot is specifically used to collect data for improving foundation models. To block it, you must target the token GPTBot in your robots.txt. This separation of concerns between search and training crawlers is a pattern also seen with Apple's Applebot-Extended.

A standalone product token, not a full user-agent, used to control Google's generative AI training. By adding User-agent: Google-Extended to your robots.txt, you can specifically disallow the use of your site's content for training models like Gemini and Vertex AI, without affecting the classic Googlebot crawler used for search indexing. This provides granular control over data usage.

The holistic set of technical and policy mechanisms used to signal that content should not be used for foundation model training. This goes beyond a single directive and involves:

• robots.txt rules: Targeting specific tokens like ClaudeBot or GPTBot.
• X-Robots-Tag headers: Applying directives to non-HTML files like PDFs.
• Terms of Service: Legal declarations that reinforce technical signals. A robust opt-out strategy layers these controls for defense in depth.

The user-agent token for Common Crawl, a non-profit that maintains a massive, open repository of web crawl data. This corpus is a foundational training source for many large language models, including Meta's Llama and others. Blocking CCBot prevents your content from entering this widely used public dataset, but it does not stop individual AI companies from crawling your site with their proprietary bots like Bytespider or ClaudeBot.

A non-standard but widely respected robots.txt directive that specifies a minimum delay in seconds between successive requests from a crawler. For example, Crawl-Delay: 10 instructs a bot to wait 10 seconds between hits. This is crucial for managing server load from aggressive AI crawlers like Bytespider or ClaudeBot without completely blocking them, preserving access while protecting infrastructure stability.