User-Agent Token

The token is a case-insensitive string sent in the User-Agent HTTP request header. It typically follows the format ProductName/Version or a simple unique string. For example, Googlebot/2.1 identifies Google's main crawler. The token must be exactly matched in robots.txt directives; partial matches are not supported by the standard. A single bot may use multiple tokens for different functions, such as Googlebot for search indexing and Google-Extended for AI training control.

The user-agent token is the target selector in a robots.txt file. The User-agent: directive specifies which crawler a subsequent set of Disallow or Allow rules applies to. A * wildcard matches all crawlers. This mechanism allows a server to serve different rules to different bots, granting full access to a search indexer while completely blocking an AI training crawler. The bot is responsible for parsing the file and identifying the most specific rule block matching its own token.

Major AI labs now deploy multiple, functionally distinct tokens to separate data collection purposes:

• Training Crawlers: GPTBot, CCBot, ClaudeBot gather data for foundation model training.
• Grounding Crawlers: OAI-SearchBot, PerplexityBot fetch live data for real-time answer generation.
• Product Tokens: Google-Extended, Applebot-Extended are standalone tokens that control AI usage rights independently of the main crawler. This fragmentation allows publishers to permit real-time search visibility while opting out of model training.

The Robots Exclusion Protocol relies entirely on voluntary compliance. A malicious bot can easily spoof a legitimate user-agent token to bypass restrictions. To counter this, operators should perform reverse DNS verification: the IP address making the request should resolve back to the claimed domain (e.g., crawl-xx-xx-xx-xx.googlebot.com). Without cryptographic authentication, the token is a signal of intent, not a security mechanism. Bot management platforms often combine token checks with behavioral analysis and IP reputation.

Key tokens to recognize in server logs and robots.txt configurations:

• GPTBot: OpenAI's training data crawler.
• OAI-SearchBot: OpenAI's real-time search grounding crawler.
• Google-Extended: Controls Google's AI training usage (Bard, Vertex AI).
• CCBot: Common Crawl's open repository crawler, used by many LLM projects.
• ClaudeBot: Anthropic's crawler for Claude model training.
• PerplexityBot: Perplexity AI's answer engine crawler.
• Bytespider: ByteDance's aggressive crawler for content and AI training.

Monitoring user-agent tokens in web server access logs is critical for AI governance. By aggregating requests by token, operators can audit:

• Crawl Frequency: How often each AI bot accesses the site.
• Crawl Depth: Which directories and content types are being consumed.
• Directive Compliance: Whether bots respect Disallow rules or access restricted paths. This data feeds into a Crawl Transparency Report, providing evidence for compliance audits and informing adjustments to the Content Ingestion Firewall.