GPTBot

The complete identification string for the training crawler is:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot

This token must be used in the User-agent field of a robots.txt file to apply directives specifically to this bot. The token includes a link to OpenAI's official documentation page, which serves as a verifiable identifier for the crawler.

OpenAI publishes the IP ranges from which GPTBot makes requests to allow for network-layer verification and firewall configuration.

• Documentation URL: https://openai.com/gptbot
• Verification Method: OpenAI recommends performing reverse DNS lookups on connecting IPs to confirm they resolve back to an openai.com domain.
• Purpose: This allows infrastructure teams to implement defense-in-depth by verifying the bot's identity at both the application layer (User-Agent) and the network layer (IP range).

To prevent GPTBot from using your site's content for training OpenAI's foundation models, add the following to your robots.txt:

codeCopy
User-agent: GPTBot
Disallow: /

• Effect: This blocks GPTBot from crawling any path on the domain.
• Granularity: You can restrict access to specific directories by replacing / with a path like /private-data/.
• Note: This directive only controls the training crawler. It does not affect the separate OAI-SearchBot used for real-time search features in ChatGPT.

To permit GPTBot to crawl only specific sections of a site while blocking everything else:

codeCopy
User-agent: GPTBot
Allow: /public-docs/
Disallow: /

• Order Matters: The Allow directive must precede the Disallow directive for the specific path to be accessible.
• Use Case: This is ideal for organizations that want to contribute public documentation or research to the training corpus while protecting proprietary application logic and user data.

OpenAI uses two distinct crawler tokens for different purposes:

• GPTBot: Crawls data to train and improve OpenAI's foundation models. Controlled by the GPTBot token.
• OAI-SearchBot: Crawls data to provide real-time, cited search results within ChatGPT. Controlled by the OAI-SearchBot token.

Disallowing GPTBot does not prevent your content from appearing in ChatGPT's browsing results. You must manage directives for both tokens independently to achieve your desired governance posture.

GPTBot adheres to standard crawl politeness conventions:

• robots.txt Compliance: The bot checks and respects robots.txt directives before crawling.
• Crawl-Delay: GPTBot respects the non-standard Crawl-Delay directive if specified in robots.txt.
• Rate Limiting: OpenAI implements internal rate limiting to avoid overwhelming servers.
• Caching: The bot caches robots.txt files, so changes may not take effect immediately. OpenAI recommends allowing up to 24 hours for directive updates to propagate.