GPTBot

GPTBot is explicitly designed to collect publicly accessible web data for the improvement of OpenAI's foundation models.

• Training corpus: Data may be used to train future iterations of models like GPT-5 or specialized variants.
• Strict exclusion: GPTBot filters out and removes any source that requires a paywall, login, or violates OpenAI's policies on personally identifiable information (PII).
• No real-time indexing: Unlike search engine bots, GPTBot does not power a real-time search index; its purpose is offline model training and capability improvement.

GPTBot fully adheres to the Robots Exclusion Protocol (RFC 9309). To block it, add the following to your robots.txt:

codeCopy
User-agent: GPTBot
Disallow: /

• Granular control: You can allow access to specific directories while blocking others using Allow directives.
• Crawl-delay: GPTBot respects the unofficial Crawl-Delay directive to manage server load.
• Validation: Use the robots.txt Tester in Google Search Console or OpenAI's own validation tools to confirm your rules are parsed correctly.

GPTBot operates from a defined set of IP address ranges published in OpenAI's documentation.

• Rate limiting: The bot is designed to be polite, respecting Crawl-Delay directives and automatically throttling requests to avoid overwhelming origin servers.
• Crawl budget: GPTBot does not consume a traditional search crawl budget, but excessive crawling can still impact server resources.
• Monitoring: Track GPTBot activity via your Content Delivery Network (CDN) logs or Security Information and Event Management (SIEM) system by filtering on the GPTBot user-agent token.

It is critical to distinguish GPTBot from the user-facing ChatGPT browsing feature.

• GPTBot: A background crawler for model training. Controlled exclusively via robots.txt.
• ChatGPT-User: The user-agent for real-time browsing when a ChatGPT user invokes the browsing tool. This fetches content on-demand to answer a specific prompt.
• Blocking strategy: Blocking GPTBot does not prevent ChatGPT-User from accessing your site during a live user session. Use a separate User-agent: ChatGPT-User block if you wish to disable real-time browsing access.

Blocking GPTBot via robots.txt is the primary technical opt-out mechanism for training data ingestion.

• No retroactive removal: Blocking GPTBot prevents future crawling but does not remove data already ingested into trained model weights. For that, see Model Unlearning Requests.
• Licensing alternative: OpenAI offers a Content Licensing API and partnership programs for publishers who wish to grant access in exchange for compensation or attribution.
• Legal context: The robots.txt block serves as a clear technical signal of objection, which is relevant in the context of evolving AI Copyright Compliance frameworks.