GPTBot

GPTBot obeys the Robots Exclusion Protocol and can be controlled via standard robots.txt syntax.

• Full Disallow:

  codeCopy
  User-agent: GPTBot
  Disallow: /

• Selective Allow:

  codeCopy
  User-agent: GPTBot
  Allow: /public/
  Disallow: /private/

• Crawl-Delay: GPTBot respects the Crawl-Delay directive to manage request frequency and avoid overwhelming origin servers.

This is distinct from the Google-Extended token, which controls ingestion for Google's Gemini and Vertex AI models.

Beyond the user-agent string, GPTBot exhibits identifiable behavioral patterns that can be used for Traffic Pattern Analysis.

• Request Cadence: Typically exhibits a methodical, non-bursty request pattern distinct from human browsing.
• Resource Focus: Primarily fetches HTML documents and text-based content; does not execute JavaScript or render pages.
• TLS Fingerprint: The JA4 hash of GPTBot's TLS Client Hello can be computed and used as a stable identifier, even if the user-agent string is spoofed.

Combining these signals with IP Reputation feeds creates a robust Bot Score for preemptive filtering.

Content scraped by GPTBot is used to train and improve OpenAI's generative models, including GPT-4 and future iterations.

• Opt-Out Mechanism: Blocking GPTBot via robots.txt is the primary technical opt-out for training data ingestion.
• Exclusion Scope: Disallowing GPTBot prevents future use of scraped content in training corpora but does not retroactively remove data from already-trained models.
• Caching: Scraped content may be cached temporarily for deduplication and processing efficiency.

For retroactive removal, organizations must pursue separate Model Unlearning Requests through OpenAI's privacy channels.

GPTBot is distinct from the crawler used for ChatGPT's browsing plugin, which fetches real-time web content to answer user queries.

• GPTBot: Scrapes data for foundation model training; obeys robots.txt for GPTBot.
• ChatGPT-User: Fetches content on behalf of a user's real-time browsing request; uses a different user-agent token.
• Implication: Blocking GPTBot does not prevent ChatGPT's browsing feature from accessing a site when a user explicitly requests it. Separate directives are required for each use case.