Google-Extended

Google-Extended functions as a standalone product token, completely separate from the primary Googlebot crawler. This architectural separation allows site owners to permit standard search indexing while explicitly blocking AI training ingestion. The token is transmitted in the User-Agent HTTP header and can be targeted directly within robots.txt directives without affecting a site's presence in Google Search results.

Control is implemented through the Robots Exclusion Protocol using a dedicated user-agent record. The directive User-agent: Google-Extended followed by Disallow: / blocks all generative AI training ingestion across the entire domain. Granular path-level control is supported, allowing organizations to expose public documentation while protecting proprietary code repositories or premium content archives from model training.

This token specifically governs content used to train Bard (now Gemini) and Vertex AI foundation models. It does not control:

• Standard search indexing (handled by Googlebot)
• Google News or Discover crawling
• Google AdsBot or other marketing crawlers
• Image or video search indexing The token's scope is narrowly defined to generative AI training pipelines, providing precise opt-out granularity.

Authentic Google-Extended crawler traffic can be verified through forward-confirmed reverse DNS lookup. Requests originate from IP addresses that resolve back to hostnames within the *.googlebot.com or *.google.com domains. This verification mechanism prevents spoofed User-Agent strings from bypassing robots.txt directives, ensuring that only legitimate Google infrastructure respects the crawl rules.

Blocking Google-Extended has zero impact on search visibility. Unlike disallowing Googlebot, which removes pages from search results, restricting Google-Extended only prevents content from being used in generative AI training datasets. This decoupling was a direct response to publisher concerns that opting out of AI training would harm organic search performance, a fear that is explicitly unfounded with this token architecture.

Google-Extended respects standard robots.txt precedence rules. The most specific matching directive applies, and the crawler honors Disallow, Allow, and Crawl-delay directives. Unlike some AI crawlers that ignore robots.txt entirely, Google-Extended is a compliant implementation of the Robots Exclusion Protocol. The token was publicly announced in September 2023 as part of Google's commitment to transparent AI data governance.

A text string sent by a web client in the HTTP request header that identifies the browser, operating system, and rendering engine to the server. It serves as the primary mechanism for crawler self-identification. Google-Extended appears as a token within Google's broader user-agent structure, allowing granular control. Server-side parsing of user-agent strings enables:

• Conditional access rules per bot
• Logging and audit trail generation
• Traffic pattern differentiation between indexers and AI scrapers

A specialized web crawler explicitly designed to scrape and ingest large volumes of internet text, images, and structured data for the purpose of pre-training or fine-tuning foundation models. These bots differ from search indexers in key ways:

• They prioritize content completeness over freshness
• They often execute JavaScript to capture dynamic content
• They may ignore caching directives that search bots respect
• They target diverse file types including PDFs, JSON, and media Google-Extended is a canonical example of a training-specific bot token.

A curated list of verified bot signatures, IP ranges, and user-agent tokens that are explicitly permitted to access a web property. This approach inverts the default-deny posture common in bot management. For AI governance:

• Googlebot (search) may be allowed while Google-Extended (training) is blocked
• Legitimate partners like Internet Archive can be selectively permitted
• Unknown or spoofed user-agents are rejected by default Allowlists provide stronger security than robots.txt alone but require ongoing maintenance as new AI crawlers emerge.