Content Exclusion Header

The header is sent as part of the HTTP response from the origin server. The canonical syntax uses a structured field format to declare exclusion intent.

• Header Name: X-Robots-Tag or a dedicated Content-Exclusion header.
• Directive: noai or noarchive to prevent long-term storage.
• Example: X-Robots-Tag: noai, noarchive
• Scope: Applies to the specific resource requested, overriding broader robots.txt rules.

Unlike robots.txt which operates on URL paths, the Content Exclusion Header provides resource-level granularity. This is critical for dynamic content or mixed-licensing scenarios.

• Dynamic Content: Apply noai to API responses containing proprietary data but not to public HTML.
• Mixed Licensing: A single page might contain public text and licensed images; the header can be applied only to the image assets.
• Wildcard Support: Not applicable; the header is strictly tied to the HTTP response it accompanies.

The header's effectiveness depends entirely on voluntary compliance by the crawler operator. Major AI labs have committed to respecting these signals.

• User-Agent Specificity: Servers can conditionally inject the header only for known AI crawler User-Agents (e.g., GPTBot, CCBot).
• Verification: Compliance is verified via Data Provenance Verification tools that audit training datasets for opted-out content.
• Non-Compliance: Unauthorized ingestion despite the header constitutes a violation of the site's terms of service and potentially copyright law.

The noarchive directive is a critical companion to AI exclusion headers. While noai signals a training opt-out, noarchive prevents the crawler from storing a cached copy entirely.

• Defense in Depth: noarchive removes the content from the crawler's cache, eliminating the source data for future training runs.
• Search Engine Origin: Originally designed to prevent search engines from displaying cached page copies.
• AI Application: Prevents the long-term storage of content in Training Data Opt-Out repositories, closing a loophole where data is stored but not immediately trained on.

A Consent Management Platform (CMP) can dynamically trigger the injection of Content Exclusion Headers based on user consent signals.

• Dynamic Injection: If a user opts out of AI training via a CMP banner, the server-side application layer can append the X-Robots-Tag: noai header to all subsequent responses for that session.
• Global Privacy Control (GPC): The header can be tied to the GPC signal, automating the opt-out at the browser level.
• Auditability: This creates a verifiable Consent Receipt chain, linking the user's preference to the technical enforcement mechanism.

The primary site-wide exclusion mechanism within the Robots Exclusion Protocol. Directives target specific user-agent strings to block AI crawlers:

codeCopy
User-agent: GPTBot
Disallow: /

Key considerations:

• Wildcard support: Disallow: /private/* for granular path exclusion
• Crawl-delay: Rate limiting for compliant bots
• Limitation: Only honored by cooperative crawlers; malicious scrapers ignore it entirely

A crawler instruction preventing long-term storage of cached page copies. Critical for AI training opt-out because:

• Blocks inclusion in static training datasets
• Prevents retrieval from cached archives during model fine-tuning
• Applies to both search engine caches and AI training repositories Can be deployed via X-Robots-Tag: noarchive HTTP header or <meta name='robots' content='noarchive'> HTML tag for comprehensive coverage.

A GDPR Article 21 provision granting individuals the absolute right to object to processing of personal data for direct marketing or legitimate interest purposes. Applied to AI training:

• Can be invoked against automated profiling and model training
• Requires organizations to cease processing unless they demonstrate compelling legitimate grounds
• Creates a legal obligation that technical exclusion headers help operationalize Forms the regulatory foundation for many content exclusion mechanisms.