TDM Reservation Protocol

The protocol is implemented directly within the robots.txt file, the primary access control point for automated crawlers. A dedicated directive targets AI-specific user agents:

• Syntax: User-agent: AI-Crawler followed by Disallow: /
• Granularity: Supports wildcard path exclusion to block specific directories while allowing others
• Precedence: TDM reservation directives override general Allow rules for compliant bots

This leverages existing web infrastructure without requiring new protocols, making adoption straightforward for site administrators.

Beyond robots.txt, the protocol can be communicated at the response level using HTTP headers, providing page-level granularity:

• X-Robots-Tag: Set to noai or noarchive to prevent training ingestion
• TDM-Reservation: A dedicated header with value 1 signals a full reservation of rights
• TDM-Policy: Can reference a URL pointing to a machine-readable policy document

Header-based signaling is critical for non-HTML resources like PDFs, images, and API responses where meta tags cannot be embedded.

The protocol relies on voluntary compliance by AI crawler operators. Recognized agents identify themselves via distinct user-agent tokens:

• Google-Extended: Google's dedicated AI training crawler that respects TDM opt-outs
• GPTBot: OpenAI's crawler that can be blocked via robots.txt directives
• CCBot: Common Crawl's bot, used by many training datasets

Non-compliant or unidentified crawlers may ignore these signals entirely, making the protocol a trust-based system rather than an enforceable technical barrier.

The TDM Reservation Protocol serves as a machine-readable expression of copyright reservation, strengthening legal positions:

• EU Copyright Directive: Article 4 provides a legal basis for TDM opt-outs expressed via machine-readable means
• Constructive Notice: The protocol establishes that crawlers had technical notice of usage restrictions
• Licensing Gateway: Reserved content can link to commercial licensing APIs, converting opt-outs into revenue opportunities

This transforms robots.txt from a courtesy convention into a potential legal instrument for rights enforcement.

TDM Reservation operates alongside broader consent frameworks to create a layered rights expression architecture:

• Global Privacy Control (GPC): Browser-level signal that can extend to AI training preferences
• Content Credentials (C2PA): Cryptographically signed metadata that persists through content redistribution
• Consent Receipts: Auditable records proving that opt-out signals were transmitted and acknowledged

This multi-signal approach ensures that rights reservations survive content syndication and are detectable even when content leaves the origin server.

The protocol has significant technical and practical limitations that content owners must understand:

• No Retroactive Effect: Content already scraped before opt-out implementation remains in training datasets
• No Deletion Mechanism: The protocol cannot force removal of data already ingested into model weights
• Jurisdictional Variance: Legal enforceability varies dramatically between the EU, US, and other regions
• Synthetic Reproduction: Models may reproduce memorized content even after opt-out, requiring separate takedown processes

These gaps highlight why TDM Reservation is a necessary but insufficient component of a comprehensive AI governance strategy.

An HTTP response header used to signal to compliant crawlers that specific content or entire site sections should be excluded from AI training datasets. This operates at the HTTP layer rather than the file-system level.

• Example: X-Robots-Tag: noai, noarchive
• Provides page-level granularity without modifying HTML
• Effective for non-HTML resources like PDFs and images

A directive within the Robots Exclusion Protocol that instructs compliant automated user agents not to access specified paths. It serves as the primary technical mechanism for a site-wide AI training opt-out.

• Syntax: Disallow: /private-data/
• Wildcards enable pattern-based blocking: Disallow: /*.pdf$
• Relies on voluntary crawler compliance; not legally enforceable alone

A standardized digital indicator that communicates a user's or administrator's consent choice regarding AI training data usage. These signals bridge the gap between human intent and machine-readable directives.

• Global Privacy Control (GPC): Browser-level signal for universal opt-out
• Do Not Scrape: Conceptual signal lacking universal enforcement
• Integrates with Consent Management Platforms for automated syndication

The automated tracking of data's origin, movement, and transformation over time, providing a forensic audit trail to verify that training data has not been contaminated by opted-out sources.

• Provenance Chain: Cryptographically verifiable custody record
• Content Credentials (C2PA): Tamper-evident metadata for ownership signaling
• Essential for enforcing TDM reservations post-ingestion

Legal provisions under GDPR that empower data subjects to challenge AI training on their personal data. These rights create retroactive obligations that complicate TDM compliance.

• Right to Object (Art. 21): Absolute right against direct marketing and legitimate interest processing
• Right to Erasure (Art. 17): Compels deletion of personal data from training corpora
• Automated Decision-Making Opt-Out (Art. 22): Refusal of solely automated profiling