Text and Data Mining Exception (TDM Opt-Out)

The TDM exception originates from Article 4 of the EU Directive on Copyright in the Digital Single Market (CDSM Directive) . It permits reproductions and extractions for text and data mining, but critically, rightsholders can expressly reserve these rights in an appropriate manner, such as machine-readable means for online content. This creates a legally binding opt-out mechanism distinct from fair use doctrines.

Unlike the US fair use doctrine, which is a flexible, case-by-case affirmative defense, the TDM opt-out is a preemptive, rights-based reservation. It does not require a balancing of four factors. If a rightsholder has properly reserved their rights, the exception simply does not apply, making it a more absolute and technically enforceable mechanism for content owners in jurisdictions implementing the CDSM Directive.

Effective enforcement relies on AI developers honoring the opt-out signal. Major foundation model providers like OpenAI (GPTBot) and Google (Google-Extended) have published user-agent tokens that respect robots.txt disallow rules. However, compliance is not universal, and the opt-out creates a legal basis for copyright infringement claims against entities that ignore the reservation and proceed with unauthorized TDM.

The opt-out applies specifically to reproductions and extractions for TDM purposes, not to the act of indexing for search engines or human browsing. The reservation can be applied to:

• Entire domains via robots.txt
• Specific subdirectories or file types
• Individual HTML pages via TDMRep meta tags This granularity allows rightsholders to permit indexing for search discovery while blocking ingestion for generative AI training.

The TDM opt-out is a technical and legal signal, distinct from but complementary to website Terms of Service (ToS). While a ToS can contractually prohibit scraping, the machine-readable opt-out provides a direct, automated instruction to crawlers. A robust defense combines both: explicit contractual prohibitions in ToS and machine-readable reservations to trigger the legal exception's inapplicability.

A Consent Management Platform extends TDM opt-out signaling beyond web crawlers to user-level data flows. CMPs capture and transmit granular consent preferences, including Article 4(3) of the DSM Directive reservations, via standardized APIs. This ensures that even if content is lawfully accessed by a human, the machine-readable metadata attached to the session prevents subsequent automated analysis without explicit authorization.

Training Data Provenance is the documented chain of custody that validates whether a TDM opt-out was respected during corpus assembly. Without robust provenance records, it is impossible to audit whether a foundation model ingested content from a rightsholder who had explicitly reserved their rights. This concept underpins Algorithmic Disgorgement claims, where models trained on infringing data may face deletion orders.

Where a TDM opt-out reserves the right to refuse, a Content Licensing API provides the programmatic on-ramp for authorized use. These interfaces allow rightsholders to define granular, machine-readable terms for text and data mining, including:

• Permitted use cases (e.g., non-commercial research only)
• Attribution requirements
• Royalty micro-payment triggers This transforms the binary opt-out into a dynamic, monetizable permissioning layer.

Machine Unlearning is the technical remedy for a violated TDM opt-out. If a model is found to have ingested data in defiance of a rights reservation, unlearning algorithms attempt to excise the influence of that specific data from the model's weights without full retraining. This process is critical for enforcing the EU AI Act's transparency obligations and responding to post-hoc copyright infringement claims.

The Fair Use Doctrine is the primary legal counterweight to the TDM exception in the United States. While the EU's DSM Directive creates a statutory opt-out mechanism, US law relies on a four-factor balancing test. Courts evaluate whether scraping for AI training constitutes transformative use, weighing the purpose, nature, amount, and market impact. The absence of a formal TDM opt-out in the US makes licensing and technical barriers the dominant control strategy.