Do Not Scrape

Do Not Scrape is a proposed preference signal directly inspired by the browser-based Do Not Track (DNT) header. The core idea is to provide a standardized, machine-readable mechanism for content owners to declare their objection to automated data extraction for AI training. Unlike DNT, which focused on behavioral tracking, this signal targets the ingestion of copyrighted text, images, and code by autonomous crawlers. It represents a shift from relying solely on server-side directives like robots.txt to a user-agent or browser-level declaration of intent.

Currently, a 'Do Not Scrape' signal has no universally recognized legal or technical standard. Unlike the Global Privacy Control (GPC), which has gained regulatory backing under laws like the CCPA, a scraping opt-out signal lacks a binding compliance framework. Its effectiveness is entirely dependent on the voluntary adherence of the AI developer or data aggregator operating the crawler. This creates a significant enforcement gap, as there is no legal penalty for ignoring the signal, making it a statement of preference rather than a technical access control.

There is no dedicated HTTP header or token standardized for 'Do Not Scrape'. Potential implementation vectors remain theoretical:

• HTTP Header: A custom request header (e.g., X-No-Scrape: 1) sent by a browser or proxy.
• Robots.txt Extension: A non-standard directive like User-agent: * Disallow-AI-Training: /.
• Meta Tag: An HTML meta tag (e.g., <meta name="ai-training" content="no-scrape" />). Without a formal RFC or W3C specification, these implementations are fragmented and unrecognized by most crawler codebases.

Do Not Scrape is a broader, less formal concept than the TDM Reservation Protocol. While TDM opt-out mechanisms specifically address the rights reservation for text and data mining under copyright law (often implemented via robots.txt), 'Do Not Scrape' is a more general expression of objection. It aims to cover all forms of automated extraction, not just those legally defined as TDM. The TDM protocol benefits from a specific legal basis in the EU's CDSM Directive, whereas 'Do Not Scrape' remains a purely aspirational signal without a corresponding legal framework.

The Global Privacy Control (GPC) serves as the closest functional analog and a potential evolutionary path. GPC is a browser-level signal that communicates a user's opt-out preference for the sale or sharing of personal data. Key differences:

• GPC: Legally recognized under CCPA; targets data sales.
• Do Not Scrape: No legal recognition; targets AI training ingestion. For a 'Do Not Scrape' signal to gain traction, it would likely need to follow GPC's model of securing regulatory endorsement and integrating directly into browser security settings.

For enterprises, relying on a non-binding 'Do Not Scrape' signal is a high-risk strategy for protecting proprietary data. The primary challenges include:

• Zero Enforcement: No legal recourse if a bad-actor AI crawler ignores the signal.
• Crawler Ignorance: Major AI labs have not publicly committed to respecting such a signal.
• False Sense of Security: It may distract from implementing robust, server-side controls like IP reputation blocking, rate limiting, and WAF rules. Until a binding standard emerges, it remains a supplementary, not a primary, defense mechanism.

The primary technical mechanism for a site-wide AI training opt-out, instructing compliant automated user agents not to access specified paths. This directive within the Robots Exclusion Protocol serves as the first line of defense against unauthorized scraping.

• Targets specific user-agent strings like GPTBot or CCBot
• Supports wildcard patterns for granular path exclusion
• Relies on voluntary compliance by crawler operators

A proposed universal browser-level signal that automatically communicates a user's opt-out preference for data sales and sharing, extending conceptually to AI training ingestion. GPC aims to make 'Do Not Scrape' a browser-native preference rather than a per-site configuration.

• Modeled after the deprecated Do Not Track header
• Legally recognized under CCPA and Colorado Privacy Act
• Requires server-side parsing to enforce

A tamper-evident metadata structure standardized by the C2PA (Coalition for Content Provenance and Authenticity) that attaches cryptographically signed provenance information to digital content. Content Credentials signal ownership and usage rights directly to AI ingestion systems.

• Embeds creator identity and usage restrictions
• Survives screenshotting and format conversion
• Enables automated rights enforcement at ingestion time

A legal provision under GDPR Article 21 granting individuals the absolute right to object to processing of personal data for direct marketing or legitimate interest purposes. This right can be invoked against AI profiling and training, providing a legal backstop where technical signals fail.

• Organizations must halt processing upon verified objection
• Applies to both controller and processor roles
• No exemption for 'legitimate interest' in AI training

The automated tracking of data's origin, movement, and transformation over time, providing a forensic audit trail to verify that training data has not been contaminated by unauthorized or opted-out sources. Data lineage transforms 'Do Not Scrape' from a request into a verifiable compliance posture.

• Captures source, transformations, and access events
• Enables post-hoc contamination detection
• Critical for model unlearning workflows