User-Agent Blocklist

The core mechanism relies on matching the User-Agent request header against a deny list of known AI crawler identifiers. Common targets include:

• GPTBot (OpenAI)
• CCBot (Common Crawl)
• anthropic-ai (Anthropic)
• Google-Extended (Google AI)

Wildcards and regex patterns handle version variants like GPTBot/1.0 or CCBot/2.0, ensuring broad coverage without manual updates for every minor release.

Blocklists are typically enforced through the Robots Exclusion Protocol using Disallow directives. A comprehensive robots.txt entry blocks agents across all paths:

codeCopy
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Granular path exclusion allows selective blocking—permitting access to public marketing pages while denying crawlers from proprietary documentation or API endpoints.

Advanced blocklists extend beyond static string matching to analyze traffic patterns and request signatures. Key behavioral indicators include:

• Request velocity: Abnormally high rates of page fetches
• TLS fingerprinting: JA3/JA4 hashes that identify bot clients regardless of declared user-agent
• Header ordering anomalies: Non-standard HTTP header sequences characteristic of automated tooling
• JavaScript challenge failures: Inability to execute client-side rendering checks

This multi-layered approach catches agents that spoof legitimate browser user-agent strings.

Blocklists are strengthened by cross-referencing request origins against known AI crawler IP ranges. Major providers publish their crawler IPs:

• OpenAI documents GPTBot's source IP ranges for firewall allow-listing
• Common Crawl publishes its crawler subnet

Reverse DNS lookups and ASN analysis verify whether traffic originates from cloud provider IPs (AWS, GCP, Azure) commonly used by unregistered scrapers. Combining IP reputation with user-agent validation prevents trivial header spoofing bypasses.

Server configurations can return 403 Forbidden or 406 Not Acceptable responses when matching blocked user-agent patterns. Implementation layers include:

• Web server level: Apache mod_rewrite or Nginx if directives
• CDN/WAF level: Cloudflare WAF rules or Fastly VCL conditions
• Application level: Middleware checks in frameworks like Express or Django

The X-Robots-Tag: noai, noindex HTTP response header provides an additional signal layer for compliant crawlers that respect header directives beyond robots.txt.

Effective blocklists require continuous curation as new AI crawlers emerge. Maintenance practices include:

• Monitoring IETF Bot Registry and industry threat feeds for new user-agent strings
• Logging 403 responses to audit blocklist effectiveness
• Testing with curl requests spoofing blocked user-agents to verify enforcement
• Version-controlling blocklist configurations alongside infrastructure-as-code

Verification ensures that legitimate search engine crawlers (Googlebot, Bingbot) are not inadvertently blocked, preserving organic search visibility while denying AI training ingestion.