Crawler Allowlist

An allowlist operates on a default-deny security posture, where all automated traffic is blocked unless it matches a pre-approved identity. This is the inverse of a blocklist, which permits all traffic except known malicious actors.

• Identity Verification: Each entry must be authenticated via multiple signals, not just a self-reported User-Agent string.
• Zero-Trust Alignment: No crawler is trusted by default; access is granted only after positive identification against the registry.
• Override Hierarchy: Allowlist entries take precedence over blanket ASN blocks or rate-limiting rules, ensuring partners like Googlebot are never throttled.

A robust allowlist entry does not rely solely on a User-Agent string, which is trivially spoofed. It binds multiple verifiable attributes into a composite identity.

• Forward Confirmed Reverse DNS: The IP resolves to a hostname, and that hostname resolves back to the same IP, confirming network ownership.
• TLS Fingerprint (JA4): The specific cipher suites and extensions in the Client Hello must match the known fingerprint of the legitimate crawler.
• ASN Verification: The originating Autonomous System Number must belong to the verified operator's registered network range.
• HTTP Header Order: The sequence and capitalization of request headers must match the known template for that bot.

Legitimate crawlers like Googlebot and Bingbot publish their IP ranges via public DNS TXT records and JSON endpoints. An effective allowlist must dynamically consume these feeds rather than relying on static configurations.

• Automated Ingestion: The system periodically fetches published IP ranges from _crawl-public-ips.lexicon.example.com TXT records.
• Cache with TTL: Ranges are cached locally with a defined time-to-live to prevent latency on every request while ensuring freshness.
• Reverse DNS Fallback: If an IP is not in the cached range, a real-time reverse DNS lookup against the verified domain suffix (e.g., *.googlebot.com) serves as a secondary check.

The allowlist must maintain a canonical list of User-Agent tokens that are considered legitimate, with strict pattern matching to prevent impersonation.

• Full String Matching: The entire User-Agent string is compared against the registry, not just a substring match. Googlebot/2.1 is distinct from FakeGooglebot/2.1.
• Token Versioning: As crawlers update their software, their User-Agent strings evolve. The registry must track historical and current tokens.
• AI-Specific Tokens: Modern allowlists must include tokens like GPTBot, CCBot, and Google-Extended to govern generative AI ingestion separately from search indexing.
• Disallow vs. Noindex: The allowlist works in conjunction with robots.txt directives; a permitted crawler may still be restricted from specific paths.

Even with verified identity, a permitted crawler must exhibit behavior consistent with its declared purpose. Anomalous patterns trigger temporary revocation.

• Request Rate Profiling: Googlebot typically fetches pages at a predictable, polite rate. A verified Googlebot IP issuing 100 requests per second is likely spoofed.
• URL Traversal Logic: Legitimate crawlers follow links methodically. A bot that jumps directly to deeply nested, unlinked URLs is suspicious.
• robots.txt Compliance: The crawler must fetch and honor robots.txt before proceeding. Failure to request this file is an immediate red flag.
• Fetch Metadata Headers: The Sec-Fetch-Site and Sec-Fetch-Mode headers must align with the expected navigation context of a crawler.

In the context of AI training data governance, an allowlist entry may carry additional metadata defining the scope of permitted use beyond simple access.

• Ingestion Purpose Tags: An entry for Google-Extended may be tagged with purpose: generative-ai-training, while Googlebot is tagged purpose: search-indexing.
• Content Licensing Integration: The allowlist can interface with a Content Licensing API to verify that a permitted crawler's operator has an active, paid license for training data extraction.
• Granular Path Permissions: A partner may be allowed to index public marketing pages but explicitly denied access to proprietary knowledge base articles via path-specific rules.
• Audit Trail: Every access decision based on the allowlist is logged immutably, creating a record of which licensed partners accessed which resources and when.