User-Agent Filtering

The core mechanism involves parsing the User-Agent string sent by the client. This string typically identifies the browser, rendering engine, and operating system. Filtering logic compares this string against a blocklist (deny known bots) or an allowlist (permit only standard browsers).

• Example Blocklist Entry: python-requests/2.28
• Example Allowlist Entry: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
• Limitation: The header is client-controlled and trivially spoofed by any sophisticated scraper.

Distinguishing between organic browser traffic and scripted HTTP libraries is the primary goal. Organic signatures are complex and vary by browser version, while synthetic signatures from libraries like curl, wget, or python-requests are often simple and declarative.

• Organic: Long, complex strings with multiple tokens.
• Synthetic: Short, library-specific strings like Go-http-client/1.1.
• Headless Browsers: Tools like Puppeteer often use a HeadlessChrome token, which can be filtered, though it is frequently overridden.

Enterprise-grade filtering moves beyond simple string matching by correlating the User-Agent against known bot registries and threat intelligence feeds. This validates if a declared bot is legitimate.

• Legitimate Bots: Googlebot, Bingbot are verified via reverse DNS lookups, not just the User-Agent string.
• Imposter Bots: A scraper claiming to be Googlebot is blocked if its IP fails a forward-confirmed reverse DNS check.
• Registry Example: The IAB/ABC International Spiders and Bots List provides a standardized taxonomy.

A key defensive technique is detecting User-Agent/OS mismatch. If a client claims to be Chrome on Windows but its TCP/IP stack fingerprint or TLS handshake parameters indicate a Linux kernel, the request is flagged as fraudulent.

• TLS Fingerprinting: The JA4 hash of the Client Hello packet must be consistent with the claimed User-Agent.
• HTTP/2 Settings: Specific frame settings differ between browsers and libraries, providing a passive inconsistency signal.

User-Agent filtering is deployed at multiple layers of the infrastructure stack, each with different performance and granularity trade-offs.

• CDN/Edge: Blocking at the edge function layer prevents unwanted traffic from reaching the origin server.
• Web Application Firewall (WAF): Applies complex, custom rulesets that combine User-Agent with IP reputation and rate data.
• Application Logic: Used for basic feature gating or logging, but not as a primary security control.

Advanced scrapers bypass basic filters by rotating through a large pool of realistic, organic User-Agent strings extracted from recent browser releases. This makes static blocklisting ineffective.

• Rotation Pools: A single botnet node may cycle through hundreds of valid User-Agent strings per hour.
• Consistency Spoofing: Sophisticated tools modify not just the User-Agent but also related JavaScript properties (navigator.userAgent, navigator.platform) to defeat inconsistency checks.
• Countermeasure: This necessitates moving to passive, stateless identification like TLS fingerprinting, which the client cannot arbitrarily change.