Rate Limiting

The mathematical logic governing request admission determines burst tolerance and memory efficiency. Common algorithms include:

• Token Bucket: A fixed-capacity bucket refills with tokens at a steady rate. Each request consumes a token; bursts are allowed until the bucket empties.
• Leaky Bucket: Processes requests at a constant rate using a FIFO queue, smoothing bursty traffic into a steady outflow regardless of ingress spikes.
• Fixed Window Counter: Divides time into discrete intervals and counts requests per window. Simple but vulnerable to boundary double-burst attacks.
• Sliding Window Log: Maintains a timestamped log of requests and counts those within a rolling time frame, offering precision at the cost of memory overhead.

Servers communicate rate limit status to clients through standardized response codes and headers, enabling programmatic backoff:

• 429 Too Many Requests: The definitive signal that the client has exceeded its quota. The response should include a Retry-After header.
• Retry-After Header: Specifies either an HTTP-date or a delta-seconds value indicating when the client can safely retry.
• X-RateLimit- Headers*: A family of custom headers (X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset) providing clients with real-time quota telemetry to avoid hitting limits.

Rate limits can be applied at multiple logical layers, each addressing a different threat vector:

• Per-IP Address: The most basic scope, effective against simple scripts but easily bypassed with distributed proxy networks.
• Per-User Account: Tied to an authenticated session or API key, providing accurate attribution regardless of IP rotation.
• Per-Endpoint: Applies distinct thresholds to resource-intensive endpoints (e.g., /search) versus lightweight ones (e.g., /health).
• Global vs. Per-Tenant: In multi-tenant architectures, a global limit protects the entire service, while per-tenant limits ensure noisy-neighbor isolation.

In horizontally scaled systems, rate limiting state must be shared across instances to maintain a global view of client behavior:

• Centralized Data Store: A high-performance in-memory cache like Redis stores counters and token buckets, accessible by all application instances with atomic increment operations.
• Consistent Hashing: Routes requests from the same client to the same enforcement node, reducing the need for cross-node synchronization.
• Edge Enforcement: Deploying rate limiting logic at the CDN edge or API Gateway layer terminates abusive traffic before it reaches origin infrastructure, preserving backend capacity.

Rate limiting is not a binary allow/block decision. Graduated responses degrade the scraper's economics without impacting legitimate users:

• Tarpitting: Intentionally delaying responses to throttled clients, wasting the attacker's connection pool and reducing extraction throughput.
• Request Queuing: Holding excess requests in a priority queue and servicing them when capacity permits, rather than rejecting them outright.
• Degraded Service Mode: Serving stale or cached content to rate-limited clients instead of executing expensive database queries, maintaining availability while protecting backend resources.

Rate limiting operates as one layer in a defense-in-depth strategy against automated extraction:

• Fingerprinting Correlation: Combining rate counters with TLS fingerprinting and browser fingerprinting allows for more accurate identification of distributed scraping campaigns that rotate IPs.
• Challenge Escalation: When a client approaches a rate threshold, the system can inject a JavaScript challenge or CAPTCHA before the hard limit is reached, filtering bots without blocking legitimate users.
• Threat Intelligence Feeds: Pre-emptively applying stricter limits to IPs flagged in commercial threat feeds blocks known scraping infrastructure before it initiates extraction.