Rate Limiting

A Token Bucket algorithm models rate limits using a conceptual bucket that holds tokens. Tokens are added to the bucket at a steady refill rate. Each request consumes one or more tokens; if the bucket is empty, the request is denied. This allows for burst handling up to the bucket's capacity while enforcing a long-term average rate.

• Key Mechanism: A bucket with a maximum capacity C and a refill rate of R tokens per second.
• Use Case: Ideal for APIs where short bursts of traffic are acceptable, such as user-initiated actions.
• Example: A bucket with capacity 10 and refill of 2 tokens/sec can handle 10 immediate requests, then 2 requests per second thereafter.

The Leaky Bucket algorithm enforces a strict, smooth output rate. Incoming requests are placed in a queue (the bucket) and processed at a constant leak rate. If the queue is full, new requests are dropped or delayed. Unlike the Token Bucket, it does not allow bursts, ensuring a uniform flow of traffic.

• Key Mechanism: A FIFO queue with a fixed size N. Requests are processed (leak) at a constant rate R.
• Use Case: Protects downstream systems by transforming erratic input traffic into a steady stream, common in network traffic shaping.
• Implementation: Can be implemented as a counter with a timestamp for the last processed request.

A Fixed Window Counter algorithm divides time into contiguous, non-overlapping windows (e.g., 1-minute intervals). A counter is maintained for each window; it increments with each request and resets at the start of the next window. The request is allowed if the counter is below the limit.

• Key Mechanism: Simple counters per predefined time window (e.g., per minute, per hour).
• Limitation: Allows double the limit at window boundaries. A limit of 100/min could see 100 requests at 00:59 and another 100 at 01:00.
• Use Case: Suitable for simple, non-strict quotas where some boundary burst is acceptable, like daily login attempts.

The Sliding Window Log algorithm maintains a timestamped log of each request within the current time window. To check a new request, it counts the timestamps within the last N seconds. This provides high precision but requires storing potentially many timestamps, leading to higher memory overhead.

• Key Mechanism: Stores a timestamp for each request. The count is the number of timestamps within [current_time - window_size, current_time].
• Advantage: Eliminates the boundary burst problem of fixed windows.
• Use Case: Critical for enforcing strict, precise rate limits where even short bursts at boundaries are unacceptable, such as financial transaction APIs.

A Sliding Window Counter is a memory-efficient approximation of the sliding window log. It combines the current fixed window's count with a weighted portion of the previous window's count. This estimates the count for the rolling window without storing individual timestamps.

• Key Mechanism: If the window is 60 seconds, and a request arrives at 1:30, the algorithm might take 70% of the count from the 1:00-1:59 window and 30% from the 1:30-2:29 window.
• Trade-off: More memory-efficient than the log but is an approximation.
• Use Case: A practical default for many production systems needing reasonable precision without unbounded memory growth, like web application firewalls.

Quality of Service (QoS) is the overall performance measurement of a service, often defined by metrics like latency, throughput, and reliability. It encompasses the techniques used to manage network or system resources to meet specific performance targets.

• Relationship to Rate Limiting: Rate limiting is a core QoS mechanism. It prevents any single user or service from consuming excessive resources, thereby protecting the agreed-upon service quality for all other consumers.
• Key Metrics: Common QoS targets include packet loss rate, jitter, and bandwidth guarantees. Rate limiting directly influences these by controlling traffic flow.

Exponential Backoff is an algorithm used by clients to space out repeated retries of a failed operation (like a network request or API call). The waiting time between attempts increases exponentially, often with added randomness.

• Relationship to Rate Limiting: It is the standard, polite client-side response to hitting a rate limit (often signaled by an HTTP 429 status code). Instead of immediately retrying and worsening congestion, the client waits progressively longer.
• Algorithm Example: A client might wait 1 second, then 2, 4, 8, 16 seconds, etc., before giving up. This prevents retry storms that can exacerbate system load.

A Service Level Objective (SLO) is a key element of a service level agreement that defines a specific, measurable target for a service's reliability or performance—such as availability or latency—over a defined period.

• Relationship to Rate Limiting: Rate limiting is an engineering tool used to defend SLOs. By preventing resource exhaustion and denial-of-service conditions, rate limiting helps ensure the system remains within its defined performance and availability targets.
• Example SLO: "99.9% of API requests will have a latency under 100ms." Uncontrolled traffic spikes would violate this; rate limiting helps maintain it.

Auto-Scaling is a cloud computing feature that automatically adjusts the number of active compute resources (like virtual machines or containers) based on observed metrics such as CPU utilization or request rate.

• Relationship to Rate Limiting: These are complementary strategies for managing load. Auto-scaling reacts to increased demand by adding resources, while rate limiting defines the maximum acceptable demand per user.
• Strategic Use: Rate limiting often defines the upper bound for scaling. If traffic hits a global rate limit, scaling may be paused, as adding more resources would not increase the permitted throughput.

A Load Balancer is a network device or software application that distributes incoming network traffic across multiple backend servers to improve responsiveness, availability, and utilization.

• Relationship to Rate Limiting: Load balancers are a common enforcement point for rate limiting. They can apply limits per client IP, per API key, or globally before traffic reaches application servers.
• Integrated Functions: Modern load balancers (e.g., API gateways, cloud load balancers) often have built-in rate limiting capabilities, acting as the first line of defense.

A Circuit Breaker is a design pattern used in software development to detect failures and prevent an application from repeatedly trying to execute an operation that's likely to fail, allowing time for the failing service to recover.

• Relationship to Rate Limiting: Both are stability patterns. Rate limiting prevents overload, while a circuit breaker reacts to it. If a downstream service starts failing (potentially due to being overwhelmed), the circuit breaker "opens" and fails fast, preventing cascading failures.
• Key Difference: Rate limiting is proactive and quota-based. A circuit breaker is reactive and failure-based.