Rate Limiting

A Fixed Window Counter algorithm tracks the number of requests in non-overlapping, consecutive time windows (e.g., per minute). The count resets at the start of each new window.

• Key Mechanism: A counter is incremented for each request in the current window. If the count exceeds the limit, subsequent requests are denied until the window resets.
• Use Case: Simple to understand and implement; useful for coarse-grained limits like daily API call quotas.
• Critical Drawback: Suffers from the boundary problem, where a burst of requests at the end of one window and the start of the next can allow 2x the intended limit in a short period (e.g., 120 requests in two adjacent minutes for a 60/minute limit).

The Sliding Window Log algorithm maintains a timestamped log of each request. The rate limit is enforced by counting requests within a dynamically moving time window preceding the current request.

• Key Mechanism: For a new request at time t, the system counts all previous requests with timestamps > t - window_size. If the count is below the limit, the request is allowed and its timestamp is logged.
• Use Case: Provides high precision and fairness, accurately enforcing limits for any rolling period. Used in financial trading APIs and strict security enforcement.
• Trade-off: Memory usage grows with request volume, as timestamps must be stored for the duration of the window. Requires efficient pruning of old entries.

The Sliding Window Counter is a hybrid algorithm that approximates the sliding window's precision with the fixed window's memory efficiency. It calculates a weighted count based on the current and previous window.

• Key Mechanism: It divides time into fixed windows but calculates the estimated count for a sliding window. For a limit of R per minute: count = previous_window_count * (time_elapsed_in_current_window / window_size) + current_window_count.
• Use Case: A practical compromise, offering better fairness than a pure fixed window without the storage overhead of a full log. Common in distributed rate limiting systems like Redis-based limiters.
• Characteristic: It is an estimation, not an exact count, but is highly accurate for enforcement and avoids the fixed window's boundary problem.

The Generic Cell Rate Algorithm (GCRA), derived from telecommunications (ATM networks), is a formalized version of the leaky bucket. It uses a Theoretical Arrival Time (TAT) to enforce a sustained rate and a maximum burst size.

• Key Mechanism: For each request, the algorithm calculates when the next request would be theoretically permitted based on the rate. If the actual arrival time is before this theoretical time (minus a tolerance for burst), the request is denied.
• Use Case: The foundation for token bucket implementations in many libraries. Provides mathematical guarantees on peak emission rate and burst tolerance. Used in systems requiring strict traffic conformance, such as network QoS.
• Precision: It is a canonical algorithm for rate limiting, defining parameters for Emission Interval (1/rate) and Delay Variation Tolerance (burst size).

The active enforcement of rate limits by deliberately delaying or rejecting requests that exceed a predefined threshold. While rate limiting defines the policy, throttling is the execution of that policy.

• Can be hard (request rejection with HTTP 429) or soft (request queuing and delayed processing).
• Protects LLM inference endpoints and external API calls made by agents from being overwhelmed.
• A core concern in agentic observability and telemetry for monitoring system health and fairness.

A classic algorithm for implementing rate limiting and network traffic shaping. It models a bucket that fills with tokens at a constant rate; each request consumes a token and can only proceed if tokens are available.

• Parameters: Bucket capacity (burst allowance) and token refill rate (sustained limit).
• Allows for short bursts of traffic up to the bucket's capacity, providing more flexibility than a strict sliding window.
• Widely used in API gateways and to manage tool calling and API execution frequency for agents.

The broader administrative system for defining, allocating, tracking, and enforcing usage limits across users, tenants, or agents. Rate limiting is a technical enforcement mechanism within a quota management framework.

• Involves metering (counting requests), accounting, and policy definition.
• Essential for multi-tenant AI platforms and enterprise AI governance to ensure fair resource distribution and cost control.
• Often integrated with billing systems and LLMOps platforms to track consumption against licenses.

A defensive strategy where a system intentionally drops or degrades non-critical requests or services to maintain stability under extreme load. It is a more aggressive form of protection than standard rate limiting.

• Prioritizes critical traffic (e.g., memory retrieval for active agent sessions) over non-critical traffic (e.g., background indexing).
• Prevents cascading failures and thrashing in distributed agentic memory architectures.
• A key technique in building resilient systems for autonomous supply chain intelligence or smart grid optimization.