Rate Limiting

A Token Bucket algorithm models rate limits as a bucket that fills with tokens at a steady rate. Each request consumes a token. This allows for burst handling up to the bucket's capacity while maintaining a long-term average rate.

• Key Mechanism: Tokens are added at a fixed interval (e.g., 1 token per 100ms). A request can proceed if a token is available.
• Use Case: Ideal for APIs where short bursts of traffic are acceptable, such as user-initiated actions in a web application.
• Implementation: Requires tracking a token count and a timestamp of the last refill. The algorithm is smoother than a fixed window for clients that use their burst allowance.

The Leaky Bucket algorithm enforces a strict, smooth output rate, regardless of input burstiness. Requests are queued in a bucket that "leaks" at a constant rate.

• Key Mechanism: Incoming requests are added to a queue (the bucket). A processor dequeues and handles requests at a fixed rate (the leak). If the queue is full, new requests are dropped or rejected.
• Use Case: Protects downstream systems by transforming erratic traffic into a steady, predictable stream. Common in network traffic shaping and payment processing systems.
• Contrast with Token Bucket: While Token Bucket allows bursts, Leaky Bucket smoothes them, enforcing a more rigid output pattern.

A Fixed Window Counter algorithm tracks the number of requests in non-overlapping, consecutive time windows (e.g., per minute). The count resets at the start of each new window.

• Key Mechanism: A counter is incremented for each request in the current window. If the count exceeds the limit, subsequent requests are denied until the window resets.
• Use Case: Simple to understand and implement; useful for coarse-grained limits like daily API call quotas.
• Critical Drawback: Suffers from the boundary problem, where a burst of requests at the end of one window and the start of the next can allow 2x the intended limit in a short period (e.g., 120 requests in two adjacent minutes for a 60/minute limit).

The Sliding Window Log algorithm maintains a timestamped log of each request. The rate limit is enforced by counting requests within a dynamically moving time window preceding the current request.

• Key Mechanism: For a new request at time t, the system counts all previous requests with timestamps > t - window_size. If the count is below the limit, the request is allowed and its timestamp is logged.
• Use Case: Provides high precision and fairness, accurately enforcing limits for any rolling period. Used in financial trading APIs and strict security enforcement.
• Trade-off: Memory usage grows with request volume, as timestamps must be stored for the duration of the window. Requires efficient pruning of old entries.

The Sliding Window Counter is a hybrid algorithm that approximates the sliding window's precision with the fixed window's memory efficiency. It calculates a weighted count based on the current and previous window.

• Key Mechanism: It divides time into fixed windows but calculates the estimated count for a sliding window. For a limit of R per minute: count = previous_window_count * (time_elapsed_in_current_window / window_size) + current_window_count.
• Use Case: A practical compromise, offering better fairness than a pure fixed window without the storage overhead of a full log. Common in distributed rate limiting systems like Redis-based limiters.
• Characteristic: It is an estimation, not an exact count, but is highly accurate for enforcement and avoids the fixed window's boundary problem.

The Generic Cell Rate Algorithm (GCRA), derived from telecommunications (ATM networks), is a formalized version of the leaky bucket. It uses a Theoretical Arrival Time (TAT) to enforce a sustained rate and a maximum burst size.

• Key Mechanism: For each request, the algorithm calculates when the next request would be theoretically permitted based on the rate. If the actual arrival time is before this theoretical time (minus a tolerance for burst), the request is denied.
• Use Case: The foundation for token bucket implementations in many libraries. Provides mathematical guarantees on peak emission rate and burst tolerance. Used in systems requiring strict traffic conformance, such as network QoS.
• Precision: It is a canonical algorithm for rate limiting, defining parameters for Emission Interval (1/rate) and Delay Variation Tolerance (burst size).

A predetermined algorithm that determines which items to remove from a cache when it reaches its capacity limit. These policies are fundamental to managing finite memory resources in agentic systems.

• Key types include Least Recently Used (LRU), Least Frequently Used (LFU), and Time-To-Live (TTL).
• Directly analogous to rate limiting, but focuses on space constraints rather than time-based request quotas.
• Essential for maintaining the performance of vector databases and in-memory caches that support agent context.

A flow control mechanism in data streaming systems where a fast-producing component is signaled to slow down when a downstream consumer cannot keep up. It is the reactive counterpart to the proactive control of rate limiting.

• Implements feedback loops to prevent system overload and data loss.
• Critical in multi-agent system orchestration where one agent's output becomes another's input.
• Often uses mechanisms like token buckets or sliding windows, similar to rate limiters, to regulate flow.

The active enforcement of rate limits by deliberately delaying or rejecting requests that exceed a predefined threshold. While rate limiting defines the policy, throttling is the execution of that policy.

• Can be hard (request rejection with HTTP 429) or soft (request queuing and delayed processing).
• Protects LLM inference endpoints and external API calls made by agents from being overwhelmed.
• A core concern in agentic observability and telemetry for monitoring system health and fairness.

A classic algorithm for implementing rate limiting and network traffic shaping. It models a bucket that fills with tokens at a constant rate; each request consumes a token and can only proceed if tokens are available.

• Parameters: Bucket capacity (burst allowance) and token refill rate (sustained limit).
• Allows for short bursts of traffic up to the bucket's capacity, providing more flexibility than a strict sliding window.
• Widely used in API gateways and to manage tool calling and API execution frequency for agents.

The broader administrative system for defining, allocating, tracking, and enforcing usage limits across users, tenants, or agents. Rate limiting is a technical enforcement mechanism within a quota management framework.

• Involves metering (counting requests), accounting, and policy definition.
• Essential for multi-tenant AI platforms and enterprise AI governance to ensure fair resource distribution and cost control.
• Often integrated with billing systems and LLMOps platforms to track consumption against licenses.

A defensive strategy where a system intentionally drops or degrades non-critical requests or services to maintain stability under extreme load. It is a more aggressive form of protection than standard rate limiting.

• Prioritizes critical traffic (e.g., memory retrieval for active agent sessions) over non-critical traffic (e.g., background indexing).
• Prevents cascading failures and thrashing in distributed agentic memory architectures.
• A key technique in building resilient systems for autonomous supply chain intelligence or smart grid optimization.