Rate Limiting

The token bucket is the most widely adopted rate limiting algorithm for model serving. A bucket is continuously refilled with tokens at a fixed rate up to a maximum burst capacity. Each inference request consumes one or more tokens, with token cost often proportional to input token count or requested max_tokens. This allows clients to burst up to the bucket size while maintaining a long-term average rate. Key properties: permits short-term bursts without violating sustained limits, naturally handles variable-cost requests, and can be implemented with constant-time complexity using atomic operations on a last-refill timestamp and current token count.

The sliding window log maintains a timestamped record of every request within a rolling time window. When a new request arrives, the algorithm evicts expired timestamps and counts remaining entries. If the count exceeds the limit, the request is rejected. This provides exact rate enforcement with no approximation, making it ideal for strict compliance scenarios. Trade-offs: higher memory consumption proportional to request volume, and O(log N) or O(N) complexity for log maintenance. Often implemented using sorted sets in Redis with ZREMRANGEBYSCORE for efficient eviction.

The fixed window counter divides time into discrete intervals and maintains a simple counter that resets at each boundary. If the counter exceeds the limit within the current window, requests are rejected until the next reset. Critical weakness: burst traffic at window boundaries can result in up to 2x the configured limit being served, as requests pile up at the end of one window and the beginning of the next. Despite this imprecision, it remains popular due to minimal memory overhead and trivial implementation using atomic increment operations.

The leaky bucket enforces a smooth, constant outflow rate by queuing requests and processing them at a fixed interval. If the queue overflows, excess requests are discarded. Unlike the token bucket, the leaky bucket eliminates all burstiness, producing a perfectly uniform processing rate. This is particularly useful for protecting downstream systems that cannot handle traffic spikes, such as legacy databases or third-party APIs with strict rate contracts. Implementation: typically realized as a bounded queue with a scheduled consumer thread or a background worker draining at a configurable rate.

In horizontally scaled model serving deployments, rate limits must be enforced globally across all replicas. Distributed rate limiting requires a shared state store—typically Redis or Memcached—to maintain counters or token buckets accessible to all server instances. Challenges: network latency to the state store adds overhead to every request, race conditions require atomic operations (INCR, Lua scripts), and eventual consistency can cause over-admission during network partitions. Best practices: use local token buckets with periodic remote synchronization to reduce per-request latency, and implement circuit breakers to fail open or closed if the state store becomes unavailable.

Well-designed rate limiting systems communicate state to clients via standard HTTP headers, enabling intelligent client-side throttling. Essential headers: X-RateLimit-Limit (maximum requests per window), X-RateLimit-Remaining (requests left in current window), X-RateLimit-Reset (Unix timestamp when the window resets), and Retry-After (seconds to wait before retrying). For generative model endpoints, limits may be expressed in tokens rather than requests, using custom headers like X-RateLimit-Remaining-Tokens. Clients should implement exponential backoff with jitter on 429 Too Many Requests responses to avoid thundering herd retry storms.