Rate Limiting

The simplest rate limiting strategy that divides time into discrete, fixed intervals (e.g., 60 seconds). A counter tracks requests within the current window, resetting to zero when the window expires.

• Mechanism: If the counter exceeds the threshold, all subsequent requests are rejected until the next window begins
• Weakness: Susceptible to boundary burst attacks, where a client sends the full quota at the very end of one window and immediately again at the start of the next
• Use case: Suitable for coarse-grained limits where precision is not critical
• Example: Allowing 100 requests per minute, resetting on the clock boundary (e.g., 12:00:00 to 12:00:59)

A precise algorithm that timestamps every request and evaluates the count within a rolling lookback period, eliminating the burst vulnerability of fixed windows.

• Mechanism: For each new request, the system counts all timestamps in the preceding time window (e.g., the last 60 seconds)
• Advantage: Provides exact rate enforcement with no boundary conditions
• Trade-off: Higher memory consumption due to storing individual timestamps per client
• Example: A client is limited to 100 requests in any rolling 60-second period, regardless of when the clock started

A flexible algorithm that allows short bursts while maintaining a long-term average rate, modeled as a bucket that fills with tokens at a steady rate.

• Mechanism: Tokens are added to the bucket at a fixed rate (e.g., 10 tokens/second). Each request consumes one token. If the bucket is empty, the request is rejected
• Burst capacity: The bucket has a maximum size, allowing accumulated tokens to handle traffic spikes
• Advantage: Smooths out bursty traffic patterns without penalizing legitimate spikes
• Example: A bucket with a capacity of 50 tokens refilling at 10 tokens/second allows a burst of 50 requests followed by a sustained rate of 10 requests/second

A traffic shaping algorithm that processes requests at a constant rate, queuing excess requests and discarding them if the queue overflows.

• Mechanism: Requests enter a FIFO queue and are processed at a fixed rate. If the queue is full, incoming requests are dropped
• Key difference from token bucket: Leaky bucket enforces a smooth output rate with no bursting, making it ideal for traffic shaping rather than just policing
• Use case: Protecting downstream services that cannot handle traffic spikes
• Example: A queue of size 100 processing at 20 requests/second ensures the backend never receives more than 20 requests/second

Coordination of rate limits across multiple API gateway instances using a shared state store, ensuring consistent enforcement in horizontally scaled deployments.

• Mechanism: A centralized data store like Redis or Memcached maintains counters accessible to all gateway nodes
• Consistency models: Implementations must balance accuracy vs. latency—eventually consistent counters may allow slight overages but reduce synchronization overhead
• Race conditions: Atomic operations like INCR and EXPIRE in Redis prevent counter corruption
• Example: A cluster of 10 API gateways all checking a shared Redis key to enforce a global limit of 1000 requests/second across all nodes

Standard HTTP headers that communicate rate limit status to clients, enabling them to self-regulate and avoid unnecessary retries.

• X-RateLimit-Limit: The maximum number of requests allowed in the window
• X-RateLimit-Remaining: The number of requests remaining in the current window
• X-RateLimit-Reset: The timestamp (Unix epoch) when the window resets
• Retry-After: Returned with a 429 Too Many Requests status, indicating how long to wait before retrying
• Best practice: Always include these headers to enable client-side backoff and reduce wasted requests

The token bucket is the most common rate-limiting algorithm in API gateways. Tokens are added to a bucket at a fixed rate. Each request consumes a token; if the bucket is empty, the request is rejected. This allows for burst handling—a client can send a burst of requests up to the bucket's capacity while still being constrained by the long-term average rate. In AI inference, this prevents a single tenant from monopolizing GPU compute while accommodating short-term traffic spikes.

The sliding window log algorithm timestamps every request and counts how many occurred in the preceding time window. Unlike fixed windows, this eliminates boundary burst attacks where a client floods requests at the window edge. For zero-trust AI endpoints, this provides precise, per-client accounting. The trade-off is higher memory consumption, as each request timestamp must be stored. Redis sorted sets are commonly used to implement this efficiently.

The leaky bucket algorithm processes requests at a constant rate, using a FIFO queue to smooth out bursty traffic. Excess requests are queued rather than rejected, up to a configured depth. This is ideal for model inference pipelines where backpressure is preferred over dropped requests. When the queue overflows, requests are discarded. This pattern maps naturally to GPU batch scheduling, where a fixed-size queue feeds the inference engine at a steady processing rate.

In horizontally scaled AI infrastructure, rate limiting must be globally consistent across multiple API gateway instances. A centralized counter store—typically Redis or Memcached—maintains the state. The Race Condition Problem is solved using atomic increment operations and Lua scripting. For zero-trust deployments, the counter store itself must reside within the trusted compute boundary and be accessed over mutually authenticated TLS to prevent counter manipulation by compromised nodes.

In a zero-trust AI platform serving multiple organizations, rate limiting enforces tenant isolation at the API layer. Each tenant receives a guaranteed request quota, preventing a noisy neighbor from degrading inference latency for others. Quotas are defined in the Policy Decision Point (PDP) and enforced at the Policy Enforcement Point (PEP)—typically the API gateway. Attributes like tenant tier, contracted tokens-per-minute, and real-time GPU utilization inform dynamic quota adjustments.

When a rate limit is exceeded, the server must communicate the state clearly. The standard HTTP response is 429 Too Many Requests with a Retry-After header indicating when the client can resume. For AI workloads, additional headers like X-RateLimit-Remaining and X-RateLimit-Reset provide transparency. Well-behaved clients implement exponential backoff with jitter to avoid thundering herd retries. In zero-trust environments, these headers are cryptographically signed to prevent spoofing.

The architectural component in a zero-trust network that evaluates access requests against policy and attributes to issue an allow or deny decision. In rate limiting, the PDP determines whether a request exceeds the defined threshold by querying the current counter state.

• Decouples policy evaluation from enforcement logic
• Evaluates attributes like client ID, IP, endpoint, and time window
• Returns a binary decision consumed by the Policy Enforcement Point (PEP)

A privileged access management practice where administrative permissions are granted for a limited, specific time window on an as-needed basis. JIT complements rate limiting by eliminating standing privileges that could be exploited for sustained API abuse.

• Access expires automatically after the defined window
• Reduces the blast radius of compromised credentials
• Often paired with temporary token issuance for API calls

The management and security inspection of data packets moving laterally between servers, containers, or workloads within a data center. Rate limiting applies to east-west traffic to prevent a compromised microservice from overwhelming peer services with excessive internal calls.

• Protects against denial-of-wallet attacks on internal APIs
• Enforces service-to-service quotas via sidecar proxies
• Critical for mesh architectures where services communicate directly

The ongoing process of re-authenticating and re-authorizing a user or device's identity and security posture throughout an active session. Rate limiting integrates with continuous verification by adjusting quotas dynamically based on real-time risk signals.

• High-risk sessions receive stricter rate limits
• Device posture changes can trigger quota reductions
• Eliminates the assumption that an authenticated session is always trusted

A security protocol that hides services by requiring a cryptographically signed packet before a firewall dynamically opens a port. SPA works alongside rate limiting to prevent unauthorized clients from even reaching the rate limiter, reducing noise from malicious scanners.

• Services remain invisible to unauthenticated clients
• Mitigates DDoS by dropping packets before application-layer processing
• Implements port knocking with modern cryptographic guarantees