Rate Limiting

Limit enforcement must be based on unique, hard-to-spoof identifiers. Relying solely on IP addresses is insufficient in the age of NAT and CGNAT. Effective strategies combine multiple attributes:

• API Key or Bearer Token: The primary identifier extracted from the Authorization header.
• Authenticated User/Subject Claim: The sub field from a validated JWT for per-user quotas.
• Combined Fingerprint: A hash of (user_id + resource_path) to apply distinct limits to different endpoints for the same client.
• IP as Secondary Signal: Used for coarse-grained blocking of volumetric attacks, not precise identification.

The choice of algorithm directly impacts fairness and burst tolerance under load. Common implementations include:

• Token Bucket: Allows short bursts of traffic up to the bucket's capacity while enforcing a long-term average rate. Ideal for APIs with spiky usage patterns.
• Leaky Bucket: Processes requests at a constant rate, queuing excess. Smoothes traffic into a steady flow, suitable for protecting downstream model inference capacity.
• Fixed Window Counter: Simple but suffers from a boundary condition where a client can double their allowed rate by sending requests at the very end of one window and the start of the next.
• Sliding Window Log/Logarithmic Counter: Tracks the exact timestamp of requests to calculate a precise rate over the last time unit, eliminating the fixed window boundary flaw at the cost of higher memory usage.

A well-behaved rate limiter must communicate its state to the client to enable backpressure and avoid wasted retries. Standard response headers include:

• X-RateLimit-Limit: The maximum number of requests permitted in the current window.
• X-RateLimit-Remaining: The number of requests left for the client in the current window.
• X-RateLimit-Reset: A Unix timestamp indicating when the current window expires and the quota resets.
• Retry-After: Returned with a 429 Too Many Requests status code, specifying the number of seconds the client must wait before retrying.

In a horizontally scaled model serving deployment with multiple replicas, a centralized or eventually consistent counter store is mandatory. In-memory counters are insufficient.

• Centralized Data Store: Use Redis or Memcached with atomic INCR operations to maintain a global counter accessible by all API gateway instances.
• Consistency Trade-offs: Accept eventual consistency for performance, understanding that a client might slightly exceed the limit during a network partition. For strict enforcement, use synchronous replication, which adds latency.
• Local Caching with Sync: Implement a hybrid model where each node maintains a local, periodically synchronized cache of the global counter to reduce latency on every request.

The action taken when a limit is breached must balance security and user experience. A layered response is most effective:

• Immediate 429 Rejection: Return HTTP 429 Too Many Requests with a Retry-After header for clients slightly over the limit.
• Progressive Delays: Introduce artificial latency (slowloris defense) for clients that persistently exceed limits, consuming their resources without an immediate hard block.
• Temporary Blacklisting: For egregious abuse or clear model extraction patterns, dynamically add the client identifier to a short-lived blocklist at the WAF or API gateway level.
• Silent Dropping: In extreme DDoS scenarios, silently drop requests without a response to conserve server resources, as generating 429 responses still incurs a cost.

Rate limiting data is a critical signal for broader security observability. Integrate counters with your monitoring stack:

• Metric Export: Emit rate_limit.exceeded and rate_limit.near_limit counters to Prometheus or Datadog for visualization and alerting.
• UEBA Correlation: Feed rate limit violation logs into a User and Entity Behavior Analytics system. A sudden spike in 429 responses for a specific API key is a high-fidelity indicator of a credential compromise or an automated extraction script.
• Audit Trail: Log every 429 event with the full request context (identifier, endpoint, timestamp) to an immutable audit trail for forensic analysis and compliance reporting.

The process of cleaning and validating all data supplied to a model inference endpoint to neutralize malicious content before processing. This includes:

• Stripping or escaping control characters and script tags
• Validating tensor shapes and data types against expected schemas
• Detecting and rejecting known adversarial perturbation patterns
• Normalizing Unicode to prevent homograph attacks

Sanitization acts as the first line of defense, ensuring that even if a request passes rate limits, malformed or weaponized payloads cannot reach the model runtime.

The enforcement of a strict data contract—often defined by an OpenAPI Specification—on all inference requests. This rejects malformed inputs before they consume compute resources.

Key enforcement points:

• Type checking: Ensuring integers aren't passed where floats are expected
• Range constraints: Rejecting pixel values outside 0-255 in vision models
• Required field enforcement: Blocking requests missing mandatory features
• String length limits: Preventing buffer overflow attempts via oversized text inputs

Schema validation eliminates the class of attacks that exploit undefined model behavior through unexpected input structures.

A defensive mechanism that analyzes sequential query patterns to identify black-box extraction attacks. Attackers attempt to replicate proprietary model functionality by sending thousands of carefully crafted queries and training a clone on the responses.

Detection signals include:

• High query velocity from a single client across diverse input space
• Systematic exploration of decision boundaries via minimally perturbed inputs
• Queries that form a uniform sampling grid over the input domain
• Requests targeting rare or edge-case regions of the feature space

When combined with rate limiting, stealing detection can dynamically escalate responses—from throttling to active deception via honeyed outputs.

A mechanism defined by RFC 7662 that allows a resource server—such as a model serving API—to query the authorization server to determine the active state and metadata of a presented access token.

This enables:

• Real-time token revocation: Blocking access immediately when a credential is compromised
• Contextual rate limiting: Applying different quotas based on token scopes or client tiers
• Token metadata inspection: Verifying that the token's intended audience matches the model endpoint
• Expiration enforcement: Catching expired tokens even if the client caches them

Introspection transforms rate limiting from a simple counter into an identity-aware enforcement point tied to live authorization state.

A specialized WAF configured with rulesets designed for machine learning APIs that inspects and filters HTTP traffic before it reaches the inference endpoint.

ML-specific protections include:

• Prompt injection signatures: Detecting attempts to override system instructions in LLM APIs
• Payload size thresholds: Blocking abnormally large inputs that could trigger resource exhaustion
• Content-Type enforcement: Rejecting requests with mismatched or missing MIME types
• Rate-based rules: Implementing coarse-grained throttling as a first-pass filter before application-layer rate limiting

A WAF for ML operates at the network edge, absorbing volumetric attacks before they consume backend compute resources.

A chronological, tamper-proof record of all access and query events against a model serving system, stored in WORM-compliant storage to ensure non-repudiation.

What gets logged:

• Every inference request: Timestamp, client identity, model version, input hash
• Rate limit events: When throttling was triggered and which client was affected
• Authorization decisions: Granted and denied access attempts with rationale
• Token lifecycle events: Issuance, refresh, introspection, and revocation

Audit trails provide the forensic evidence needed to investigate model extraction attempts and demonstrate compliance with frameworks like SOC 2 and the EU AI Act.