Rate Limit Backoff Strategy Prompt

This prompt is for infrastructure and SRE engineers who need to generate a client-side retry plan when an external API returns rate-limit headers (HTTP 429, Retry-After, X-RateLimit-Remaining). The job is to produce a machine-readable backoff schedule—not a generic retry loop—that respects the server's stated limits, avoids thundering herd, and stays within a total deadline budget. Use it when you control the retry logic in your application code and need the model to compute a safe timing sequence from live response headers.

Do not use this prompt when you need a full circuit breaker implementation, when the API does not return standard rate-limit headers, or when the retry decision belongs inside a reverse proxy or API gateway rather than application code. It is also the wrong tool if you are designing a global rate-limiter across multiple clients; this prompt handles a single client's backoff plan against one endpoint's response. The prompt assumes you have already parsed the rate-limit headers and can inject them as structured input—it does not parse raw HTTP responses.

Wire this into your retry middleware: capture the rate-limit headers from the failed response, feed them into the prompt along with your deadline and max retries, and use the output JSON to configure your backoff executor. Validate the output before acting on it—reject any plan that exceeds your deadline, uses negative delays, or schedules more retries than your configured maximum. For high-throughput systems, consider caching the backoff plan per endpoint to avoid repeated model calls on identical header values.

This section provides a copy-ready prompt template that instructs a model to act as a client-side rate limit handler. The prompt is designed to be injected into a system or developer message in your AI harness. It takes raw rate limit headers and a request context as input, then produces a structured retry plan. The plan includes specific wait times with jitter, a maximum retry budget, and a deadline-aware abort condition to prevent thundering herds and wasted retries against an already overloaded service.

textCopy
You are a client-side rate limit handler for an API gateway. Your job is to produce a safe, standards-compliant retry plan based on the rate limit headers returned by a 429 Too Many Requests response.

## INPUT
- Rate Limit Headers: [RATE_LIMIT_HEADERS]
- Request Context: [REQUEST_CONTEXT]
- Current Time (UTC): [CURRENT_TIME]
- Absolute Deadline (UTC): [ABSOLUTE_DEADLINE]

## CONSTRAINTS
- You MUST respect the `Retry-After` header if present and valid.
- If `Retry-After` is absent, you MUST calculate a wait time using exponential backoff based on the attempt count in [REQUEST_CONTEXT].
- You MUST add full jitter to every calculated wait time: `wait = random_between(0, calculated_wait)`.
- The total number of retries MUST NOT exceed [MAX_RETRIES].
- If the next calculated retry time would exceed [ABSOLUTE_DEADLINE], you MUST abort and recommend a failure response.
- You MUST NOT suggest a retry if the `X-RateLimit-Remaining` header is 0 and no `Retry-After` or reset time is provided.

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "decision": "retry" | "abort" | "fail_open",
  "retry_after_seconds": number | null,
  "retry_at_utc": "ISO8601 timestamp" | null,
  "jitter_applied": boolean,
  "attempt_number": number,
  "reasoning": "string explaining the calculation and decision"
}

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

To adapt this template, replace each square-bracket placeholder with values from your application context. [RATE_LIMIT_HEADERS] should be a structured block of the actual headers from the 429 response (e.g., Retry-After, X-RateLimit-Remaining, X-RateLimit-Reset). [REQUEST_CONTEXT] should include the current attempt number and the original request method and path. [ABSOLUTE_DEADLINE] is critical for user-facing operations; set it based on your client's overall timeout budget. [MAX_RETRIES] should be a small integer, typically 3-5. [EXAMPLES] should be replaced with one or two few-shot examples of correct retry and abort decisions. [RISK_LEVEL] should be set to high for mutating operations (POST, PUT, DELETE) to enforce stricter abort conditions, or low for idempotent reads (GET). After adapting, test the prompt against a matrix of header scenarios including missing headers, malformed values, and edge cases where the deadline is imminent.

This prompt is designed to be called from within a retry middleware or sidecar that intercepts 429 Too Many Requests responses from upstream APIs. The harness is responsible for extracting the rate limit headers (Retry-After, X-RateLimit-Remaining, X-RateLimit-Reset), injecting them into the prompt's [RATE_LIMIT_HEADERS] placeholder, and executing the returned backoff plan. The model's job is not to perform the wait itself, but to produce a structured, jittered retry schedule that the application layer can enforce. This separation ensures the model handles the reasoning about backoff policy while the application retains control over actual timing and state mutation.

The implementation flow should follow a strict sequence: (1) The HTTP client receives a 429 and captures all standard and vendor-specific rate limit headers. (2) The harness constructs the prompt context by populating [RATE_LIMIT_HEADERS] with the raw header values, [MAX_RETRY_BUDGET_SECONDS] with the total time the caller is willing to wait, and [DEADLINE] with the absolute Unix timestamp after which the operation must be abandoned. (3) The model returns a JSON object containing retry_after_seconds, jitter_strategy (e.g., full_jitter, decorrelated_jitter), max_attempts, and a backoff_schedule array of recommended wait times. (4) The harness validates the output schema strictly: retry_after_seconds must not exceed the remaining budget, max_attempts must be a positive integer, and the sum of all scheduled waits plus estimated request latency must stay under the deadline. If validation fails, the harness should fall back to a safe default exponential backoff with full jitter and log the validation error for review.

To avoid thundering herd problems in distributed systems, the harness must ensure that the jitter strategy is applied client-side even if the model recommends a specific approach. The model's jitter_strategy field should be treated as a recommendation; the harness should enforce that any retry delay is randomized within the prescribed bounds. Additionally, the harness should maintain a circuit breaker that tracks consecutive 429 responses across multiple calls to the same endpoint. If the circuit opens, all retries should be suspended and the operation should escalate to a dead-letter queue or human review. Log every retry decision—including the prompt's raw output, the validated schedule, and the actual wait time used—so that backoff behavior is fully observable and tunable in production. The next step is to integrate this prompt into your existing HTTP client retry layer and run it against a rate-limited test endpoint to verify that the harness correctly respects deadlines and avoids synchronized retry storms.