Rate Limit Retry-After Compliance Prompt

This prompt is designed for reliability engineers and platform operators who need to ensure that AI agents strictly obey server-specified retry windows defined by HTTP Retry-After headers. The core job-to-be-done is converting a raw 429 (Too Many Requests) response into a compliant, delayed retry schedule that respects the server's instructions, communicates the delay to end users, and fails safely if the wait exceeds a maximum acceptable threshold. Use this when your agent's tool-calling layer must be provably compliant with external API rate limits to avoid being throttled or banned.

The ideal user is an engineer integrating an AI agent into a production system where upstream APIs enforce rate limits with specific Retry-After values (in seconds or as an HTTP-date). Required context includes the raw rate limit response headers, the current system time, and a pre-defined [MAX_WAIT_SECONDS] policy. Do not use this prompt for generic backoff strategies where the server does not provide a Retry-After header; for those cases, use the Rate Limit Header Parsing and Backoff Prompt Template instead. This prompt is also inappropriate for enforcing client-side, self-imposed rate limits where no server instruction exists.

Before implementing, confirm that your agent harness can parse HTTP response headers and inject them into the prompt's [RATE_LIMIT_RESPONSE_CONTEXT] placeholder. The output is a structured decision, not just a delay timer, so you must wire the resulting retry_at_iso8601 timestamp into your agent's scheduler. Avoid using this prompt in fire-and-forget scenarios where the agent cannot be paused and resumed; the entire value lies in the agent's ability to halt execution and resume precisely when the server permits.

This template instructs an agent to respect HTTP 429 responses by parsing the Retry-After header, calculating a compliant retry schedule, and communicating the delay to the user. It is designed for reliability engineers embedding rate-limit compliance directly into agent tool-call logic. The prompt forces the agent to treat the server's retry window as authoritative, preventing the common failure mode where agents ignore backoff instructions and hammer throttled endpoints.

textCopy
You are an agent that strictly obeys rate-limit responses from external services.

When a tool call returns an HTTP 429 status code, you MUST:
1. Extract the value from the `Retry-After` response header.
2. If the value is a delay in seconds, schedule the retry exactly that many seconds from the current time.
3. If the value is an HTTP-date, calculate the delay as the difference between that date and the current time.
4. Apply jitter of +/- [JITTER_RANGE] seconds to avoid thundering-herd effects, but never retry earlier than the server-specified window.
5. If the calculated retry time exceeds [MAX_WAIT_SECONDS], do NOT retry. Instead, return a deadline-exceeded response.

For every rate-limited response, produce a structured output:
{
  "action": "retry" | "deadline_exceeded",
  "retry_after_seconds": [PARSED_DELAY],
  "scheduled_retry_at": "[ISO8601_TIMESTAMP]",
  "jitter_applied_seconds": [JITTER_VALUE],
  "user_message": "[CLEAR_EXPLANATION_OF_DELAY]",
  "remaining_budget_seconds": [MAX_WAIT_SECONDS_MINUS_TOTAL_WAITED]
}

If the action is "deadline_exceeded", the user_message must explain that the retry window exceeds the maximum allowed wait time and suggest alternative actions.

[CONSTRAINTS]
- Never retry before the server-specified Retry-After window.
- Never ignore a 429 response and retry immediately.
- Track cumulative wait time across multiple retries for the same task.
- If the same endpoint returns 429 on the retry, extract the new Retry-After value and recalculate.

[EXAMPLES]
Example 1: Retry-After: 120, current time 10:00:00, jitter +5s, max wait 300s
Output: action=retry, retry_after_seconds=120, scheduled_retry_at=10:02:05, jitter=+5, user_message="Rate limited. Retrying in approximately 2 minutes."

Example 2: Retry-After: 600, current time 10:00:00, max wait 300s
Output: action=deadline_exceeded, retry_after_seconds=600, user_message="Server requested a 10-minute wait, which exceeds the 5-minute maximum. Task will not be retried."

Adapt this template by adjusting [JITTER_RANGE] and [MAX_WAIT_SECONDS] to match your service's SLA and user experience requirements. For high-throughput systems, keep jitter tight (1-5 seconds) to avoid excessive drift. For user-facing agents, set [MAX_WAIT_SECONDS] to a value that aligns with acceptable response latency—typically 30-300 seconds depending on the workflow. The structured output schema is designed for direct ingestion into logging and monitoring pipelines; do not remove fields without updating downstream consumers. Before deploying, validate that your agent's tool-call harness actually passes HTTP response headers into the prompt context—this is the most common integration gap.

This prompt is not a standalone chatbot instruction. It is a compliance function that should be called inside a tool execution wrapper whenever an HTTP 429 response is received. The wrapper must extract the Retry-After header value (either a delay-seconds integer or an HTTP-date), capture the tool name and arguments that triggered the rate limit, and pass them into the prompt as [RATE_LIMIT_RESPONSE]. The model's job is to produce a structured compliance decision, not to execute the retry itself. Your application code owns the actual sleep timer and retry dispatch.

Wire the prompt into a retry decision pipeline with these stages: (1) The HTTP client intercepts a 429 response and parses the Retry-After header into a normalized delay-seconds value. If the header is an HTTP-date, compute the delta from the current server time. (2) If the computed delay exceeds [MAX_WAIT_SECONDS], bypass the prompt entirely and escalate to the [ESCALATION_HANDLER] immediately—no model call needed. (3) Otherwise, invoke the prompt with the parsed delay, the tool identifier, the attempt count, and the session's remaining budget. (4) Parse the model's JSON output and validate that retry_after_seconds matches the header-derived value exactly. If the model hallucinates a different delay, reject the output and use the header value directly. (5) If the decision is retry, enqueue a delayed job using your scheduler (e.g., Celery, Temporal, BullMQ) with the exact delay plus optional jitter. If escalate, route to the escalation handler with the model's user_message and reasoning. If abort, log the termination and release any held resources.

Validation and safety checks are mandatory. Before acting on the model's output, confirm: retry_after_seconds is a positive integer within [MAX_WAIT_SECONDS], decision is one of the three allowed enum values, and user_message is present and non-empty for escalate and abort decisions. If the model returns retry but the cumulative retry time for this tool would exceed the session's remaining [BUDGET_REMAINING], override the decision to escalate with a budget-exhaustion reason. Log every decision—including overrides—as a structured audit record with the tool name, attempt number, header value, model decision, final action taken, and timestamp. This audit trail is essential for debugging rate-limit storms and proving compliance to API providers.

Model choice matters here. This prompt requires precise instruction-following and low hallucination on numeric values. Use a model with strong JSON mode and low latency—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Flash are appropriate. Avoid models known to drift on exact numeric constraints. Set temperature=0 and enable structured output mode with the [OUTPUT_SCHEMA] provided. If your platform cannot guarantee structured output, add a post-processing regex fallback that extracts the JSON block and re-validates. Do not use this prompt with a general-purpose chat model that might paraphrase the delay or invent a different backoff strategy.

What to avoid: Do not let the model control the actual timer or retry dispatch. The prompt advises on compliance; your code enforces it. Do not skip the exact-match check on retry_after_seconds—this is the most common failure mode, where the model rounds or adjusts the server-specified delay. Do not use this prompt for non-429 errors; 5xx retries have different semantics and should use a separate backoff prompt. Finally, do not deploy this without a circuit breaker that counts consecutive 429s per tool and escalates if the rate limit is not clearing after N attempts—otherwise you risk an infinite retry loop that the prompt alone cannot prevent.