Retry with Exponential Backoff Instruction Prompt

This prompt is for platform engineers and SREs embedding retry logic into AI workflow harnesses where a model's tool call or API request has failed due to a transient error. The job-to-be-done is to programmatically generate a safe, jittered backoff schedule and a set of escalation triggers from raw error context, preventing the harness from hammering a degraded downstream service. The ideal user is a developer building an AI agent orchestrator, a CI/CD pipeline, or a distributed API gateway who needs a machine-readable retry plan—not a human-readable explanation of what went wrong.

Use this prompt when you have a specific error payload (HTTP status code, Retry-After header, error type, and current attempt count) and you need a structured output that dictates the next wait_ms, the next retry deadline, and the conditions for escalation. Do not use this prompt for one-off manual debugging, for errors that require a semantic fix (like a malformed SQL query), or when the failure is non-transient (e.g., an authentication error). The prompt assumes the caller has already classified the error as retryable; it does not perform root-cause analysis. The output must be consumed by a state machine that enforces the schedule, not by a human making a judgment call.

Before wiring this into production, define your max retry budget and total latency SLO. The prompt includes placeholders for [MAX_RETRIES] and [LATENCY_BUDGET_MS] to enforce these limits. If the generated schedule would exceed the budget, the output must trigger an escalation action instead of a retry. Always validate the output schema before sleeping the harness, and log every retry decision for observability. In high-risk financial or healthcare workflows, require a human approval step before executing the final escalation path.

The prompt below is designed to be embedded in a retry harness. It receives structured error context—including the HTTP status code, retry count, and any Retry-After header—and returns a machine-readable backoff instruction. The output is a strict JSON schema that your orchestration layer can parse to decide whether to sleep, retry immediately, or escalate to a human operator. This template is not a generic retry advisor; it is purpose-built for distributed systems where latency budgets, retry storms, and idempotency guarantees are first-class concerns.

textCopy
You are a retry scheduler for a distributed AI workflow harness. Your job is to produce a safe, jittered exponential backoff instruction from error context. You must never recommend an infinite retry loop. You must respect the latency budget and escalate when retries are futile.

## INPUT
- [ERROR_CONTEXT]: JSON object containing:
  - `status_code`: integer (HTTP status or internal error code)
  - `retry_count`: integer (number of attempts so far, 0-indexed)
  - `max_retries`: integer (hard cap on total attempts)
  - `latency_budget_ms`: integer (remaining time budget in milliseconds)
  - `retry_after_header`: string or null (value of Retry-After header, if present)
  - `error_type`: string (one of: "rate_limited", "server_error", "timeout", "conflict", "unavailable", "unknown")
  - `idempotency_key`: string or null (key for deduplication, if provided)

## OUTPUT_SCHEMA
Return ONLY a valid JSON object with these fields:
- `action`: "retry" | "escalate" | "fail"
- `delay_ms`: integer (milliseconds to wait before retry; 0 if action is not "retry")
- `jitter_ms`: integer (random jitter added to delay; 0 if action is not "retry")
- `escalation_reason`: string or null (required if action is "escalate" or "fail")
- `next_retry_at`: string or null (ISO 8601 timestamp for next attempt; null if action is not "retry")
- `idempotency_key_preserved`: boolean (true if the key should be reused on retry)

## CONSTRAINTS
1. For 429 (rate_limited): use the `retry_after_header` value if present and valid; otherwise calculate exponential backoff: base_delay = min(1000 * 2^retry_count, 60000).
2. For 5xx (server_error, unavailable): use exponential backoff with base_delay = min(1000 * 2^retry_count, 30000).
3. For 409 (conflict): delay = 500ms + random jitter up to 1000ms. Preserve the idempotency key.
4. For 408 (timeout): base_delay = min(2000 * 2^retry_count, 30000).
5. Always add jitter: random value between 0 and base_delay * 0.3.
6. If (delay_ms + jitter_ms) > latency_budget_ms, escalate immediately with reason "latency_budget_exceeded".
7. If retry_count >= max_retries, escalate with reason "max_retries_exceeded".
8. Never retry on 400, 401, 403, 404: escalate with reason "non_retryable_status".
9. If error_type is "unknown" and status_code is not in the retryable set, escalate.
10. Output ONLY the JSON object. No markdown, no explanation.

## EXAMPLES
Input: {"status_code": 429, "retry_count": 2, "max_retries": 5, "latency_budget_ms": 50000, "retry_after_header": null, "error_type": "rate_limited", "idempotency_key": "ikey-abc"}
Output: {"action": "retry", "delay_ms": 4000, "jitter_ms": 800, "escalation_reason": null, "next_retry_at": "2026-01-15T10:30:04.800Z", "idempotency_key_preserved": true}

Input: {"status_code": 500, "retry_count": 5, "max_retries": 5, "latency_budget_ms": 1000, "retry_after_header": null, "error_type": "server_error", "idempotency_key": null}
Output: {"action": "escalate", "delay_ms": 0, "jitter_ms": 0, "escalation_reason": "max_retries_exceeded", "next_retry_at": null, "idempotency_key_preserved": false}

## CURRENT REQUEST
Error context: [ERROR_CONTEXT]

To adapt this template, replace [ERROR_CONTEXT] with a serialized JSON object from your error handler. If your system uses gRPC status codes or custom error types instead of HTTP, update the error_type enum and the constraint rules accordingly. The output schema is intentionally flat to simplify parsing in retry middleware. If you need additional fields—such as a retry_reason for logging or a fallback_endpoint for circuit-breaking—extend the schema but keep the action field as the primary dispatch key.

Before deploying, validate that your harness enforces the max_retries and latency_budget_ms caps even if the model returns an unexpected action. The prompt is an instruction, not a guarantee. In high-risk payment or provisioning workflows, add a human-approval gate when action is escalate and log the full error context alongside the model's escalation reason for auditability.

This prompt is designed to be called by an orchestrator or gateway layer that already has access to the original failed request, the error response (HTTP status, tool-call error, validation failure), and a configurable retry policy. The prompt itself does not execute the retry—it generates the next retry instruction (delay, jitter, max-retry check, escalation trigger) that your harness then enforces. Keep the prompt stateless: pass in the full error context and retry history each time, and let the harness own the actual sleep, retry counter, and deadline clock.

Wiring pattern: 1) The harness catches a failure and serializes the error into [ERROR_CONTEXT] (status code, error message, retry-after header if present, tool name, argument snapshot). 2) The harness appends the current [RETRY_HISTORY] (attempt number, timestamps, previous backoff durations, and any partial results). 3) The harness calls the prompt with these inputs plus [MAX_RETRIES], [LATENCY_BUDGET_MS], and [ESCALATION_TARGET]. 4) The prompt returns a structured decision: action (retry | escalate | abort), delay_ms, jitter_ms, next_attempt_number, and reasoning. 5) The harness validates the output against a schema, checks that delay_ms does not exceed the remaining latency budget, and either sleeps and retries or routes to the escalation target. Never let the prompt directly sleep or make HTTP calls—the harness owns all side effects.

Validation and safety checks to add in the harness: Confirm that next_attempt_number is strictly greater than the current attempt and does not exceed [MAX_RETRIES]. Verify that delay_ms + jitter_ms is non-negative and does not exceed the remaining [LATENCY_BUDGET_MS]. If the prompt returns action: retry but the remaining budget is exhausted, override to escalate and log the override. Implement a circuit breaker that counts consecutive escalation decisions and stops calling the prompt entirely if the failure rate crosses a threshold (e.g., 5 escalations in 60 seconds). Log every prompt input and output for observability—this prompt is a control-plane decision and must be auditable. For high-risk workflows (payments, provisioning, clinical data), require a human-in-the-loop approval before the harness acts on an escalate decision that would notify an on-call engineer or modify production state.

Model choice and latency considerations: This prompt is latency-sensitive because it sits on the critical path of a retry loop. Use a fast, small model (e.g., GPT-4o-mini, Claude Haiku, or a fine-tuned local model) with a strict timeout (500–1000 ms). If the model call itself times out, the harness should fall back to a deterministic backoff formula (e.g., min(base_delay * 2^attempt + random_jitter, max_delay)) and log the fallback. Do not retry the retry-instruction prompt—that creates a nested retry storm. The prompt's output schema must be simple enough to validate with a JSON Schema validator before the harness acts on it; reject and fall back to deterministic backoff if validation fails.