Exponential Backoff with Jitter Configuration Prompt Template

This prompt is designed for SRE and platform engineering teams who need to move beyond generic retry logic and generate precise, production-ready backoff parameters for API calls. The core job-to-be-done is calibrating a retry policy against a specific endpoint's Service Level Objectives (SLOs), error characteristics, and traffic patterns. You should use this prompt when you are designing a new retry policy from scratch, recalibrating an existing policy after an SLO update, or standardizing retry behavior across multiple services to prevent retry storms. The ideal user has access to endpoint latency percentiles, error rate budgets, and peak traffic profiles, and needs the output to be fed directly into a retry harness or SDK configuration, not just read as a document.

You should not use this prompt for simple client-side retries where a fixed delay is sufficient, or for fire-and-forget asynchronous jobs where response latency is irrelevant. It is also the wrong tool if you are debugging a single failed request—this prompt designs a policy, it does not diagnose an incident. The prompt requires concrete input parameters like [TARGET_ENDPOINT_SLO], [ERROR_BUDGET_REMAINING], and [PEAK_REQUESTS_PER_SECOND]. If you cannot provide these, the generated configuration will be based on assumptions and is not safe for production. The output is a structured configuration block, not a narrative explanation, making it suitable for direct ingestion by infrastructure-as-code tools or retry libraries.

Before using this prompt, ensure you have gathered the necessary context: the endpoint's p95 and p99 latency, its error budget as a percentage of total requests, and the maximum acceptable client-side wait time. After generating the configuration, you must validate it against your retry harness to ensure it does not violate your latency budget or cause a retry storm under peak load. The next step is to run the provided eval criteria, such as checking that the max_delay plus jitter does not exceed the client timeout, before deploying the policy to a staging environment.

This prompt template is designed to be pasted directly into your prompt library or retry policy generation tool. It instructs the model to act as an SRE engineer and produce a complete retry configuration for an API endpoint. The template uses square-bracket placeholders for all dynamic inputs, ensuring you can adapt it to any service without modifying the core instruction structure. The output is a strict JSON schema, making it suitable for direct consumption by application code or infrastructure-as-code modules.

textCopy
You are an SRE engineer designing a retry policy for an API endpoint. Your task is to produce an exponential backoff configuration with jitter, calibrated to the provided Service Level Objectives (SLOs) and endpoint characteristics.

**Inputs:**
- Endpoint SLOs: [ENDPOINT_SLOS]
- Endpoint Characteristics: [ENDPOINT_CHARACTERISTICS]
- Error Budget Remaining: [ERROR_BUDGET_REMAINING]
- Upstream Dependencies: [UPSTREAM_DEPENDENCIES]

**Constraints:**
- The total possible retry window must not exceed the endpoint's latency SLO.
- The configuration must prevent retry storms and synchronized thundering herd problems.
- The jitter strategy must be explicitly defined (e.g., 'full', 'decorrelated', 'equal').
- If the error budget is below 10%, prefer a more conservative configuration and flag this in the output.

**Output Schema:**
Produce a single JSON object with the following structure:
{
  "retry_policy_name": "string",
  "base_delay_ms": number,
  "max_delay_ms": number,
  "max_retries": number,
  "jitter_strategy": "full" | "decorrelated" | "equal",
  "backoff_multiplier": number,
  "total_retry_window_ms": number,
  "retryable_errors": ["string"],
  "non_retryable_errors": ["string"],
  "circuit_breaker_threshold": number,
  "rationale": "string"
}

To adapt this template, replace the placeholders with concrete data from your service catalog or monitoring system. For [ENDPOINT_SLOS], provide the target latency and availability (e.g., 'p99 latency < 200ms, availability > 99.9%'). For [ENDPOINT_CHARACTERISTICS], describe the endpoint's idempotency, typical response times, and request cost. The [ERROR_BUDGET_REMAINING] should be a percentage derived from your SLO burn rate. After generating the configuration, always validate the JSON output against the schema and run a simulation to ensure the total retry window does not violate the latency SLO. For high-risk financial or healthcare endpoints, a human reviewer must approve the generated configuration before it is applied to production infrastructure.

This prompt is designed to be called from within a retry middleware or sidecar process, not as a standalone chatbot interaction. The harness should invoke the model when a retryable error (e.g., 429, 503, connection timeout) is encountered and the current retry count is zero, meaning the system needs to compute a fresh backoff schedule. The prompt expects structured inputs: the endpoint's SLO targets, the error response details, and the current retry budget state. The output is a JSON configuration block that the harness parses and uses to set timers for subsequent retry attempts.

Harness integration steps: 1) Intercept the failed request and extract the HTTP status code, Retry-After header (if present), and the endpoint's latency SLO from your service catalog. 2) Populate the prompt's [ENDPOINT_SLO], [ERROR_CONTEXT], and [RETRY_BUDGET] placeholders. 3) Call the model with response_format set to JSON mode and validate the output against the expected schema before acting on it. 4) Parse the base_delay_ms, max_delay_ms, jitter_strategy, and max_retries fields. 5) For each subsequent retry attempt, compute the delay using the formula: delay = min(base_delay_ms * 2^attempt, max_delay_ms) with the specified jitter strategy applied (e.g., full_jitter = random(0, delay), decorrelated_jitter = random(base_delay_ms, delay)). 6) If the retry count exceeds max_retries or the cumulative delay breaches the endpoint's latency SLO, stop retrying and escalate to the dead-letter queue or incident management pipeline.

Validation and safety checks: Before enqueuing a retry, validate that the computed delay does not cause a retry storm by checking if the jitter window is wide enough to distribute requests across your fleet. Log every retry decision—including the computed delay, jitter strategy, attempt number, and the error that triggered it—to your observability platform for trace correlation. Implement a circuit breaker that monitors the failure rate over a rolling window; if the prompt's output is consistently producing max_retries values that are exhausted without success, the harness should temporarily open the circuit and route traffic to a fallback endpoint or return a cached response. Never use the prompt's output to override SLO-based hard stops: the harness must enforce a maximum cumulative retry window derived from the endpoint's p99 latency budget, regardless of what the model suggests.