Retry Storm Simulation Scenario Prompt

This prompt is for resilience engineers and SREs who need to design a controlled chaos experiment to test how their distributed system behaves under cascading retry amplification. The core job-to-be-done is quantifying the blast radius of a downstream latency spike before it causes a production incident. Use it when you have a multi-service architecture where at least two services are configured to retry on failure, and you need to model the multiplicative effect of those retries under degraded conditions. The ideal user has access to service-level retry configurations, client library documentation, and baseline latency metrics for the services in the call chain.

The prompt requires specific inputs to produce a valid simulation scenario: the retry policy for each service in the call path (max attempts, backoff strategy, jitter configuration), the normal and degraded latency profiles for the downstream dependency, and the expected request rate at the entry point. Without these inputs, the model will make assumptions that may not match your system's actual behavior. The prompt explicitly validates for missing idempotency assumptions—a critical safety check because non-idempotent operations under retry amplification can cause duplicate side effects that are worse than the latency degradation itself. It also predicts dead letter queue accumulation rates, which helps you size your DLQ storage and alerting thresholds before running the experiment.

Do not use this prompt for simple single-service timeout testing where no retry chain exists, or for systems where retries are disabled at every layer. It is also inappropriate for testing circuit breaker behavior in isolation—circuit breakers are a mitigation for retry storms, not the storm itself. If your goal is to validate that a circuit breaker opens correctly under load, use the Third-Party Dependency Failure Simulation Prompt instead. For systems where all retries are bounded by a single global timeout with no fan-out, the amplification factor will always be 1.0, making this prompt's complexity unnecessary. The prompt assumes you can inject latency at the downstream dependency; if you cannot safely inject latency in production or a production-like staging environment, you should first establish that capability before using this scenario.

Before running the generated scenario, validate the output against your actual service topology. The prompt asks the model to calculate an expected retry amplification factor, but this calculation depends on accurate retry configuration inputs. A common failure mode is providing the intended retry policy rather than the actual deployed configuration—check your client library defaults, service mesh settings, and any infrastructure-layer retries (like Envoy or Istio) that may add hidden retry layers. After generating the scenario, review the dead letter queue predictions against your DLQ's current depth and processing rate to ensure the experiment won't cause a DLQ backlog that outlasts the test window.

This prompt is a design tool, not an execution tool. It produces a scenario specification that you then implement in your chaos engineering platform of choice (Gremlin, Chaos Mesh, Litmus, or custom scripts). The output includes abort conditions and monitoring queries; wire these into your observability platform before starting the experiment. Always run this scenario in a controlled environment with a limited blast radius first, even if the prompt's calculations suggest the amplification is manageable. Retry storms can saturate thread pools, exhaust connection limits, and cascade into unrelated services in ways that static analysis cannot fully predict.

This prompt template is designed to be pasted directly into your AI harness for generating retry storm simulation scenarios. It instructs the model to produce a structured, executable scenario that resilience engineers can use to test cascading retry amplification in distributed systems. The output includes retry policy configurations, backoff strategy variations, downstream latency injection parameters, and expected retry amplification factor calculations. Replace each square-bracket placeholder with your system's specific architecture details, failure domains, and testing constraints before execution.

textCopy
You are a resilience engineering assistant. Your task is to design a retry storm simulation scenario for a distributed system.

## System Context
- Architecture: [ARCHITECTURE_DESCRIPTION]
- Services involved: [SERVICE_NAMES_AND_ROLES]
- Communication protocol: [SYNC_ASYNC_MIX]
- Current retry policies: [EXISTING_RETRY_CONFIGS]
- Known idempotency guarantees: [IDEMPOTENCY_DETAILS]
- Dead letter queue configuration: [DLQ_CONFIG]

## Scenario Requirements
- Failure injection point: [FAILURE_TARGET_SERVICE]
- Initial failure type: [TIMEOUT_ERROR_THROTTLE]
- Simulation duration: [DURATION]
- Request rate baseline: [BASELINE_RPS]

## Output Schema
Produce a JSON object with the following structure:
{
  "scenario_name": "string",
  "failure_injection": {
    "target_service": "string",
    "failure_type": "string",
    "latency_injection_ms": number,
    "error_rate_percent": number,
    "duration_seconds": number
  },
  "retry_policy_variations": [
    {
      "policy_name": "string",
      "max_retries": number,
      "backoff_strategy": "fixed|exponential|jittered|decorrelated",
      "initial_backoff_ms": number,
      "max_backoff_ms": number,
      "timeout_ms": number
    }
  ],
  "amplification_analysis": {
    "expected_amplification_factor": number,
    "calculation_method": "string",
    "peak_retry_rate_rps": number,
    "cascading_services_affected": ["string"]
  },
  "dead_letter_predictions": {
    "expected_dlq_volume": number,
    "accumulation_rate_per_second": number,
    "dlq_capacity_breach_risk": "low|medium|high|critical"
  },
  "idempotency_gaps": [
    {
      "service": "string",
      "operation": "string",
      "risk_description": "string",
      "recommended_safeguard": "string"
    }
  ],
  "monitoring_signals": [
    {
      "metric_name": "string",
      "warning_threshold": "string",
      "critical_threshold": "string",
      "observation_point": "string"
    }
  ],
  "safe_abort_conditions": ["string"]
}

## Constraints
- Assume no idempotency unless explicitly stated in the system context.
- Flag every operation that lacks idempotency guarantees as a risk.
- Calculate amplification factor as: (initial_requests * retry_multiplier * downstream_fanout) / initial_requests.
- Include at least three distinct backoff strategy variations.
- Identify cascading services that will receive amplified retry traffic.
- Predict dead letter queue accumulation if retries exhaust.
- Define clear abort conditions to prevent test-induced production impact.

## Validation Checks
After generating the scenario, self-check for:
1. Missing idempotency assumptions for any service in the retry path.
2. Unrealistic backoff ceilings that would cause retry overlap.
3. Amplification factor that ignores downstream fan-out.
4. Absent monitoring signals for each cascading service.
5. Abort conditions that trigger too late to prevent overload.

After pasting this template, adapt the placeholders to match your specific system. The [ARCHITECTURE_DESCRIPTION] should include service dependencies and call paths. The [EXISTING_RETRY_CONFIGS] field is critical—if left empty, the model will assume no retry policies exist and the amplification analysis will be incomplete. Run the output through a JSON schema validator before feeding it into your chaos engineering or load testing tool. For high-risk production-adjacent environments, require a human resilience engineer to review the scenario's abort conditions and blast radius before execution.

The retry storm simulation scenario prompt is designed to be integrated into a chaos engineering or resilience testing pipeline, not used as a standalone chat interaction. The prompt expects structured inputs—retry policy configurations, backoff strategy parameters, downstream latency profiles, and idempotency assumptions—and produces a machine-readable scenario definition that can be fed into fault injection tools like Gremlin, Chaos Mesh, or custom load generators. Treat the output as a test specification artifact that must pass validation before execution, not as a final runbook.

Wire the prompt into your testing harness by first collecting the required inputs from your service mesh configuration, retry middleware settings, and distributed tracing data. Pass these as structured JSON in the [RETRY_POLICIES], [BACKOFF_STRATEGIES], and [DOWNSTREAM_LATENCY_PROFILES] placeholders. The model will return a scenario definition containing retry amplification factor calculations, dead letter accumulation predictions, and missing idempotency flag warnings. Validate the output against a schema that checks: (1) amplification factor is a numeric value with explicit derivation steps, (2) dead letter queue size predictions include time-bounded accumulation rates, (3) any missing idempotency key assumptions are flagged with severity levels, and (4) the scenario includes safe-abort conditions tied to specific metric thresholds. Reject outputs that lack these fields or contain unresolvable placeholder references.

Before executing the generated scenario in production or staging, run it through a dry-run validation step that simulates the retry storm against a sandboxed service instance with identical retry configuration but isolated downstream dependencies. Log the predicted vs. actual amplification factor, dead letter accumulation rate, and any idempotency-related duplicate processing events. Store these validation results alongside the prompt version, input parameters, and model identifier for auditability. If the scenario involves production traffic, require explicit human approval with a review checklist covering blast radius boundaries, monitoring alert thresholds, and rollback procedures. Never execute retry storm scenarios without first confirming that circuit breakers and rate limiters are configured to contain cascading failures beyond the target service boundary.