Failover and Resilience Load Test Prompt

This prompt is for SREs and performance engineers who need to validate high-availability configurations under load. It generates a detailed test scenario that combines steady-state traffic with controlled failure injections, failover timing expectations, and data consistency checks during the transition. Use this prompt when you have a defined HA architecture, a known traffic profile, and a list of failure modes you want to exercise. The output is a test plan, not a script. It is designed to be fed into a load testing tool configuration or used as a specification for a chaos engineering experiment.

Do not use this prompt if you lack architectural diagrams, monitoring definitions, or baseline performance data. The prompt assumes you can describe your system's components, their dependencies, and the expected failover mechanism. Before invoking it, gather your service topology, health-check configurations, client retry policies, and any known timing thresholds for leader election, DNS propagation, or connection draining. The prompt will ask you to specify these as [ARCHITECTURE_DESCRIPTION], [FAILURE_MODES], and [TRAFFIC_PROFILE] placeholders. If you cannot articulate how your system detects failure and what 'healthy' looks like for each component, the generated scenario will be speculative rather than actionable.

This prompt is most effective when you have already completed baseline performance characterization and bottleneck hypothesis generation. It builds on the assumption that you know your system's steady-state behavior and are now ready to test its behavior during degradation. The output will include explicit validation criteria—such as maximum acceptable failover time, zero data loss assertions, and consistency check queries—that you should wire into your test harness as pass/fail gates. If your goal is to discover unknown failure modes rather than validate known ones, start with a chaos engineering steady-state hypothesis prompt instead.

This prompt template is designed to be copied directly into your prompt engineering environment or AI harness. It instructs the model to produce a detailed, executable failover and resilience load test scenario. The template uses square-bracket placeholders for all system-specific details, such as your architecture, failure modes, and acceptance criteria. Replace each placeholder with concrete values from your system's design documents, runbooks, and SLOs before sending the prompt to the model.

textCopy
You are a senior Site Reliability Engineer designing a failover and resilience load test.

Your task is to produce a complete test scenario that combines steady production-representative traffic with controlled failure injections. The scenario must validate both planned and unplanned failover paths, measure recovery timing, and check data consistency during the transition.

## SYSTEM CONTEXT
- Architecture Description: [SYSTEM_ARCHITECTURE_DESCRIPTION]
- High-Availability Configuration: [HA_CONFIGURATION_DETAILS]
- Component Dependency Map: [COMPONENT_DEPENDENCY_MAP]
- Data Consistency Requirements: [DATA_CONSISTENCY_REQUIREMENTS]

## TRAFFIC PROFILE
- Steady-State Load Model: [STEADY_STATE_LOAD_MODEL]
- Request Mix and Throughput Targets: [REQUEST_MIX_AND_THROUGHPUT]
- User Journey Scripts: [USER_JOURNEY_SCRIPTS]

## FAILURE INJECTIONS
- Planned Failover Triggers: [PLANNED_FAILOVER_TRIGGERS]
- Unplanned Failure Scenarios: [UNPLANNED_FAILURE_SCENARIOS]
- Network Partition or Latency Injection Parameters: [NETWORK_INJECTION_PARAMETERS]
- Resource Exhaustion Scenarios: [RESOURCE_EXHAUSTION_SCENARIOS]

## ACCEPTANCE CRITERIA
- Maximum Acceptable Failover Time (seconds): [MAX_FAILOVER_TIME]
- Allowed Error Budget Consumption: [ERROR_BUDGET_CONSUMPTION]
- Data Consistency Check Queries: [DATA_CONSISTENCY_CHECKS]
- Required Monitoring Signals: [MONITORING_SIGNALS]

## CONSTRAINTS
- Blast Radius Limits: [BLAST_RADIUS_LIMITS]
- Safe Abort Conditions: [SAFE_ABORT_CONDITIONS]
- Environment Restrictions: [ENVIRONMENT_RESTRICTIONS]

## OUTPUT SCHEMA
Produce a JSON object with the following structure:
{
  "scenario_name": "string",
  "objectives": ["string"],
  "preconditions": {
    "system_state": "string",
    "monitoring_readiness": ["string"],
    "rollback_plan": "string"
  },
  "load_phase": {
    "ramp_up_minutes": number,
    "steady_state_duration_minutes": number,
    "traffic_profile": {
      "requests_per_second": number,
      "write_percentage": number,
      "read_percentage": number,
      "key_endpoints": ["string"]
    }
  },
  "failure_injections": [
    {
      "injection_name": "string",
      "injection_time_seconds": number,
      "failure_type": "planned_failover | unplanned_crash | network_partition | resource_exhaustion",
      "target_component": "string",
      "injection_command_or_method": "string",
      "expected_blast_radius": "string"
    }
  ],
  "observations": [
    {
      "observation_name": "string",
      "metric_or_log_query": "string",
      "expected_behavior": "string",
      "failure_indicator": "string"
    }
  ],
  "data_consistency_checks": [
    {
      "check_name": "string",
      "query_or_procedure": "string",
      "expected_result": "string",
      "timing_relative_to_injection_seconds": number
    }
  ],
  "success_criteria": {
    "max_failover_time_seconds": number,
    "max_error_rate_percent": number,
    "data_consistency_required": boolean,
    "all_monitoring_alerts_triggered": boolean
  },
  "abort_conditions": ["string"],
  "risks_and_assumptions": ["string"]
}

## INSTRUCTIONS
1. Design the scenario so that steady traffic continues during the entire failover sequence.
2. Include at least one planned failover and one unplanned failure injection.
3. Specify exact monitoring queries that will confirm failover completion and data consistency.
4. Define clear abort conditions that prevent cascading failures beyond the blast radius.
5. Note any assumptions about the system state or configuration that could invalidate the test.

After copying the template, replace each placeholder with values from your system's architecture diagrams, runbooks, and SLO documents. The [SYSTEM_ARCHITECTURE_DESCRIPTION] should include component names, data flow directions, and redundancy mechanisms. For [FAILURE_INJECTION_PARAMETERS], be specific about the tooling you will use (e.g., Chaos Mesh, Gremlin, AWS Fault Injection Service) and the exact commands or API calls. The output schema is designed to be directly consumed by a test orchestration harness, so ensure all timing values and metric queries are machine-readable. Before executing the generated scenario, have a second SRE review the blast radius limits and abort conditions to confirm they match your environment's safety constraints.

This prompt is designed to be a planning and design artifact, not a direct executor. The primary integration point is a test planning pipeline where an SRE or performance engineer provides system architecture details, HA configuration, and failure injection constraints, and the prompt returns a structured test scenario document. The output should be treated as a human-reviewed specification before any load generation tool (k6, Locust, JMeter, Gatling) or chaos engineering platform (Gremlin, Chaos Mesh, Litmus) executes the scenario. Because failover tests carry inherent risk of production-impacting side effects, the harness must enforce a manual approval step between scenario generation and execution.

Integration workflow: (1) Collect inputs from your service catalog, runbooks, and HA configuration files into the [SYSTEM_ARCHITECTURE], [HA_CONFIGURATION], and [FAILURE_INJECTION_CONSTRAINTS] placeholders. (2) Submit the prompt to a model with strong reasoning capabilities (Claude 3.5 Sonnet or GPT-4o) with temperature set to 0.2–0.4 to balance creativity in failure scenario design with consistency. (3) Parse the output against the [OUTPUT_SCHEMA] using a JSON schema validator. If validation fails, retry once with the validation error injected into the [CONSTRAINTS] field. If it fails twice, escalate to a human with the raw output and error log. (4) Route the validated scenario to a review queue where an SRE must approve the failure injection plan, blast radius, and abort conditions before any automation executes it. (5) After approval, transform the scenario into tool-specific configuration (e.g., k6 test script, Gremlin experiment definition) using a separate translation step—do not ask the LLM to generate executable scripts directly.

Critical safety checks: Before execution, the harness must verify that the generated scenario includes explicit abort conditions, a defined blast radius, and expected failover timing thresholds. If any of these are missing or ambiguous, the scenario must be rejected regardless of schema validity. Log every generation attempt, validation result, and human approval decision for auditability. For high-risk environments (production, revenue-critical systems), require two-person approval. Never allow automated execution of a generated failover scenario without these gates—this is the single most important rule in the harness.