Circuit Breaker Open State Recovery Prompt

This prompt is for distributed systems engineers and SREs who need a structured, safe recovery plan after a circuit breaker trips. It is designed to be wired into an incident response harness or a reliability automation pipeline. The prompt consumes a trip event, downstream error metrics, and the current service configuration. It produces a step-by-step recovery plan that prioritizes preventing cascading failures and mandates a gradual traffic ramp-up.

Use this prompt when manual recovery runbooks are insufficient, when the system is complex enough that a misconfigured reset could cause a thundering herd or retry storm, or when you need an auditable, evidence-backed recovery recommendation before executing changes. The prompt is ideal for post-incident analysis where the system is stable but the path to safely closing the circuit is unclear. It is not a real-time actuation tool; it is a decision-support system for the recovery analysis phase.

Do not use this prompt for real-time circuit breaker actuation. It is for the recovery analysis phase after a trip has occurred and the system is stable. Avoid using it when the downstream service is still actively failing, when you lack error metrics or configuration data, or when an automated rollback is already in progress. In those cases, the prompt will produce a plan based on stale or incomplete information, which can lead to an unsafe recovery recommendation.

This prompt template is designed for distributed systems engineers who need a structured, safe recovery plan when a circuit breaker enters an open state. It consumes a trip event, downstream error rates, service configuration, and deployment topology to produce a step-by-step recovery or configuration change recommendation. The prompt enforces a gradual ramp-up strategy, explicit rollback criteria, and a prohibition on immediate full-restoration to prevent cascading failures.

textCopy
You are an SRE assistant specialized in distributed system resilience. Your task is to produce a safe recovery plan for a circuit breaker that has transitioned to an OPEN state.

## INPUT
- Trip Event Details: [TRIP_EVENT]
- Downstream Error Rates (last 5 minutes): [DOWNSTREAM_ERROR_RATES]
- Service Configuration (timeout, threshold, half-open max requests): [SERVICE_CONFIG]
- Deployment Topology (callers, callees, fallback paths): [TOPOLOGY]
- Recent Deployments or Config Changes: [RECENT_CHANGES]
- Current Circuit Breaker State: OPEN

## CONSTRAINTS
- Do NOT recommend immediate full restoration of traffic.
- Every recovery step must include a success criterion and a rollback trigger.
- The plan must start with a traffic ramp of no more than 10% of normal load.
- If downstream error rates exceed [ERROR_THRESHOLD]% during any ramp step, recommend immediate halt and escalation.
- If the root cause is unknown, recommend diagnostic steps before any traffic restoration.
- Prefer configuration changes over code deployments unless a bug is confirmed.

## OUTPUT_SCHEMA
Return a JSON object with the following structure:
{
  "diagnosis": {
    "likely_root_cause": "string",
    "confidence": "low|medium|high",
    "evidence": ["string"]
  },
  "recovery_plan": [
    {
      "step_number": integer,
      "action": "string",
      "traffic_percentage": integer or null,
      "success_criterion": "string",
      "rollback_trigger": "string",
      "expected_duration_seconds": integer
    }
  ],
  "escalation_criteria": ["string"],
  "recommended_configuration_changes": [
    {
      "parameter": "string",
      "current_value": "string",
      "proposed_value": "string",
      "rationale": "string"
    }
  ],
  "do_not_do": ["string"]
}

## RISK_LEVEL
[RISK_LEVEL]

If RISK_LEVEL is "high", add a human-approval gate before any automated action. If RISK_LEVEL is "critical", produce only a diagnostic summary and escalate immediately.

To adapt this template, replace each square-bracket placeholder with live data from your monitoring stack, deployment tracker, and incident context. The [TRIP_EVENT] should include the exact timestamp, the circuit name, and the failure count that triggered the open state. The [DOWNSTREAM_ERROR_RATES] must be a recent time-series snapshot—stale data leads to unsafe ramp decisions. The [ERROR_THRESHOLD] should be set conservatively; a common starting point is 5% for critical paths and 15% for non-critical ones. Before wiring this into an automated harness, test the prompt against at least three historical incidents where the correct recovery action is known, and confirm the model does not recommend a full traffic restore as step one.

This prompt is designed to be called by an automation harness—not a human chat interface—when a circuit breaker transitions to the open state. The harness should trigger on a circuit breaker event from your observability stack (e.g., a webhook from Prometheus Alertmanager, a PagerDuty incident, or a custom operator watching circuit breaker metrics). The prompt receives structured context about the trip event, downstream error rates, and the current service configuration, then returns a recovery plan that the harness can either execute automatically (for low-risk changes like adjusting thresholds) or queue for human approval (for high-risk changes like modifying the circuit breaker's half-open max request count).

Wiring the prompt into your workflow: The harness should construct the prompt's [CIRCUIT_BREAKER_EVENT], [DOWNSTREAM_METRICS], and [SERVICE_CONFIG] inputs from your monitoring and configuration systems. After calling the model, validate the output against a strict schema: the recovery plan must include a ramp_up_strategy field with explicit step counts and cooldown durations, a configuration_changes array with old and new values, and a risk_assessment block. Reject any output that proposes removing the circuit breaker entirely or setting the failure threshold to an unreasonably high value. For high-risk environments, route the plan to a human reviewer via a Slack message or ticket before applying any configuration changes. Log every prompt invocation, the raw model output, the validation result, and the final action taken for post-incident review.

Retry and fallback logic: If the model returns an unparseable response or fails schema validation, retry once with a simplified prompt that includes the validation error message. If the second attempt also fails, escalate to the on-call engineer with the original circuit breaker event context and a note that automated recovery planning failed. Never retry more than twice for the same event—circuit breaker recovery is time-sensitive, and waiting too long defeats the purpose. For model selection, prefer a model with strong structured output capabilities and low latency (e.g., Claude 3.5 Sonnet or GPT-4o with JSON mode). Avoid using smaller, less reliable models for this workflow because an incorrect recovery plan can cause cascading failures.

Testing the harness before production: Build a test suite that replays historical circuit breaker trip events through the harness and checks that the generated recovery plans are safe and actionable. Include adversarial test cases: a trip event with 100% downstream error rates should produce a plan that keeps the circuit open and escalates for manual investigation, not a plan that attempts a gradual ramp-up. A trip event caused by a single slow request during a deployment should produce a plan that allows a quick half-open probe. Run these tests as part of your CI pipeline whenever the prompt template or validation logic changes. Monitor the harness in production by tracking the time from circuit breaker trip to recovery plan delivery, the human approval rate, and whether applied plans actually resolved the downstream errors without causing new trips.