Failover Behavior Verification Prompt

This prompt is for reliability engineers and SREs who need to verify that a failover event—typically triggered after a configuration change, deployment, or manual cutover—behaved exactly as expected. The job-to-be-done is not just confirming that the secondary system is now active, but producing a structured, evidence-backed report that proves the failover was clean. The ideal user is an engineer who understands the system's architecture, has access to pre- and post-failover metrics, logs, and client-side telemetry, and needs a consistent, auditable artifact to attach to an incident record or change request. The required context includes the expected failover target, the trigger event timestamp, a baseline of normal operating behavior, and the specific data consistency and client impact dimensions that matter for this service.

Do not use this prompt when you only need a simple binary 'is it up?' health check. A standard uptime monitor or a basic curl test is more appropriate for that. This prompt is also the wrong tool when the failover is still in progress or when the system is in a degraded, mid-recovery state; the verification logic assumes a stable post-failover state to compare against the baseline. Avoid it when you lack access to both pre- and post-event data, as the core value is the structured comparison. Finally, this prompt is not a substitute for automated end-to-end failover testing in a staging environment; it is a production verification tool for an event that has already occurred.

The output is a formal failover test report, not a conversational summary. It must include precise timing deltas, data consistency checks (e.g., record counts, checksums, or last-written record verification), and client impact observations (e.g., error rate spikes, latency changes, or connection resets). The prompt is designed to be wired into a post-action verification harness where its structured output can be programmatically validated. Before relying on this report, you should confirm that the model had access to the correct time-bounded data windows and that the report's pass/fail conclusion is cross-referenced with your own monitoring dashboards. The next step after generating this report is to attach it to the change record and, if the verification fails, immediately trigger the rollback decision prompt.

This prompt template is designed for reliability engineers who need to verify that a failover event—triggered manually or automatically after a configuration change—completed successfully and without silent failures. It instructs the model to act as a verification analyst, consuming pre-collected evidence such as logs, metrics, and client-side observations, and producing a structured report that can be used as an audit artifact or an automated gate before marking an incident as resolved. The template uses square-bracket placeholders for all variable inputs, making it straightforward to integrate into a CI/CD pipeline, a runbook automation system, or a manual review workflow.

textCopy
You are a failover verification analyst for a production infrastructure team. Your task is to review the provided evidence and produce a structured failover test report. You must not speculate beyond the provided data. If evidence is missing or inconclusive, flag it explicitly.

## INPUTS
- [FAILOVER_EVENT_DESCRIPTION]: A natural language description of the expected failover event, including the triggering condition, the source and target systems, and the expected behavior.
- [TIMING_EVIDENCE]: Log excerpts, timestamps, or monitoring data showing the start time, cutover duration, and completion time of the failover.
- [DATA_CONSISTENCY_EVIDENCE]: Checksums, row counts, replication lag metrics, or application-level integrity check results from before and after the failover.
- [CLIENT_IMPACT_EVIDENCE]: Client-side error logs, latency measurements, connection reset counts, or user-reported issues during the failover window.
- [CONFIGURATION_CHANGE_RECORD]: The specific configuration change that preceded or triggered the failover, including the change ID, who approved it, and the expected effect.

## OUTPUT_SCHEMA
Return a single JSON object with the following structure:
{
  "verification_id": "string, a unique identifier for this verification run",
  "failover_event": "string, summary of the failover event from the evidence",
  "verdict": "PASS | FAIL | INCONCLUSIVE",
  "timing_assessment": {
    "observed_cutover_duration_ms": "number or null",
    "expected_cutover_duration_ms": "number or null",
    "within_sla": "boolean",
    "anomalies": ["string, description of any timing anomalies"]
  },
  "data_consistency_assessment": {
    "checks_passed": ["string, name of each check that passed"],
    "checks_failed": ["string, name of each check that failed with details"],
    "overall_consistent": "boolean",
    "anomalies": ["string, description of any data anomalies"]
  },
  "client_impact_assessment": {
    "errors_observed": "boolean",
    "error_details": ["string, summary of each error type and count"],
    "latency_spike_observed": "boolean",
    "connection_resets": "number or null",
    "user_reports": ["string, summary of user-reported issues"]
  },
  "silent_failure_flags": ["string, any indicators of failures that did not trigger alerts"],
  "recommendations": ["string, actionable next steps based on findings"],
  "evidence_gaps": ["string, any missing evidence that prevented a conclusive assessment"]
}

## CONSTRAINTS
- Do not invent evidence. If a field cannot be populated from the provided inputs, use null or an empty array.
- If the verdict is INCONCLUSIVE, you must populate the `evidence_gaps` field with specific missing data.
- Flag any discrepancy between the [CONFIGURATION_CHANGE_RECORD] and the observed behavior in the `silent_failure_flags` field.
- Use precise timestamps and metric values from the evidence; do not round or approximate unless the evidence is ambiguous.
- If client impact is observed, classify its severity as "none", "minor", "major", or "critical" based on the error rate and user reports.

To adapt this template, replace each square-bracket placeholder with data from your observability stack, configuration management system, and incident management tool. The [TIMING_EVIDENCE] and [DATA_CONSISTENCY_EVIDENCE] fields should be populated with raw log lines, metric queries, or pre-computed summaries—avoid editorializing before passing them to the model. If your environment uses a specific SLA for failover duration, hardcode that value into the expected_cutover_duration_ms field in the output schema or pass it as an additional constraint. Before deploying this prompt in an automated pipeline, validate that the JSON output conforms to the schema using a strict JSON Schema validator, and implement a retry with a repair prompt if validation fails. For high-risk production systems, route any verdict of FAIL or INCONCLUSIVE to a human reviewer before closing the incident, and log the full prompt and response as an audit record.

The failover verification prompt is designed to be invoked automatically after a configuration change triggers a failover event, not as a standalone chat interaction. Wire it into your CI/CD or runbook automation pipeline so that the prompt receives structured telemetry, timing logs, and client-side observation data as input. The model's job is to produce a machine-readable report, not a narrative summary for a human to read first. The output should be parsed by your orchestration layer to decide whether to auto-close the change ticket, page the on-call, or trigger a rollback. Treat this prompt as a decision-support component inside a larger reliability workflow, not as the final authority.

Build the harness around a strict input schema. Before calling the model, assemble a JSON payload containing: the failover start and end timestamps, a list of expected vs. actual service endpoints, client-side error rate and latency percentiles from your observability platform, any data consistency check results (e.g., checksum comparisons, row counts), and a [CHANGE_CONTEXT] string describing the configuration change that initiated the failover. Validate this payload against a schema before sending it to the model—reject the run if required fields like actual_failover_duration_ms or client_impact.error_budget_consumed are missing. After receiving the model's output, validate the JSON structure against your expected [OUTPUT_SCHEMA] (fields like failover_successful, data_consistency_passed, client_impact_severity, and recommendations). If parsing fails, retry once with a stricter prompt that includes the parse error message. If the retry also fails, escalate to a human with the raw output and the validation error attached.

Log every invocation, including the full prompt, the model's raw response, the parsed output, and the final action taken (auto-close, warn, escalate). This audit trail is critical for post-incident review and for tuning the prompt's thresholds over time. Pay special attention to silent failure modes: a failover that appears successful at the infrastructure layer but introduces data inconsistency or elevated client-side errors. Your eval harness should include golden test cases that simulate exactly these scenarios—e.g., a failover where all health checks pass but a downstream replica is lagging by 30 seconds. Run these evals on a schedule and after any prompt change. If the model's failover_successful field is true but the eval expects false, block the prompt update and investigate the regression before it reaches production.