Metric Spike Root Cause Hypothesis Prompt

This playbook is for SREs and on-call engineers who need to move from observing an anomalous metric spike to generating a testable root cause hypothesis. The prompt takes structured metric data, infrastructure topology, and a known failure mode catalog and produces ranked hypotheses with supporting evidence and specific validation steps. Use this when you have a clear spike in latency, error rate, or throughput and need to narrow the search space before diving into raw logs.

The ideal user has access to a monitoring dashboard or API that can provide a time-series slice of the anomalous metric, a service dependency map, and a list of recent deployment or configuration changes. The prompt requires these inputs to be pre-assembled into the [METRIC_DATA], [TOPOLOGY], and [CHANGE_LOG] placeholders. Without this structured context, the model will generate plausible but ungrounded speculation. The prompt also expects a [FAILURE_MODE_CATALOG]—a curated list of known failure patterns for your system—to anchor hypotheses in observed reality rather than generic troubleshooting advice.

Do not use this prompt for general debugging without metric data, or when the incident is purely a user-reported functional bug with no observable telemetry anomaly. It is also unsuitable for security incident response where the primary signal is an alert from an intrusion detection system rather than a performance metric. If you lack a clear spike in a specific metric, start with the Log Anomaly Pattern Extraction Prompt or the Incident Timeline Extraction Prompt instead. After generating hypotheses, always validate them against raw logs, traces, or direct system inspection before declaring a root cause.

This template is designed to be copied directly into your incident response tooling or AI harness. It accepts structured observability data—metric names, spike characteristics, infrastructure topology, and recent changes—and produces a ranked list of root cause hypotheses. Each hypothesis includes supporting evidence from the provided data, testability criteria, and a confidence estimate. The template uses square-bracket placeholders that you must replace with real data from your monitoring stack before sending to the model.

codeCopy
You are an SRE root cause analyst. Your task is to generate testable root cause hypotheses for an anomalous metric spike.

## INPUT DATA

### Metric Spike Details
- Metric name: [METRIC_NAME]
- Observed value: [OBSERVED_VALUE]
- Baseline value (7-day average at this time): [BASELINE_VALUE]
- Spike magnitude (multiples of baseline): [SPIKE_MAGNITUDE]
- Spike start time (UTC): [SPIKE_START_TIME]
- Spike duration so far: [SPIKE_DURATION]
- Direction: [SPIKE_DIRECTION] (options: upward, downward)
- Rate of change: [RATE_OF_CHANGE] (options: sudden, gradual, oscillating)

### Correlated Metrics
For each correlated metric, provide:
- Metric name: [CORRELATED_METRIC_NAME]
- Behavior during spike: [CORRELATED_BEHAVIOR] (options: also spiking, dropping, flat, inverse correlation)
- Lag relative to primary spike: [CORRELATED_LAG] (options: leading by X minutes, lagging by X minutes, simultaneous)

### Infrastructure Context
- Affected services: [AFFECTED_SERVICES]
- Affected hosts/pods/instances: [AFFECTED_INSTANCES]
- Affected availability zones or regions: [AFFECTED_REGIONS]
- Service dependency graph (upstream and downstream): [DEPENDENCY_GRAPH]
- Relevant load balancer or traffic routing: [TRAFFIC_ROUTING]

### Recent Changes (last 24 hours)
- Deployments: [RECENT_DEPLOYMENTS]
- Configuration changes: [RECENT_CONFIG_CHANGES]
- Feature flag toggles: [RECENT_FEATURE_FLAGS]
- Infrastructure scaling events: [RECENT_SCALING_EVENTS]
- Third-party dependency status: [THIRD_PARTY_STATUS]

### Known Failure Modes
- Documented failure modes for affected services: [KNOWN_FAILURE_MODES]
- Previous similar incidents: [PREVIOUS_INCIDENTS]

## OUTPUT REQUIREMENTS

Generate exactly [HYPOTHESIS_COUNT] root cause hypotheses, ranked by likelihood. For each hypothesis, produce:

1. **Hypothesis statement**: A specific, falsifiable statement describing the suspected root cause.
2. **Supporting evidence**: Which data points from the input support this hypothesis.
3. **Contradicting evidence**: Which data points, if any, weaken this hypothesis.
4. **Confidence score**: Low, Medium, or High with a brief justification.
5. **Test to confirm**: A specific, executable check that would confirm or rule out this hypothesis.
6. **Test to rule out**: A specific check that would falsify this hypothesis.
7. **Expected time to test**: Estimated minutes to execute the confirmation test.
8. **Immediate mitigation**: If the hypothesis is correct, what action should be taken now.

## CONSTRAINTS
- Do not fabricate evidence. Only use data provided in the input.
- If the input data is insufficient to generate strong hypotheses, state what additional data would be needed.
- Distinguish between correlation and causation explicitly.
- Flag any hypothesis that requires human judgment to confirm.
- If [RISK_LEVEL] is "high", include a warning about the blast radius of each proposed mitigation.

Adaptation guidance: Replace each bracketed placeholder with live data from your observability stack—Prometheus, Datadog, Grafana, or your internal monitoring system. The [DEPENDENCY_GRAPH] field should include both upstream callers and downstream dependencies so the model can reason about propagation direction. The [KNOWN_FAILURE_MODES] field is critical for hypothesis quality; populate it from your incident database, runbooks, or service READMEs. If you lack structured data for a field, replace it with "Not available" rather than omitting it, so the model can explicitly note data gaps. For production use, wrap this prompt in a harness that validates the output contains exactly the requested number of hypotheses, each with all eight required fields, before surfacing results to an on-call engineer.

This prompt is designed to be called during an active incident or post-incident review when an anomalous metric spike has been identified. The implementation harness should treat the prompt as a hypothesis generation engine, not a final answer. The output is a set of testable hypotheses that an on-call engineer must validate against live systems, logs, and deployment records before taking action. Wire this prompt into your incident management bot, Slack command, or internal SRE dashboard so that responders can invoke it with a metric name, spike window, and relevant topology context.

The application layer must enforce strict input validation before calling the model. Require the [METRIC_NAME], [SPIKE_TIMESTAMP], [SPIKE_MAGNITUDE], and [INFRASTRUCTURE_TOPOLOGY] fields to be non-empty. If the topology is large, pre-process it to include only services upstream and downstream of the affected component. For model choice, use a model with strong reasoning capabilities (such as GPT-4o or Claude 3.5 Sonnet) because the task requires correlating metric behavior with failure mode knowledge. Set temperature to 0.2–0.3 to balance structured output with some hypothesis diversity. Implement a retry wrapper that catches malformed JSON outputs, extracts the raw text, and re-prompts the model with a repair instruction: 'The previous output was not valid JSON. Return only the corrected JSON object matching the schema.' After three failed retries, log the failure and escalate to a human responder with the raw model output attached.

Post-processing must validate that each hypothesis in the returned array contains a non-empty hypothesis, supporting_evidence, and test_steps field. Strip any hypotheses where the confidence_score is outside the 0.0–1.0 range or where test_steps is fewer than two concrete actions. Log every invocation with the input parameters, model version, raw response, and validated output for post-incident auditability. Never automatically execute remediation based on these hypotheses; always require a human to review the output, run the test steps, and confirm or reject each hypothesis before any mitigation is applied. The harness should surface the hypotheses in your incident channel with a clear 'Needs Human Validation' label and a button to mark each hypothesis as confirmed, rejected, or inconclusive.