HTTP 5xx Error Burst Root Cause Hypothesis Prompt

This prompt is designed for Site Reliability Engineers (SREs) and on-call developers who are responding to an active or recently resolved burst of HTTP 5xx server errors. The core job-to-be-done is to rapidly reduce a high volume of raw error logs, deployment events, and dependency status checks into a ranked list of testable root cause hypotheses. The prompt assumes you have already gathered a representative sample of error logs, a timeline of recent production changes, and the current status of critical upstream dependencies. It is not a replacement for your observability platform's anomaly detection; it is a reasoning accelerator that structures your investigation when the alert fires and you need to form a coherent starting point under time pressure.

The ideal user is a responder who can provide structured, factual context. The prompt requires three concrete inputs: a set of representative 5xx log lines with timestamps, a list of recent deployments or configuration changes with their timestamps, and a status summary of upstream dependencies (e.g., databases, internal services, third-party APIs). Without this data, the model will be forced to speculate, which defeats the purpose. The prompt is most effective when the error burst is correlated with a known change or dependency failure. It is less useful for long-running, low-grade error rates that lack a clear temporal boundary, or for errors where the root cause is already known and documented in a runbook.

Do not use this prompt when the incident is still in the detection phase and you lack a coherent log sample. Do not use it as a substitute for a postmortem process; it generates hypotheses, not a final root cause determination. The output must be treated as a starting point for human-led investigation, not as a diagnosis to be acted upon without verification. In regulated or high-risk environments, any remediation based on a hypothesis must be reviewed by a human operator before execution. The next step after receiving the output is to test the top-ranked hypothesis against your observability data, not to blindly apply a suggested fix.

This prompt is the core engine for turning a chaotic burst of 5xx logs into a structured, testable set of hypotheses. It is designed to be dropped into an SRE investigation workflow, where it forces the model to correlate errors with recent changes, upstream dependencies, and resource metrics rather than just summarizing log lines. The template uses square-bracket placeholders for all dynamic inputs, ensuring you can wire it directly into your incident response tooling or a manual runbook.

codeCopy
You are an SRE investigator analyzing a burst of HTTP 5xx errors.

Your task is to generate a ranked list of root cause hypotheses, each supported by specific evidence from the provided data. Do not just summarize the logs. Correlate the error patterns with recent deployments, upstream dependency status, and resource metrics to find the most likely trigger.

## INPUT DATA

### 5xx Error Log Sample
[Aggregated or sampled 5xx log lines with timestamps, status codes, error messages, and affected endpoints]

### Recent Deployment Timeline
[List of recent deployments, config changes, or feature flags toggled in the [TIME_WINDOW] before the error burst, with timestamps]

### Upstream Dependency Status
[Health status, error rates, and latency for all critical upstream services and databases during the incident window]

### Resource Metrics
[CPU, memory, I/O, connection pool, and thread pool metrics for the affected service instances]

## OUTPUT SCHEMA

Produce a JSON object with the following structure:
{
  "incident_summary": "A one-sentence summary of the error burst (e.g., 'Sudden spike of 502 errors on /api/checkout starting 14:03 UTC, correlating with a payment gateway timeout').",
  "hypotheses": [
    {
      "rank": 1,
      "hypothesis": "A clear, concise statement of the root cause hypothesis.",
      "evidence": ["List of specific log entries, metric spikes, or deployment events that support this hypothesis."],
      "confidence": "high|medium|low",
      "suggested_verification": "A concrete step to prove or disprove this hypothesis (e.g., 'Check payment gateway latency dashboard for 14:00-14:05 UTC')."
    }
  ],
  "abstention_reason": null
}

## CONSTRAINTS

- Rank hypotheses by likelihood. The top hypothesis should be the single most probable cause.
- If the provided data is insufficient to form any hypothesis with at least 'low' confidence, set the 'hypotheses' array to empty and provide a clear 'abstention_reason' explaining what critical data is missing (e.g., 'No upstream dependency status provided, cannot rule out external service failure').
- Do not invent data, timelines, or events not present in the input.
- Focus on the trigger of the error burst, not generic system health advice.

To adapt this template, replace each bracketed placeholder with data from your observability stack. The [TIME_WINDOW] should be a narrow slice (e.g., 5-10 minutes) around the onset of the error burst. The power of this prompt lies in the abstention_reason field; it prevents the model from hallucinating a cause when data is missing, which is critical for high-stakes incident response. Before putting this into production, run it against a set of historical incidents where you know the true root cause to calibrate the confidence levels and ensure the evidence cited is factually grounded in your input data.

This prompt is designed to be called during an active incident, not as a post-mortem tool. Wire it into your incident response bot or runbook automation so that an SRE can trigger it with a slash command (e.g., /hypothesize window=15m service=api-gateway). The harness must gather the required inputs—recent deployment events, upstream dependency status, and resource metrics—before assembling the prompt. Do not call the model with incomplete data; if a data source is unavailable, the harness should inject an explicit [DATA_GAP] marker into the relevant section of the prompt so the model knows to abstain or reduce confidence rather than hallucinate a cause.

Input assembly and tool integration: The harness should query your observability stack (e.g., Datadog, Grafana, CloudWatch) for the time window around the 5xx burst. Pull deployment events from your CI/CD pipeline (e.g., GitHub Actions, ArgoCD) and upstream health from your dependency monitors. Structure these as the [DEPLOYMENT_TIMELINE], [UPSTREAM_STATUS], and [RESOURCE_METRICS] placeholders. For the [LOG_SAMPLES] placeholder, sample representative 5xx error logs—include the full error message, status code, and service name, but strip PII or sensitive headers before injection. The harness should also attach a [CONSTRAINTS] block that includes your incident severity level and any known safe/unsafe actions (e.g., 'do not suggest rolling back the payment service without approval').

Validation and retry logic: The model output must conform to a strict JSON schema with a hypotheses array, each containing rank, cause, evidence, confidence, and suggested_action. Validate the output before surfacing it to the incident channel. If the JSON is malformed, retry once with a repair prompt that includes the validation error. If the model abstains (returns an empty hypotheses array with a data_gaps explanation), surface that abstention directly—do not retry. For each hypothesis, check that evidence fields reference actual data from the input (e.g., a specific deployment timestamp or metric spike). Flag any hypothesis where evidence appears fabricated for human review.

Human review and escalation: The harness should post the ranked hypotheses to the incident channel (e.g., Slack, PagerDuty) with a clear disclaimer that these are AI-generated leads, not confirmed root causes. Require a human SRE to acknowledge or dismiss each hypothesis before any automated remediation is triggered. For high-severity incidents (SEV1/SEV0), the harness should escalate to the incident commander immediately and not wait for model output if the prompt call exceeds a 30-second timeout. Log every prompt invocation, the raw model response, the validated output, and the human disposition (accepted/rejected) for post-incident review and prompt improvement. Never allow the model's suggested_action to execute automatically—always route through a human approval step or a runbook with a manual gate.