API Error Rate Spike Investigation Prompt

Use this prompt when you are an API platform engineer or on-call SRE responding to an active HTTP 5xx storm and you need to isolate the fault domain before taking remediation action. The prompt is designed for situations where error rates have crossed a defined threshold, multiple endpoints or services may be affected, and you have access to structured observability data—error code distributions, per-endpoint latency and error rate time series, recent deployment records, dependency health checks, and traffic shift metrics. The primary job-to-be-done is correlating error signals across these dimensions to produce a ranked list of probable fault domains (e.g., a specific service, database, upstream dependency, or deployment artifact) and a mitigation priority list, not a full root cause analysis or remediation runbook.

Do not use this prompt for isolated single-endpoint failures with an obvious stack trace, for client-side 4xx error investigations, or when you lack deployment and dependency data. The prompt assumes you can provide structured context: a time window, affected endpoints with error counts and codes, a deployment timeline, dependency health status, and traffic pattern summaries. If you only have raw logs or a single alert, use the Stack Trace Root Cause Analysis Prompt or Log Pattern Anomaly Detection Prompt first to assemble structured inputs. This prompt is also not a substitute for automated incident severity classification—pair it with the Incident Severity Classification Prompt if you need standardized severity assignment alongside fault-domain isolation.

The prompt produces a fault-domain isolation report with explicit evidence-to-hypothesis links, confidence indicators, and a mitigation priority list ordered by impact and reversibility. Before relying on the output, validate that the model has correctly correlated timing between deployments and error onset, has not conflated correlation with causation, and has grounded each hypothesis in specific observability signals you provided. For production incident response, always require human review of the fault-domain ranking before executing any mitigation action such as rollback, circuit breaker adjustment, or traffic shifting. Wire the output into your incident channel as a structured investigation artifact, not as an automated remediation trigger.

This template is designed to be wired into an automated investigation harness or used manually by an on-call engineer. It expects structured inputs from your observability stack, deployment history, and source code repository. The placeholders in square brackets must be replaced with real data before execution. Do not run this prompt with unresolved placeholders—the model will hallucinate or produce generic advice that wastes time during an incident.

textCopy
You are an SRE investigator analyzing an API error rate spike.
Your goal is to produce a fault-domain isolation report and a mitigation priority list.

## INPUT DATA

### Error Telemetry
[ERROR_TELEMETRY_JSON]
<!-- Expected: JSON with fields: time_window_start, time_window_end, total_requests, error_count, error_rate, top_error_codes (with counts), top_failing_endpoints (with counts), latency_p99_by_endpoint, error_budget_remaining_pct -->

### Deployment History
[DEPLOYMENT_HISTORY_JSON]
<!-- Expected: JSON array of recent deployments with fields: deploy_id, service, version, deployed_at, rollback_status, diff_summary, changed_files -->

### Dependency Health
[DEPENDENCY_HEALTH_JSON]
<!-- Expected: JSON with fields: downstream_service_status, database_status, cache_status, queue_status, external_api_status, circuit_breaker_state -->

### Traffic Profile
[TRAFFIC_PROFILE_JSON]
<!-- Expected: JSON with fields: requests_per_second_over_time, top_callers, region_distribution, auth_success_rate, rate_limiter_activation_count -->

### Relevant Source Context
[SOURCE_CONTEXT]
<!-- Expected: Relevant code snippets from the repository for the top-failing endpoints, recent deployment diffs, and configuration files. Include file paths and line numbers. -->

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "incident_summary": "string: one-sentence description of the spike",
  "fault_domain_isolation": {
    "primary_domain": "DEPLOYMENT | DEPENDENCY | TRAFFIC | CONFIGURATION | UNKNOWN",
    "confidence": "HIGH | MEDIUM | LOW",
    "evidence": ["string: specific evidence item with source reference"],
    "alternative_domains": ["string: other possible causes ranked by likelihood"]
  },
  "correlation_findings": [
    {
      "signal_a": "string: e.g., 'deployment v2.4.1 at 14:32'",
      "signal_b": "string: e.g., '5xx spike starting at 14:33 on /api/checkout'",
      "correlation_strength": "STRONG | MODERATE | WEAK",
      "causal_hypothesis": "string: brief explanation of possible causal link"
    }
  ],
  "mitigation_priority_list": [
    {
      "rank": 1,
      "action": "string: specific mitigation step",
      "expected_impact": "HIGH | MEDIUM | LOW",
      "risk": "string: what could go wrong",
      "rollback_plan": "string: how to undo this action",
      "evidence_basis": "string: which finding supports this action"
    }
  ],
  "unanswered_questions": ["string: what you still need to know to complete diagnosis"],
  "recommended_escalation": "NONE | ENGINEERING_LEAD | DEPENDENCY_TEAM | INFRASTRUCTURE | EXECUTIVE"
}

## CONSTRAINTS
- Ground every finding in the provided telemetry, deployment history, or source code. Do not speculate without citing evidence.
- If the data is insufficient to isolate a fault domain, set primary_domain to UNKNOWN and list what additional data is needed in unanswered_questions.
- Prioritize mitigation actions that are reversible. The top-ranked action should be the safest high-impact step.
- Do not recommend a full rollback unless deployment correlation is STRONG and the deployment touched the failing endpoints.
- If error budget remaining is below 10%, note this in the incident_summary and escalate urgency in mitigation_priority_list.
- If circuit breakers are open for any dependency, that dependency must appear in fault_domain_isolation.

## RISK LEVEL
[RISK_LEVEL]
<!-- Set to HIGH, MEDIUM, or LOW based on error budget burn rate and user impact. HIGH triggers mandatory human review of all findings before action. -->

To adapt this template for your environment, replace the telemetry placeholders with data from your actual observability tools (Datadog, Prometheus, Honeycomb, etc.). The [SOURCE_CONTEXT] placeholder should be populated by a retrieval step that pulls relevant files from your repository based on the top-failing endpoints and recent deployment diffs. If you are building an automated harness, pre-process the raw observability data into the expected JSON shapes before inserting them into the prompt. The [RISK_LEVEL] field should be computed from your SLO burn rate, not guessed—wire it to your error budget monitoring. After the model returns the JSON output, validate it against the schema before surfacing it to an on-call engineer. Any finding with confidence LOW or correlation_strength WEAK should be flagged for human review before action is taken.

This prompt is designed to be the reasoning core of an automated incident response system, not a one-off chat interaction. The implementation harness must feed the model structured, time-bounded data from your observability stack and enforce a strict output contract that downstream tools can parse. The primary integration points are your metrics backend (for error rates, latency, and traffic data), your CI/CD pipeline (for recent deployment events), and your service dependency map. The harness should assemble these into the [CONTEXT] block before invoking the model, ensuring the prompt receives a complete, consistent snapshot of the incident window rather than relying on the model to fetch data itself.

The harness should enforce a two-phase execution pattern. In Phase 1, the model receives the full prompt and produces the fault-domain isolation report. A JSON schema validator must check the output for required fields (fault_domain_candidates, mitigation_priority, evidence_correlation) before the report is surfaced to an on-call engineer. If validation fails, the harness should retry once with the validation errors appended to the [CONSTRAINTS] block. In Phase 2, if the model identifies a specific deployment or configuration change as the primary suspect, the harness can optionally invoke a separate tool (such as a rollback dry-run or a diff generator) and feed the result back for a refined recommendation. Never allow autonomous rollback execution from this prompt's output alone; always require human approval for any infrastructure mutation.

For model selection, prefer a model with strong structured output capabilities and a context window large enough to hold at least 50 data points per endpoint plus deployment metadata. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are suitable defaults. Set temperature to 0 or a very low value (0.1) to maximize deterministic, repeatable analysis. Log every invocation—including the full prompt, raw response, validation result, and the on-call engineer's eventual action—to an audit table. This traceability is critical for postmortem reviews and for tuning the prompt's thresholds (e.g., what constitutes a 'spike' in your environment). The next step is to define your eval harness using historical incidents where the root cause is known, so you can measure whether the prompt's fault_domain_candidates ranking places the true cause in the top position.