Output Quality Incident Response Prompt Template

This prompt is for incident response teams, SREs, and ML engineers who need to triage a production AI quality failure quickly. Use it when model outputs have degraded below acceptable thresholds, when downstream systems are receiving malformed or harmful responses, or when an automated quality gate has triggered an alert. The prompt produces a structured incident assessment that includes root cause hypotheses, affected output scope, severity classification, and immediate mitigation steps. It assumes you already have observability data, sample outputs, and quality metrics available.

Do not use this prompt for routine output review or pre-release testing; it is designed for active incident response where time to mitigation matters. The prompt requires concrete evidence as input—such as eval scores, error logs, and flagged examples—and will not perform well if asked to speculate without data. Before invoking this prompt, ensure you have gathered at least three representative failure samples, the output of your quality gate or eval pipeline, and any relevant deployment or model version change logs. The prompt is most effective when paired with a human-in-the-loop approval step before any automated mitigation action is executed.

After running the prompt, validate the severity classification against your internal incident taxonomy. If the model proposes a root cause that involves a recent prompt or model change, cross-reference it against your CI/CD and model registry logs. The output of this prompt should feed directly into your incident management tooling, not replace it. Avoid using this prompt for low-urgency quality fluctuations or for outputs that are still in staging; it is calibrated for production impact where every minute of degraded quality affects users or downstream systems.

This section provides a copy-ready prompt template you can drop directly into your incident response runbook or automated remediation pipeline. The template is designed to receive raw incident data—such as error logs, affected output samples, and detection timestamps—and produce a structured incident assessment with root cause hypotheses, severity classification, and immediate mitigation steps. Every square-bracket placeholder must be replaced with incident-specific data before the prompt is sent to the model. Do not leave placeholders unresolved in production; missing context will degrade the quality of the assessment and may produce hallucinated incident details.

textCopy
You are an incident response analyst for a production AI system. Your task is to assess an output quality incident and produce a structured incident assessment.

## INCIDENT DATA
[INCIDENT_REPORT]

## AFFECTED OUTPUT SAMPLES
[OUTPUT_SAMPLES]

## DETECTION CONTEXT
- Detection time: [DETECTION_TIMESTAMP]
- Detected by: [DETECTION_METHOD]
- Affected pipeline stage: [PIPELINE_STAGE]
- Output volume affected (estimated): [AFFECTED_VOLUME]

## OUTPUT SCHEMA
Return a valid JSON object with exactly these fields:
{
  "incident_id": "string",
  "severity": "critical|high|medium|low",
  "root_cause_hypotheses": [
    {
      "hypothesis": "string (description of one possible root cause)",
      "likelihood": "high|medium|low",
      "supporting_evidence": "string (specific evidence from the incident data)"
    }
  ],
  "affected_output_scope": {
    "output_types": ["string"],
    "estimated_impact_percentage": "number (0-100)",
    "downstream_systems_affected": ["string"]
  },
  "immediate_mitigation_steps": [
    {
      "step": "string (actionable mitigation step)",
      "priority": "critical|high|medium|low",
      "owner": "string (suggested team or role)"
    }
  ],
  "recommended_investigation_actions": ["string"],
  "confidence_assessment": {
    "overall_confidence": "high|medium|low",
    "uncertainty_sources": ["string"]
  }
}

## CONSTRAINTS
- Do not fabricate incident details not present in the input data.
- If evidence is insufficient for a high-confidence root cause, state that explicitly and lower the likelihood rating.
- Severity must reflect actual downstream impact, not just output volume.
- Mitigation steps must be actionable within the current incident response window.
- If the incident data contains PII or credentials, do not reproduce them in the assessment.

Before integrating this prompt into your incident response pipeline, verify that your incident data feed populates every placeholder completely. The [INCIDENT_REPORT] placeholder should contain the raw incident description, error messages, and any initial observations from the on-call responder. The [OUTPUT_SAMPLES] placeholder must include actual model outputs that triggered the quality gate—include both passing and failing examples if available to help the model calibrate severity. If your incident management system already assigns an incident ID, pass it through the harness rather than relying on the model to generate one. After receiving the model's assessment, always validate the JSON structure against the declared schema before routing the assessment to your incident commander or posting it to your incident channel. A failed parse at this stage delays response time when it matters most.

This prompt is designed to be the first automated responder in an incident management pipeline, not a standalone chatbot. It should be triggered by a monitoring alert that detects a statistically significant deviation in output quality metrics—such as a drop in your LLM-as-judge scores, a spike in user-reported issues, or a schema validation failure rate exceeding a defined threshold. The prompt's primary job is to structure the initial chaos into a consistent incident document that on-call engineers can immediately act on, reducing mean-time-to-acknowledge (MTTA).

To wire this into an application, wrap the prompt in a function that is called by your monitoring webhook. The function must assemble the [INCIDENT_TRIGGER_DETAILS] and [RECENT_OUTPUT_SAMPLES] from your logging database or observability platform. Critical harness checks include: (1) Input validation—ensure [RECENT_OUTPUT_SAMPLES] contains a representative mix of failing and passing examples, not just the worst cases, to prevent biased root cause analysis. (2) Output schema validation—parse the model's JSON response and strictly validate the severity field against your internal SEV levels (e.g., SEV1-SEV5) and confirm that affected_output_scope contains a quantifiable percentage or count. If validation fails, retry once with a stricter [CONSTRAINTS] block appended to the prompt. (3) Human-in-the-loop gating—for any incident classified as SEV1 or SEV2, the system must not auto-post to a shared incident channel. Instead, it should create a draft incident ticket and page the primary on-call with a summary, requiring human acknowledgment before broadcast.

For logging and evaluation, store the full prompt, the raw model output, the parsed incident JSON, and the human's final severity reclassification in a structured incident log. This dataset becomes your ground truth for evaluating the prompt's classification accuracy over time. A common failure mode is the model hallucinating a root cause that sounds plausible but is not grounded in the provided [RECENT_OUTPUT_SAMPLES]. To mitigate this, implement an automated eval that checks if the evidence array in the output directly quotes or references the input samples. If the evidence grounding score falls below a threshold, flag the incident assessment for mandatory human review, regardless of its severity classification. Avoid using this prompt for real-time, sub-second gating decisions; it is an asynchronous diagnostic tool, not a low-latency filter.