Example Drift Root Cause Analysis Prompt

This prompt is a diagnostic instrument for incident responders and MLOps engineers investigating accuracy drops in production AI systems. When a drift alert fires and you know the examples no longer match production data, this prompt takes the drift alert details, a sample of recent production inputs, and the history of your few-shot example set. It outputs hypothesized root causes with confidence ratings, such as seasonal shift, product change, user population change, or upstream data pipeline change. Use this prompt after drift detection and before committing to a costly example refresh. It is a diagnostic tool, not a replacement for statistical drift metrics.

The prompt works best when you have already confirmed drift through quantitative methods—embedding distance, distribution divergence, or staleness scoring—and need to explain why the shift occurred. Provide the prompt with a structured drift alert payload (affected example indices, divergence scores, timestamp), a representative sample of 20–50 recent production inputs, and metadata about your example set (creation date, intended domain, last refresh date). The model will return a ranked list of hypotheses, each with a confidence rating and supporting evidence drawn from the inputs you provided. This structured output feeds directly into your incident response runbook: low-confidence hypotheses trigger deeper investigation, while high-confidence product-change hypotheses may route to the engineering team that owns the upstream data source.

Do not use this prompt as a substitute for statistical drift detection. It cannot compute embedding distances or run Kolmogorov-Smirnov tests. Do not use it when you lack a representative production sample—the hypotheses will be speculative. Do not use it for real-time alerting; it is a post-detection analysis step. The output is a set of informed hypotheses, not a root cause verdict. Always validate the top hypothesis with the team that owns the data pipeline or product surface before acting. For regulated systems, log the hypothesis output alongside the drift metrics and the human reviewer's confirmation as part of your audit trail.

This prompt is designed for incident responders who have received an automated drift alert and need to move from detection to diagnosis. It takes the raw evidence—a sample of recent production inputs, the current few-shot example set, and the drift alert metadata—and produces a structured root cause analysis with ranked hypotheses. The goal is to accelerate the investigation phase before committing engineering time to a fix.

textCopy
You are an AI incident analyst specializing in prompt example drift. Your task is to perform a root cause analysis given a drift alert, a sample of recent production inputs, and the current few-shot example set.

## INPUTS
[DRIFT_ALERT_DETAILS]
[PRODUCTION_INPUT_SAMPLE]
[CURRENT_EXAMPLE_SET]
[EXAMPLE_SET_VERSION_HISTORY]

## INSTRUCTIONS
1. Compare the production input sample against the current example set. Identify concrete differences in topic distribution, vocabulary, user intent, input complexity, and entity types.
2. Review the example set version history for recent changes that could explain the drift.
3. Generate 3-5 hypothesized root causes ranked by likelihood. For each hypothesis, provide:
   - A clear label (e.g., "Seasonal Topic Shift", "Product Feature Change", "User Population Change", "Upstream Data Pipeline Change")
   - Supporting evidence from the inputs
   - A confidence score (Low, Medium, High)
   - A suggested next step to validate or rule out the hypothesis
4. If no single cause is clear, state that explicitly and recommend a multi-factor investigation.

## CONSTRAINTS
- Do not fabricate evidence. Only reference patterns visible in the provided data.
- If the production sample or example set is insufficient for a conclusion, flag data gaps as a finding.
- Use precise language. Avoid vague terms like "seems different" without quantifying the difference.

## OUTPUT FORMAT
Return a JSON object with the following schema:
{
  "analysis_summary": "string",
  "data_quality_notes": ["string"],
  "hypotheses": [
    {
      "label": "string",
      "confidence": "Low|Medium|High",
      "evidence": ["string"],
      "suggested_validation": "string"
    }
  ],
  "recommended_immediate_action": "string"
}

To adapt this prompt, replace the square-bracket placeholders with real data from your monitoring systems. The [DRIFT_ALERT_DETAILS] should include the drift score, affected example indices, and the detection method used. The [PRODUCTION_INPUT_SAMPLE] should be a representative batch of recent user inputs, ideally 50-200 examples. The [CURRENT_EXAMPLE_SET] is the full set of few-shot examples currently in the prompt. The [EXAMPLE_SET_VERSION_HISTORY] should include timestamps and summaries of recent changes. If your drift detection system already clusters production inputs, include those cluster labels to improve the analysis. For high-severity incidents, always have a human review the hypotheses before acting on them—this prompt accelerates investigation but does not replace engineering judgment.

This prompt is designed to be triggered by an upstream drift alert, not run manually on a schedule. The implementation harness should accept a structured alert payload containing the production sample, the current example set, and the historical example set versions. The prompt's output is a set of hypothesized root causes with confidence ratings, which must be treated as investigative leads—not definitive conclusions. The harness is responsible for validating that the model's output conforms to the expected schema before surfacing it to an on-call responder.

Wire the prompt into an incident management pipeline with the following stages. First, a validation gate checks that the input payload includes all required fields: drift_alert_metadata, production_sample, current_example_set, and example_set_history. Missing fields should abort the workflow and log a structured error. Second, the prompt is executed against a model with strong reasoning capabilities (such as GPT-4o or Claude 3.5 Sonnet) with temperature=0 to maximize reproducibility. Third, the output is parsed and validated against the expected JSON schema: a list of hypotheses, each with a cause_category, description, confidence_score (0.0–1.0), and supporting_evidence array. Any output that fails schema validation should trigger a single retry with the validation error appended to the prompt as a correction instruction. If the retry also fails, escalate to a human with the raw output and validation errors attached.

Log every invocation with the alert ID, model version, prompt version, input hashes, output, validation result, and latency. This audit trail is critical for post-incident review and for tuning the drift detection thresholds that feed this prompt. Avoid wiring this prompt directly into an automated example refresh pipeline—its output is a diagnostic hypothesis, not an action. The next step after receiving the root cause hypotheses should be a human review or a separate evaluation prompt that corroborates the leading hypothesis against additional evidence before any example set changes are made.