Golden Set Deviation Diagnosis Prompt

This prompt is a single-pair diagnostic tool for QA engineers and AI reliability teams responding to an active eval alert. Its job is to take one production response that scored below threshold and its corresponding golden answer, then produce a structured deviation report. The report must separate acceptable paraphrasing from genuine quality regressions—semantic drift, missing facts, hallucinated additions, and formatting non-compliance—so the team can decide whether to roll back, patch the prompt, fix a retrieval gap, or tune the eval rubric itself.

Use this when an automated eval score drops and you need to understand the nature of the failure before acting. The ideal user has the production response, the golden answer, the eval rubric that flagged the failure, and ideally the trace context (retrieved passages, tool calls, or prior turns) that produced the response. The prompt expects these as explicit inputs and will return a categorized diff. Do not use this for pre-release regression testing against a full golden dataset; it is not designed for bulk comparison or statistical analysis. It is a root-cause investigation tool for a single failure instance.

Before running this prompt, confirm that the eval failure is not a known infrastructure issue, a model provider outage, or a deployment misconfiguration. If the failure is part of a broader incident, use the Incident Trace Correlation prompt first to isolate common patterns. If you suspect the golden answer itself is stale or contaminated, run the Golden Set Contamination Check prompt before diagnosing the deviation. This prompt assumes the golden answer is trustworthy; if it is not, the deviation report will mislead you. After receiving the report, if the root cause points to a retrieval gap, hand off to the Retrieval Gap Root-Cause prompt. If it points to a prompt version regression, use the Prompt Version Rollback Trigger Analysis prompt to decide on a rollback.

The prompt below is designed for a QA engineer or AI reliability developer who has a known-good reference answer and a production response that may have drifted. It forces the model to act as a forensic comparator, not a creative writer. The output is a structured diff that separates acceptable paraphrasing from genuine regressions, making it suitable for automated eval pipelines or manual triage. Before using this prompt, ensure you have the production trace available for context, as the model may need to reference retrieval steps or tool calls to explain certain deviations.

codeCopy
You are a forensic QA comparator. Your task is to compare a production response against a golden-set reference answer and produce a structured deviation diagnosis.

## INPUTS
- Golden Answer: [GOLDEN_ANSWER]
- Production Response: [PRODUCTION_RESPONSE]
- Evaluation Rubric Dimensions: [RUBRIC_DIMENSIONS]
- Production Trace Summary (optional): [TRACE_SUMMARY]

## CONSTRAINTS
- Do not rewrite or improve either answer. Only compare.
- Treat the golden answer as the ground truth for factual content.
- Distinguish between acceptable paraphrasing and genuine quality regressions.
- If the production response adds information not in the golden answer, flag it as a potential hallucination unless the trace summary confirms it came from retrieved context.
- If the golden answer contains information missing from the production response, flag it as a factual omission.
- For formatting violations, compare against the expected output schema: [OUTPUT_SCHEMA].

## OUTPUT FORMAT
Return a JSON object with the following structure:
{
  "verdict": "PASS" | "FAIL" | "REVIEW_REQUIRED",
  "summary": "One-sentence summary of the comparison outcome.",
  "deviations": [
    {
      "type": "semantic_drift" | "factual_omission" | "hallucinated_addition" | "format_violation" | "acceptable_paraphrase",
      "severity": "CRITICAL" | "MAJOR" | "MINOR" | "INFO",
      "golden_excerpt": "Relevant excerpt from the golden answer.",
      "production_excerpt": "Relevant excerpt from the production response.",
      "explanation": "Why this deviation matters and which rubric dimension it affects.",
      "trace_correlation": "If trace summary is provided, note any trace event that explains this deviation."
    }
  ],
  "rubric_dimension_scores": {
    "[DIMENSION_NAME]": {
      "score": 0-100,
      "justification": "Brief explanation of the score."
    }
  },
  "requires_human_review": true | false,
  "human_review_reason": "If requires_human_review is true, explain why."
}

## INSTRUCTIONS
1. Parse both answers and identify all factual claims, structural elements, and formatting choices.
2. Compare each claim in the golden answer against the production response. Flag omissions.
3. Compare each claim in the production response against the golden answer. Flag additions not explained by the trace.
4. Evaluate formatting against the output schema. Flag structural violations.
5. Score each rubric dimension based on the deviations found.
6. Set verdict to PASS if no CRITICAL or MAJOR deviations exist. Set to FAIL if any CRITICAL deviation exists. Set to REVIEW_REQUIRED if MAJOR deviations exist without CRITICAL ones.
7. If the trace summary is provided, correlate deviations to trace events where possible.

Adapt this template by replacing the square-bracket placeholders with your actual data. The [RUBRIC_DIMENSIONS] placeholder should contain a list of the specific quality dimensions you are evaluating, such as factual accuracy, completeness, citation correctness, or tone. The [OUTPUT_SCHEMA] placeholder should describe the expected structure of the production response, including required fields, types, and formatting rules. If you do not have a trace summary, remove the [TRACE_SUMMARY] placeholder and the trace correlation instructions. For high-stakes domains like healthcare or finance, always set requires_human_review to true for any FAIL or REVIEW_REQUIRED verdict and route the output to a human reviewer before taking action. Test this prompt against at least five known deviation cases before integrating it into an automated pipeline to calibrate severity thresholds for your specific use case.

This prompt is designed to operate as a diagnostic step within an automated eval failure pipeline, not as a standalone manual review tool. When a production response fails a quality eval against a golden set, the pipeline should automatically invoke this prompt with the failed response, the golden answer, the eval rubric, and the relevant trace segments. The prompt's structured diff output becomes the primary evidence packet for root-cause classification, enabling the system to route the failure to the correct remediation workflow—prompt rollback, retrieval tuning, few-shot example refresh, or human review—without requiring an engineer to manually reconstruct what went wrong.

Integration pattern: The prompt expects four inputs that must be assembled programmatically before invocation. [PRODUCTION_RESPONSE] comes from the response store or trace log. [GOLDEN_ANSWER] is fetched from the golden set database using the test case ID attached to the eval failure event. [EVAL_RUBRIC] is the specific rubric that flagged the failure, including the dimension scores and the failing threshold. [TRACE_CONTEXT] should include the retrieved passages, tool-call outputs, and any truncated context-window evidence from the production trace—this is critical for distinguishing a hallucination from a retrieval gap. Validation layer: Before the prompt output enters any automated decision path, apply a schema validator that checks for the required fields (deviation_category, semantic_drift, missing_facts, hallucinated_additions, formatting_violations, severity, trace_evidence_links). If the output fails schema validation, retry once with a stricter [OUTPUT_SCHEMA] constraint. If it fails again, escalate to a human reviewer with the raw trace and both responses attached.

Model selection and cost: This is a reasoning-heavy diagnostic task that benefits from a capable model with strong instruction-following and structured output discipline. Use a model with JSON mode or structured output support (e.g., GPT-4o, Claude 3.5 Sonnet, or equivalent). Do not route this to a lightweight or cost-optimized model—false negatives in deviation detection will cause the pipeline to miss regressions that should trigger rollbacks. Logging and audit: Every invocation should log the full prompt input, the structured diff output, the eval failure event ID, and the pipeline's routing decision. This audit trail is essential for retrospective analysis when a deviation pattern recurs across multiple failures. Human review gate: For any deviation classified as severity: critical or where hallucinated_additions contains claims about regulated domains (healthcare, finance, legal), the pipeline must not auto-remediate. Route to a human review queue with the structured diff pre-filled as the investigation starting point. Next step: After the deviation is diagnosed, feed the structured output into the appropriate remediation prompt—Prompt Version Rollback Trigger Analysis for regression patterns, Retrieval Gap Root-Cause for Eval Failure for missing evidence, or Hallucination Spot-Check Against Eval Rubric for verification before action.