Golden Answer vs Production Answer Pairwise Judge Prompt

This prompt is a precision instrument for QA engineers and AI ops teams who need to gate a new prompt version, model upgrade, or configuration change before it reaches production. Its job is to compare a single production model output against a known, curated golden reference answer for the same input and produce a structured win/loss/tie judgment with detailed rationale. You should use it when you have a golden dataset of input-output pairs that represent your quality bar and you need to automate the pairwise comparison step in your release pipeline. The prompt is designed to catch regressions—cases where the new output is worse than the golden answer—and to flag ties or wins that still warrant human review for subtle quality shifts.

This is not a general-purpose evaluation prompt. Do not use it to generate golden answers, discover edge cases, or evaluate multi-turn conversations without significant adaptation. It assumes you already have a curated golden answer and a production trace to compare. It is also not a substitute for a full eval harness; it is one judge in a larger pipeline that should include assertion checks, semantic equivalence thresholds, and human calibration. The prompt works best when the golden answer is a single, definitive reference, not a range of acceptable responses. If your quality bar allows multiple valid outputs, you will need to pair this with a rubric-based judge or a semantic similarity check to avoid false regressions.

Before using this prompt, ensure you have: (1) a stable golden dataset with input-output pairs that represent your quality bar, (2) a production trace containing the model output you want to evaluate, and (3) a clear definition of what constitutes a regression for your use case. The prompt includes a severity classification field, so you must calibrate what 'critical,' 'major,' and 'minor' mean for your product. If you are evaluating outputs in a regulated domain, always route critical and major regressions to human review. The next step after reading this section is to copy the prompt template, wire it into your eval harness, and run a calibration pass against a set of known-good and known-bad pairs to validate the judge's alignment with your quality standards.

This prompt template is the core of the Golden Answer vs Production Answer Pairwise Judge workflow. It instructs an LLM judge to evaluate two answers side by side—one from a trusted golden dataset and one from the current production system—and produce a structured judgment. The output includes a winner, a detailed rationale, a regression severity classification, and the quality dimensions affected. Use this template when you need automated, repeatable comparisons to gate prompt changes before release or to monitor production drift over time.

textCopy
You are an expert QA evaluator comparing two answers generated from the same input. Your job is to determine which answer is better, whether a regression has occurred, and how severe any quality degradation is.

## INPUT
[INPUT]

## GOLDEN ANSWER (Reference)
[GOLDEN_ANSWER]

## PRODUCTION ANSWER (Candidate)
[PRODUCTION_ANSWER]

## EVALUATION CRITERIA
[EVALUATION_CRITERIA]

## OUTPUT SCHEMA
Return a valid JSON object with exactly these fields:
{
  "judgment": "golden_wins" | "production_wins" | "tie",
  "confidence": 0.0 to 1.0,
  "rationale": "Concise explanation of the comparison, citing specific differences.",
  "regression_detected": true | false,
  "regression_severity": "none" | "low" | "medium" | "high" | "critical",
  "affected_dimensions": ["accuracy", "completeness", "clarity", "tone", "safety", "format", "citation", "other"],
  "golden_strengths": ["What the golden answer did better"],
  "production_strengths": ["What the production answer did better"],
  "recommendation": "Optional: suggested fix or investigation path if regression detected."
}

## CONSTRAINTS
[CONSTRAINTS]

## POSITION BIAS CHECK
Before finalizing, swap the order of the two answers in your mental review and confirm your judgment does not change. If it does, flag this in the rationale.

## EXAMPLES
[EXAMPLES]

Now evaluate the pair above and return only the JSON object.

Adapt this template by populating the square-bracket placeholders with your specific data and rules. [INPUT] should contain the original user query or system prompt that produced both answers. [GOLDEN_ANSWER] is the trusted reference output from your curated dataset. [PRODUCTION_ANSWER] is the output from the prompt version under test. [EVALUATION_CRITERIA] should list the quality dimensions that matter for your use case—accuracy, tone, format compliance, citation faithfulness, or domain-specific requirements. [CONSTRAINTS] can include rules like "penalize hallucinations heavily," "prefer concise answers unless detail is explicitly requested," or "flag any PII leakage as critical." [EXAMPLES] should contain one or two few-shot demonstrations of correct judgments to calibrate the judge. Always include the position bias check instruction; without it, judges frequently favor the first answer shown. For high-stakes domains, route critical severity judgments to human review before accepting the result.

The Golden Answer vs Production Answer Pairwise Judge Prompt is not a one-shot evaluation tool; it is a measurement instrument that must be embedded in a harness that controls for known LLM judge failure modes. The core implementation wraps the prompt in a loop that runs each comparison twice with swapped positions (A/B then B/A) to detect position bias. If the judge flips its preference when the order changes, the comparison is flagged as unreliable and either discarded or escalated to human review. The harness also injects calibration pairs—known wins, losses, and ties—at a configurable rate (typically 10–20% of comparisons) to measure judge drift over a batch run.

The harness should enforce a strict output schema on every judge response. Use a JSON schema validator to confirm the presence of judgment (win/loss/tie), winner, rationale, regression_severity (none/minor/major/critical), and affected_quality_dimensions (an array of labels like accuracy, completeness, tone, format). If the model returns malformed JSON, retry once with a repair prompt that includes the raw output and the expected schema. After two failures, log the comparison as eval_error and exclude it from aggregate metrics. For high-risk domains such as healthcare or finance, route all regression_severity: critical judgments to a human review queue before accepting them as true regressions.

Inter-rater reliability should be computed per batch. Track Cohen's kappa or Gwet's AC1 between the LLM judge and human-annotated calibration pairs, and between the judge's own A/B and B/A decisions. If agreement drops below a threshold (e.g., kappa < 0.6), pause the evaluation pipeline and investigate—common causes include prompt drift, model version changes, or a shift in the production output distribution that the judge's rubric no longer covers. Log every judgment with the prompt version, model identifier, timestamp, position order, and raw input pair so that QA engineers can reproduce any individual decision during an audit.

Model choice matters for judge reliability. Frontier models (GPT-4o, Claude 3.5 Sonnet) tend to produce more consistent pairwise judgments than smaller models, but they still exhibit position bias and occasional preference for verbose or stylistically different answers over substantively correct ones. If cost or latency requires a smaller judge model, increase the calibration pair frequency and tighten the agreement thresholds. For teams running large-scale regression suites, consider batching comparisons and running them asynchronously with a configurable concurrency limit to avoid rate-limit throttling while keeping total evaluation time predictable.

Wire the harness output into your release gate. Aggregate pairwise results into a regression dashboard that shows win/loss/tie distributions per quality dimension, judge reliability metrics, and a trend line comparing the current prompt version against the previous one. Set explicit promotion criteria: for example, the new prompt must not lose more than 5% of comparisons on accuracy and must have zero critical severity regressions. If the gate fails, block the release and route the losing cases to a prompt engineer for root cause analysis. Avoid the trap of treating a single judge run as ground truth—pairwise judgments are a signal, not a verdict, and the harness exists to quantify how much to trust that signal.