Pairwise Comparison Prompt for RAG Answer Quality

This playbook is for RAG system builders who need to compare two grounded answers against the same source material and decide which one is better. Unlike absolute scoring, pairwise comparison forces a relative judgment that is often more reliable for champion-challenger testing, model selection, and RLHF data pipelines. Use this prompt when you have two answers generated from the same retrieval context and you need a structured preference verdict based on faithfulness, citation accuracy, completeness, and source conflict handling. This prompt includes a hallucination cross-check and citation verification harness that makes it suitable for production evaluation pipelines where groundedness is the primary safety property.

The ideal user is an AI engineer or evaluation lead running a champion-challenger test between two RAG configurations, an ML engineer curating preference pairs for DPO fine-tuning, or a platform team member building an automated regression gate that compares a new prompt or model version against the current production baseline. The required context includes the original user question, the shared set of retrieved source passages, and the two answers being compared. Without this shared grounding material, the comparison reduces to stylistic preference rather than factual accuracy, which defeats the purpose of this prompt.

Do not use this prompt when the two answers are generated from different retrieval contexts, when you need an absolute quality score rather than a relative preference, or when the evaluation criteria extend beyond groundedness into dimensions like creativity, brand voice, or user engagement that lack source evidence. This prompt is also inappropriate for single-answer evaluation, multi-turn conversation assessment, or cases where the source material itself is contradictory or incomplete without explicit handling instructions. For those scenarios, use a rubric-based absolute scoring prompt, a multi-turn evaluation prompt, or a source quality pre-check respectively.

Before running this prompt at scale, calibrate its judgments against human preference labels on a representative sample of your domain data. Position bias is a known failure mode in pairwise LLM judges—always randomize the order of answers A and B, and run a subset of comparisons in both orders to measure sensitivity. The hallucination cross-check harness described in the implementation section adds a second verification pass that independently extracts claims from each answer and checks them against the source passages, providing a safety net when the primary comparison misses unsupported statements.

This prompt template is designed to be dropped into an evaluation harness that supplies two candidate answers and the source context from which they were generated. It forces a structured, evidence-backed preference based on faithfulness, citation accuracy, completeness, and source conflict handling. The placeholders are intentionally granular so you can control the comparison dimensions, output schema, and risk tolerance without rewriting the core instruction.

textCopy
You are an evaluation judge comparing two answers generated by a Retrieval-Augmented Generation (RAG) system. Both answers were produced from the same set of source documents. Your job is to determine which answer is better and explain why.

## SOURCE MATERIAL
[SOURCE_CONTEXT]

## ANSWER A
[ANSWER_A]

## ANSWER B
[ANSWER_B]

## COMPARISON DIMENSIONS
Evaluate both answers on the following dimensions. For each dimension, state which answer performs better and provide a brief evidence-backed justification.

1. **Faithfulness**: Does the answer contain only claims supported by the source material? Flag any unsupported claims or hallucinations.
2. **Citation Accuracy**: Are citations present, correctly placed, and do they point to the right source passages? Flag miscitations or missing citations where a claim requires support.
3. **Completeness**: Does the answer address all parts of the user's question without omitting key information present in the sources?
4. **Source Conflict Handling**: If sources contradict each other, does the answer acknowledge the conflict rather than silently picking one side or blending contradictory claims?

## OUTPUT FORMAT
Return a JSON object with the following structure:

{
  "preferred_answer": "A" | "B" | "TIE",
  "confidence": 0.0-1.0,
  "dimension_verdicts": {
    "faithfulness": {"winner": "A" | "B" | "TIE", "justification": "string"},
    "citation_accuracy": {"winner": "A" | "B" | "TIE", "justification": "string"},
    "completeness": {"winner": "A" | "B" | "TIE", "justification": "string"},
    "source_conflict_handling": {"winner": "A" | "B" | "TIE", "justification": "string"}
  },
  "overall_justification": "string summarizing the key reasons for the preference",
  "hallucination_flags": [
    {"answer": "A" | "B", "claim": "string", "severity": "CRITICAL" | "MAJOR" | "MINOR"}
  ],
  "citation_errors": [
    {"answer": "A" | "B", "claim": "string", "cited_source": "string", "error_type": "MISCITATION" | "MISSING_CITATION" | "IRRELEVANT_CITATION"}
  ]
}

## CONSTRAINTS
- Base your judgment ONLY on the source material provided. Do not use external knowledge.
- If both answers are equally good or equally flawed, use "TIE" and explain why neither is clearly better.
- Flag every unsupported claim in the hallucination_flags array, even if the answer is otherwise preferred.
- If a dimension is not applicable (e.g., no citations exist in either answer), mark it as "TIE" and explain.
- Confidence should reflect how certain you are in the preference, not the quality of the preferred answer.
- [ADDITIONAL_CONSTRAINTS]

## EXAMPLES
[FEW_SHOT_EXAMPLES]

Adapt this template by adjusting the comparison dimensions to match your RAG system's quality priorities. If citation accuracy is irrelevant because your system doesn't produce citations, remove that dimension and update the output schema. Add few-shot examples that demonstrate your expected preference behavior, especially for edge cases like partially correct answers, conflicting sources, or answers that are faithful but incomplete. The [ADDITIONAL_CONSTRAINTS] placeholder lets you inject domain-specific rules—for example, preferring shorter answers when quality is equal, or deprioritizing answers that use speculative language. Always validate the JSON output against the schema before accepting the verdict, and log dimension-level scores for drift monitoring over time.

This prompt is designed to operate as a stateless evaluation function within a larger RAG quality pipeline. It expects two complete answer objects and a shared set of source documents as input, and it returns a structured preference verdict. The implementation harness must handle input assembly, position randomization, output validation, and result storage. Do not use this prompt directly on raw model outputs without first attaching the source documents that grounded each answer. The prompt assumes answers are already generated and that the comparison is the only task.

Position bias mitigation is mandatory. Before calling the prompt, randomly swap the order of [ANSWER_A] and [ANSWER_B] and record the swap state in a metadata field. After receiving the preference verdict, remap the result back to the original answer identities. Without this, the model may systematically prefer the first or second position regardless of quality. Implement this as a pre-processing step in your harness, not as a prompt instruction. Log the swap state alongside every comparison result for later audit and bias analysis.

Output validation must be strict. Parse the model response as JSON and validate that the preference field contains exactly one of the allowed values: ANSWER_A, ANSWER_B, or TIE. Reject any response that does not conform to the output schema. If the model returns a preference for an answer that does not exist in the input, log the mismatch and retry with a clarified prompt. For high-stakes evaluation pipelines, implement a retry budget of 2-3 attempts with exponential backoff before escalating to human review. Store raw model responses alongside parsed results for debugging judge drift over time.

Citation verification requires a post-processing step. The prompt asks the judge to flag unsupported claims, but the harness should independently cross-check any hallucination flags against the provided source documents using a separate grounding check. Do not treat the judge's hallucination detection as authoritative without verification. For production systems, maintain a disagreement log where the judge's hallucination flags are compared against a secondary fact-checking prompt or human audit sample. This dual-layer approach catches judge errors before they corrupt your evaluation metrics.

Model choice matters for judge reliability. This prompt works best with frontier models that have strong instruction-following and structured output capabilities. Avoid using smaller or older models as judges for RAG quality comparison, as they are more likely to produce inconsistent verdicts or hallucinate citations. If cost constraints require a smaller judge model, increase the retry budget and lower the confidence threshold for automatic acceptance. Always run a calibration set of 50-100 human-labeled comparison pairs through the judge before trusting its output in production. Track judge-human agreement rates over time and trigger a recalibration review if agreement drops below your defined threshold.