Side-by-Side Factual Accuracy Comparison Prompt

This prompt is for content verification teams and evaluation engineers who need to determine which of two AI-generated outputs is more factually accurate when compared against a known reference or source material. The job-to-be-done is not to judge style, tone, or helpfulness in a vacuum, but to produce a structured, evidence-linked preference based on claim-level accuracy, error severity, and the quality of any corrections present. The ideal user is someone running a champion-challenger test for a RAG pipeline, auditing a summarization model against source documents, or curating preference data for fine-tuning where factual reliability is the primary axis of improvement.

Use this prompt when you have a single, authoritative reference text and two candidate outputs that should be grounded in that reference. The prompt requires you to supply the reference material, the two outputs, and a defined output schema that captures the preference verdict, a justification, a list of factual errors with severity ratings, and explicit links to the evidence in the source. Do not use this prompt when the reference material is absent, when the outputs are creative or subjective (e.g., marketing copy, poetry), or when you are comparing stylistic elements without a factual ground truth. In those cases, a general pairwise preference prompt or a tone-and-style comparison prompt is more appropriate.

The primary constraint is that this prompt performs a verification task, not a generation task. It assumes the reference is correct and complete. If the reference itself contains errors, the comparison will be systematically flawed. Before wiring this into an automated evaluation pipeline, you must validate that your reference data is accurate and that both candidate outputs are intended to be grounded in it. For high-stakes domains such as healthcare, legal, or finance, always route outputs flagged with high-severity errors for human review, and log the evidence links for auditability. The next step is to copy the prompt template, replace the placeholders with your reference and outputs, and run it against a small set of known cases to calibrate the judge's severity before scaling to production.

This template is designed for content verification teams who need to compare two AI-generated outputs against known facts or reference data. The prompt forces the model to evaluate each output at the claim level, link claims to evidence, assess error severity, and produce a justified preference. Use this when you have authoritative reference material and need a traceable, auditable comparison rather than a subjective quality judgment.

textCopy
You are a factual accuracy evaluator. Your task is to compare two outputs against provided reference facts and determine which output is more factually accurate.

## REFERENCE FACTS
[REFERENCE_FACTS]

## OUTPUT A
[OUTPUT_A]

## OUTPUT B
[OUTPUT_B]

## EVALUATION CRITERIA
[EVALUATION_CRITERIA]

## INSTRUCTIONS
1. Extract every factual claim from Output A and Output B separately.
2. For each claim, determine whether the reference facts support it, contradict it, or provide no evidence either way.
3. Classify each contradiction by severity:
   - CRITICAL: Reverses a key fact, introduces dangerous misinformation, or materially changes the meaning.
   - MAJOR: States a wrong value, date, name, or figure that changes the factual record.
   - MINOR: Imprecise wording, rounding errors, or omissions that do not change the core meaning.
4. For supported claims, cite the specific reference fact that supports them.
5. For contradicted claims, cite the specific reference fact that contradicts them.
6. For claims with no evidence, mark them as UNSUPPORTED and explain why the reference facts are silent.
7. Do not penalize an output for including true information that the other output omitted, unless the omitted information was explicitly required by [EVALUATION_CRITERIA].
8. Do not penalize stylistic differences, word choice, or formatting unless they introduce factual errors.

## OUTPUT FORMAT
Return a JSON object with this exact structure:
{
  "output_a_claims": [
    {
      "claim": "exact claim text",
      "verdict": "SUPPORTED|CONTRADICTED|UNSUPPORTED",
      "severity": "CRITICAL|MAJOR|MINOR|null",
      "evidence_reference": "specific reference fact or null",
      "explanation": "brief explanation"
    }
  ],
  "output_b_claims": [
    {
      "claim": "exact claim text",
      "verdict": "SUPPORTED|CONTRADICTED|UNSUPPORTED",
      "severity": "CRITICAL|MAJOR|MINOR|null",
      "evidence_reference": "specific reference fact or null",
      "explanation": "brief explanation"
    }
  ],
  "comparison_summary": {
    "output_a_contradictions": {"critical": 0, "major": 0, "minor": 0},
    "output_b_contradictions": {"critical": 0, "major": 0, "minor": 0},
    "output_a_unsupported": 0,
    "output_b_unsupported": 0
  },
  "preference": "OUTPUT_A|OUTPUT_B|TIE",
  "preference_justification": "explain why one output is preferred based on factual accuracy, not style or completeness",
  "confidence": 0.0_to_1.0,
  "tie_breaker_applied": "if TIE, explain which tie-breaking dimension was used from [TIE_BREAKING_CRITERIA]"
}

## CONSTRAINTS
[CONSTRAINTS]

## TIE-BREAKING CRITERIA
[TIE_BREAKING_CRITERIA]

Adaptation guidance: Replace [REFERENCE_FACTS] with your ground-truth document, structured data, or verified source text. Replace [OUTPUT_A] and [OUTPUT_B] with the two model outputs under comparison. Use [EVALUATION_CRITERIA] to specify what factual dimensions matter most for your use case—for example, 'accuracy of financial figures takes priority over narrative fluency.' Use [CONSTRAINTS] to add domain-specific rules such as 'do not flag rounding differences under 1% as errors' or 'treat missing mandatory disclosures as CRITICAL contradictions.' Use [TIE_BREAKING_CRITERIA] to define how ties should be resolved, such as 'prefer the output with fewer unsupported claims' or 'prefer the output that cites more reference facts.' The output schema is intentionally verbose to support downstream audit trails and evidence linking. If you need a lighter-weight version for high-throughput pipelines, remove the per-claim arrays and keep only the comparison summary and preference fields.

Before deploying this prompt into a production pipeline, validate that the JSON output matches the expected schema on at least 20 test pairs with known accuracy differences. Common failure modes include the model conflating style preferences with factual accuracy, failing to extract implicit claims, or producing evidence references that are paraphrases rather than direct citations. For high-risk domains such as healthcare, legal, or financial content, always route outputs with confidence below 0.8 or CRITICAL contradictions to human review. Wire the prompt into an evaluation harness that logs every comparison, tracks inter-rater agreement when multiple judges evaluate the same pair, and alerts on score drift over time.

This prompt is designed to operate as a stateless comparison function inside a larger fact-checking pipeline. It expects two pre-generated outputs and a structured evidence payload. The application layer is responsible for assembling these inputs, calling the model, parsing the structured verdict, and routing the result. Do not use this prompt as a standalone chatbot interaction; it must be wrapped in an API call with strict output validation before the result is trusted downstream.

Input assembly is the critical pre-call step. The [OUTPUT_A] and [OUTPUT_B] fields should be the raw text responses from the two systems under comparison. The [REFERENCE_FACTS] field must contain the ground-truth evidence, formatted as a list of atomic claims with source citations. If your evidence comes from a RAG pipeline, extract and deduplicate the supporting passages before populating this field. Position bias mitigation requires two calls per comparison pair: one with Output A in the first position and one with Output B in the first position. If the verdicts disagree, flag the pair for human review. Log the raw position-swapped results for inter-rater reliability analysis over time.

Output validation must happen before the verdict is accepted. Parse the model response as JSON and verify: (1) the preferred_output field contains exactly A or B, (2) the confidence field is a float between 0.0 and 1.0, (3) the claim_level_accuracy object contains numeric counts that sum correctly, and (4) every entry in error_instances includes a non-empty claim, error_type, severity, and evidence_citation. If validation fails, retry once with the same inputs and a stronger format reminder appended to the prompt. If the retry also fails, route the pair to a human review queue with the raw model output attached for diagnosis.

Model choice matters for this workflow. Use a model with strong instruction-following and structured output capabilities, such as gpt-4o or claude-3-5-sonnet. Set temperature=0 to maximize reproducibility. If you are comparing outputs from a smaller or less capable model, be aware that the judge model's own factual limitations can introduce bias—it may penalize correct but obscure claims it doesn't recognize. For high-stakes domains (healthcare, finance, legal), always include a human review gate for pairs where confidence is below 0.8 or where error severity is critical. Log every verdict with the full input payload, model response, validation result, and human override for auditability.

Integration pattern: wrap this prompt in a function compare_factual_accuracy(output_a, output_b, reference_facts, domain_risk_level) that handles position swapping, validation, retries, and escalation. Return a structured result object that downstream systems can act on: {winner, confidence, error_summary, requires_human_review, audit_log}. Never expose raw model output directly to end users or automated decision systems without this validation layer. For continuous monitoring, track the rate of validation failures, position-swap disagreements, and human overrides—these metrics will surface judge drift or evidence quality problems before they corrupt your comparison pipeline.