Ground-Truth Answer Comparison Prompt for RAG

This prompt is for RAG system builders who need to validate retrieval-augmented outputs against a known-correct answer. It acts as an automated judge, producing a structured comparison report that covers answer correctness, citation accuracy, and source faithfulness. Use this when you have a golden dataset of question-reference answer pairs and need to measure how well your RAG pipeline performs before shipping a new retriever, prompt, or model. It also detects abstention, distinguishing between a model that correctly refuses to answer and one that produces a wrong answer. This is not a prompt for generating answers. It is a prompt for evaluating them.

The ideal user is an AI engineer or evaluation lead running a regression suite before a RAG pipeline change. You should have a validated golden dataset where each record contains a user question, a known-correct reference answer, and the retrieved context chunks that were provided to the generation model. The prompt expects the model's generated answer alongside this ground truth. It is designed for pre-release gates, not real-time production monitoring. If you need to evaluate outputs without a reference answer, use a rubric-based judge instead. If you need to evaluate multi-turn conversations, this single-turn comparison prompt will miss context-switching failures.

Do not use this prompt when the reference answer is itself uncertain, when multiple valid answers exist with no clear ground truth, or when the evaluation requires domain expertise beyond what the judge model can reliably assess. For high-stakes domains like healthcare or legal review, always route flagged disagreements to a human reviewer rather than treating the judge's output as final. Wire the prompt into a test harness that logs every comparison, tracks score distributions over time, and alerts on sudden regressions. The next section provides the copy-ready template you can adapt to your schema and tolerance thresholds.

This prompt template is the core of the Ground-Truth Answer Comparison playbook. It instructs the model to act as an evaluator, comparing a generated answer against a reference answer and the provided source context. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to integrate into an evaluation harness. The goal is a structured report that assesses answer correctness, citation accuracy, and source faithfulness, while also detecting when the RAG system abstained from answering.

textCopy
You are an expert evaluation judge for a Retrieval-Augmented Generation (RAG) system. Your task is to compare a generated answer against a known-correct ground-truth answer and the provided source context. Produce a structured comparison report.

## INPUTS
**User Question:** [USER_QUESTION]

**Generated Answer:** [GENERATED_ANSWER]

**Ground-Truth Answer:** [GROUND_TRUTH_ANSWER]

**Source Context (provided to the RAG system):** [SOURCE_CONTEXT]

## EVALUATION CRITERIA
1.  **Answer Correctness:** Does the generated answer contain the same core information as the ground-truth answer? Is it factually accurate based on the ground truth?
2.  **Citation Accuracy:** Are the claims in the generated answer supported by the provided source context? Check for hallucinations.
3.  **Source Faithfulness:** Does the generated answer faithfully represent the information in the source context, or does it misinterpret it?
4.  **Abstention Detection:** Did the RAG system correctly abstain from answering (e.g., "I don't know") if the source context lacked the necessary information? If the ground-truth answer indicates the question is unanswerable, the generated answer should abstain.

## OUTPUT FORMAT
Respond with a single JSON object conforming to this schema:
{
  "overall_score": "PASS" | "FAIL",
  "answer_correctness": {
    "score": "PASS" | "FAIL" | "PARTIAL",
    "explanation": "A concise explanation comparing the generated and ground-truth answers."
  },
  "citation_accuracy": {
    "score": "PASS" | "FAIL" | "NO_CITATIONS",
    "unsupported_claims": ["List of claims in the generated answer not found in the source context."],
    "explanation": "A concise explanation of the citation audit."
  },
  "source_faithfulness": {
    "score": "PASS" | "FAIL",
    "explanation": "A concise explanation of whether the generated answer misinterprets the source context."
  },
  "abstention_detection": {
    "score": "CORRECT_ABSTENTION" | "INCORRECT_ABSTENTION" | "CORRECT_ANSWER" | "INCORRECT_ANSWER" | "NOT_APPLICABLE",
    "explanation": "A concise explanation of the abstention analysis."
  }
}

## CONSTRAINTS
- Base your evaluation strictly on the provided inputs.
- Do not use outside knowledge.
- If the generated answer is empty or contains only an abstention phrase, evaluate it as an abstention.

To adapt this template, start by customizing the OUTPUT FORMAT schema to match your specific evaluation needs. You might add a severity field to unsupported_claims or a partial_credit_details object. The EVALUATION CRITERIA section can also be expanded with domain-specific rules, such as numerical tolerance checks for financial data. For high-stakes evaluations, always pair this prompt with a human review step, especially for FAIL or PARTIAL scores, to validate the model's judgment before it is used in a regression test suite or performance report.

This prompt is designed to operate as a programmatic evaluation step within a RAG system's CI/CD or regression testing pipeline, not as a one-off manual check. The implementation harness should treat the LLM call as a deterministic function: it receives a structured input containing the user question, the RAG-generated answer, and the ground-truth reference, and it must return a machine-readable comparison report. The primary integration point is immediately after a RAG pipeline produces an answer, where this prompt acts as an automated judge before the answer reaches an end user or a dataset curator. The harness must enforce a strict JSON output schema, as the comparison report's fields—such as answer_correctness, citation_accuracy, source_faithfulness, and abstention_detected—are consumed by downstream metric aggregators and release gates.

To wire this into an application, wrap the prompt call in a function that constructs the [USER_QUESTION], [RAG_ANSWER], and [GROUND_TRUTH_REFERENCE] placeholders from your test dataset. The model should be instructed to return a JSON object conforming to a predefined [OUTPUT_SCHEMA], which your harness must validate immediately. Implement a validation layer using a tool like Pydantic or JSON Schema that checks for the presence and correct types of all required fields. If validation fails, implement a retry strategy with a maximum of 2 additional attempts, feeding the validation error message back into the prompt's [CONSTRAINTS] or as a follow-up correction request. Log every raw response, validation result, and retry attempt to your observability platform (e.g., a comparison_eval log stream) for trace analysis. For high-stakes domains like healthcare or finance, route any comparison where answer_correctness is false or citation_accuracy falls below a configurable threshold (e.g., 0.9) to a human review queue, attaching the full comparison report and source artifacts.

Model choice matters for this harness. Use a model with strong instruction-following and JSON mode capabilities, such as GPT-4o, Claude 3.5 Sonnet, or a fine-tuned evaluator model. Set a low temperature (0.0–0.2) to maximize deterministic scoring. The harness should also implement a circuit breaker: if the retry loop exhausts all attempts or the model returns a refusal, mark the evaluation as indeterminate and escalate for manual review rather than silently passing or failing the RAG output. Finally, avoid the anti-pattern of using this prompt to evaluate every single production query in real-time; it is best suited for offline batch evaluation, regression test suites, and periodic sampling. For real-time guardrails, pair it with a lighter-weight, higher-recall groundedness check and reserve this detailed comparison for deep-dive analysis and release validation.