Source-Answer Alignment Rubric Prompt

This prompt is an LLM-as-judge evaluation tool for teams that need to measure how faithfully a generated answer aligns with its cited source material. Use it when you are building an automated evaluation pipeline, comparing two model outputs against the same evidence, or running regression tests on a RAG system. The prompt produces a structured rubric assessment across three dimensions: faithfulness (are claims supported?), coverage (are all source-relevant points included?), and contradiction detection (does the answer conflict with the evidence?). It is designed for evaluation engineers, AI quality teams, and RAG pipeline builders who need a repeatable, score-based alignment check before answers reach users.

This prompt assumes you already have a generated answer and the full text of its cited sources. It does not retrieve sources, verify citation formatting, or check source authority. Pair it with a citation verification prompt if you need to validate that citations point to real documents. The rubric output is structured for downstream automation: you can threshold the scores to gate deployment, route low-faithfulness outputs for human review, or feed pairwise comparison results into a model selection decision. For high-risk domains such as healthcare or legal, always include human review of rubric results before acting on them.

Do not use this prompt when you need real-time answer generation, source retrieval, or user-facing explanations. It is an offline evaluation tool, not a production guardrail. If your goal is to detect hallucinated citations rather than assess answer-source alignment, use the hallucinated citation detection prompt instead. For span-level claim-to-source mapping, use the claim-to-source linking prompt. This rubric prompt works best as part of a broader evaluation suite that includes regression tests, human annotation spot checks, and output validation guardrails.

This prompt template implements a detailed source-answer alignment rubric for LLM-as-judge evaluation. It assesses faithfulness, coverage, and contradiction between a full answer and its cited sources. The template supports both single-answer evaluation and pairwise comparison for model selection or regression testing. Replace the square-bracket placeholders with your specific answer text, source documents, and evaluation parameters before running it in your harness.

textCopy
You are an expert evaluator assessing how well an answer aligns with its cited sources. Your task is to produce a detailed alignment assessment using the structured rubric below.

## INPUT

**Answer to Evaluate:**
[ANSWER_TEXT]

**Cited Sources:**
[SOURCE_DOCUMENTS]

**Evaluation Context (optional):**
[EVALUATION_CONTEXT]

## RUBRIC DIMENSIONS

For each dimension, assign a score from 1 (poor) to 5 (excellent) and provide a brief justification with specific evidence.

### 1. Faithfulness
Does the answer only make claims that are directly supported by the cited sources? Penalize any statement not grounded in the provided sources.
- Score 5: Every claim is directly traceable to a specific source passage.
- Score 3: Most claims are supported, but some minor extrapolations or embellishments exist.
- Score 1: Multiple unsupported claims or fabrications present.

### 2. Coverage
Does the answer incorporate all relevant information from the cited sources, or does it omit material facts?
- Score 5: All material facts from sources are represented accurately.
- Score 3: Key facts are included, but some relevant details are missing.
- Score 1: Critical information from sources is omitted or misrepresented.

### 3. Contradiction Detection
Does the answer contain any statements that directly contradict the cited sources?
- Score 5: No contradictions found; answer is fully consistent with all sources.
- Score 3: Minor inconsistencies in phrasing or emphasis, but no factual contradictions.
- Score 1: Direct factual contradictions with one or more sources.

### 4. Source Utilization
Are the cited sources used appropriately? Check for cherry-picking, out-of-context quotes, or ignoring source qualifiers.
- Score 5: Sources are used faithfully with appropriate context and qualifiers preserved.
- Score 3: Sources are generally used well, but some context or nuance is lost.
- Score 1: Sources are misrepresented, quoted out of context, or key qualifiers are stripped.

## OUTPUT FORMAT

Return a JSON object with this exact structure:

{
  "overall_score": <float between 1.0 and 5.0>,
  "dimension_scores": {
    "faithfulness": {"score": <int 1-5>, "justification": "<specific evidence>"},
    "coverage": {"score": <int 1-5>, "justification": "<specific evidence>"},
    "contradiction_detection": {"score": <int 1-5>, "justification": "<specific evidence>"},
    "source_utilization": {"score": <int 1-5>, "justification": "<specific evidence>"}
  },
  "unsupported_claims": ["<list any claims lacking source support>"],
  "contradictions_found": ["<list any contradictions with source references>"],
  "missing_information": ["<list material facts from sources not in answer>"],
  "summary": "<2-3 sentence overall assessment>"
}

## PAIRWISE COMPARISON (if applicable)

[PAIRWISE_INSTRUCTIONS: If comparing two answers, evaluate both independently using the rubric above, then add a "comparison" field with "winner": "A" or "B" or "tie" and "rationale": "<explanation>". Provide both answers as [ANSWER_A] and [ANSWER_B] with the same source set.]

## CONSTRAINTS

- Only use the provided sources as ground truth. Do not bring in external knowledge.
- If a source contains uncertainty language (e.g., "may," "suggests," "unclear"), the answer must preserve that uncertainty.
- Flag any claim that appears factual but cannot be located in the sources.
- For pairwise comparison, evaluate each answer independently before comparing.

Adapt this template by adjusting the rubric dimensions to match your domain's requirements. For regulated industries, add dimensions for compliance language preservation or disclaimer inclusion. For technical documentation, add a dimension for procedural accuracy. The output schema can be extended with domain-specific fields such as regulatory_flags or technical_accuracy_notes. Always test the rubric against a golden dataset of known alignments and misalignments before deploying to production evaluation pipelines. For high-stakes assessments, route low-scoring outputs to human review rather than relying solely on the automated score.

This prompt is designed for an LLM-as-judge evaluation harness, not a user-facing chatbot. It expects a structured input containing a full answer and its cited sources, and it produces a detailed rubric assessment. The primary integration point is an automated evaluation pipeline that runs after answer generation, comparing model outputs against a golden dataset or performing pairwise model comparisons. The harness must enforce the input schema, validate the output structure, and log results for analysis.

Input assembly requires merging the generated answer with its retrieved sources into the [ANSWER] and [SOURCES] placeholders. If your RAG pipeline produces citations inline, extract and normalize them into a list of source objects with id and content fields before calling this prompt. Output validation should parse the JSON response and verify that all required rubric dimensions (faithfulness, coverage, contradiction) are present with numeric scores and textual justifications. Reject malformed responses and retry with a stricter format instruction appended to the prompt. For pairwise comparison mode, ensure both [ANSWER_A] and [ANSWER_B] are populated and validate that the output includes a comparison object with a winner field and per-dimension breakdown.

Model selection matters: use a model with strong instruction-following and structured output capabilities, such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro. Avoid smaller models that may drift on the rubric scale or omit required justification fields. Retry logic should handle JSON parse failures (up to 2 retries with escalating format constraints) and refusal responses where the model declines to evaluate. Logging must capture the raw prompt, the full output, parse success/failure, and the extracted scores for dashboarding. For high-stakes evaluations (e.g., model release gates), route a sample of assessments to human review to calibrate the LLM judge against human annotators and detect rubric drift over time. Never treat the rubric scores as ground truth without periodic calibration checks.